Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions backend/cosmetology-app/docs/design/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -382,10 +382,19 @@ The search infrastructure consists of several key components:
4. **Populate Handler**: A Lambda function for bulk indexing all provider data from DynamoDB
5. **Provider Update Ingest Handler**: A Lambda function for updating documents in OpenSearch whenever provider records are updated in DynamoDB.

### Document model (Cosmetology vs. JCC)

Unlike the JCC CompactConnect model, which indexes **one OpenSearch document per provider** (with that provider’s licenses nested in a single document), Cosmetology indexes **one document per license**. Each document repeats the same top-level provider fields you would see on a provider detail response, while the `licenses` array contains **only the license represented by that document** (effectively one license entry per document).

Cosmetology needs to support searching and listing **rows of license records** by license number in the search UI. OpenSearch pagination (`from`/`size`, `search_after`, etc.) applies to **documents**, not to entries inside a nested array. Splitting each license into its own document lets the UI paginate natively at license granularity. It also keeps the search API response model consistent across the compacts.

Most practitioners only have one multi-state license, so this model does not significantly increase the size of storage used by the OpenSearch domain.

### Index Structure

Provider documents are stored in compact-specific indices with the naming convention: `compact_{compact}_providers_{version}`
(e.g., `compact_aslp_providers_v1`). We use index aliases to provide a stable reference to the current version of each index, allowing read and write operations to be transparently redirected during planned index migrations or upgrades. This enables seamless index schema changes without requiring app code changes, as applications and APIs can continue to reference the alias rather than a specific index name. See [OpenSearch index alias documentation](https://docs.opensearch.org/latest/im-plugin/index-alias/) for more information.
Documents are stored in compact-specific indices with the naming convention: `compact_{compact}
_providers_{version}`
(e.g., `compact_cosm_providers_v1`). We use index aliases to provide a stable reference to the current version of each index, allowing read and write operations to be transparently redirected during planned index migrations or upgrades. This enables seamless index schema changes without requiring app code changes, as applications and APIs can continue to reference the alias rather than a specific index name. See [OpenSearch index alias documentation](https://docs.opensearch.org/latest/im-plugin/index-alias/) for more information.

#### Index Management

Expand All @@ -394,7 +403,7 @@ domain is first created. It ensures the indices/aliases exist with the correct m

#### Index Mapping

Each provider document contains all information you would see from the provider detail api endpoint with `readGeneral` permission. See the [application code](../../lambdas/python/search/handlers/manage_opensearch_indices.py) for the current mapping definition.
Each indexed document corresponds to **one license** and uses the same overall shape as the provider detail API with `readGeneral` permission. See the [application code](../../lambdas/python/search/handlers/manage_opensearch_indices.py) for the current mapping definition. Document construction (one sanitized document per license, including composite `documentId`) is implemented in [search/utils.py](../../lambdas/python/search/utils.py).

The index uses a custom ASCII-folding analyzer for name fields, which allows searching for names with international
characters using their ASCII equivalents (e.g., searching "Jose" matches "José").
Expand All @@ -408,7 +417,7 @@ The Search API provides two endpoints for querying the OpenSearch domain:
POST /v1/compacts/{compact}/providers/search
```

Returns provider records matching the query. Response includes the full provider document with licenses and privileges.
Returns one result row per indexed document (one per license). Each hit is a full provider-shaped document for that license row (including the single license in `licenses` and generated privileges as applicable).

### Document Indexing

Expand All @@ -422,7 +431,7 @@ OpenSearch. This function is invoked manually through the AWS Console for:
The function:
1. Scans the provider table using the `providerDateOfUpdate` GSI
2. Retrieves complete provider records for each provider
3. Sanitizes data using `ProviderGeneralResponseSchema`
3. Expands each provider into **one OpenSearch document per license** (sanitized via `ProviderOpenSearchDocumentSchema`)
4. Bulk indexes documents

**Resumable Processing**: If the function approaches the 15-minute Lambda timeout, it returns pagination information in the
Expand All @@ -442,11 +451,11 @@ The function:
3. The DynamoDB stream handler queries the data and indexes the change into OpenSearch after the ~30 second delay of sitting in SQS
4. The `populate_provider_documents` Lambda function finally indexes the stale data into OpenSearch, overwriting the change indexed by the DynamoDB stream handler

For this reason, it is recommended that this process be run during a period of low traffic. Given that it is a one-time process to initially populate the table, the risk is low and if needed, the Lambda function can be run again to synchronize all the provider documents.
For this reason, it is recommended that this process be run during a period of low traffic. Given that it is a one-time process to initially populate the table, the risk is low and if needed, the Lambda function can be run again to synchronize all indexed documents.

#### Updates via DynamoDB Streams

To keep the OpenSearch index synchronized with changes in the provider DynamoDB table, the system uses DynamoDB Streams to capture all modifications made to provide records (see [AWS documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html)). This ensures that provider documents in OpenSearch are updated automatically whenever records are created, modified, or deleted in the provider table.
To keep the OpenSearch index synchronized with changes in the provider DynamoDB table, the system uses DynamoDB Streams to capture all modifications made to provider records (see [AWS documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html)). This ensures that the corresponding license documents in OpenSearch are updated automatically whenever records are created, modified, or deleted in the provider table.

**Architecture Flow:**

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2533,7 +2533,6 @@
"jurisdictionAdverseActionsNotificationEmails",
"jurisdictionName",
"jurisdictionOperationsTeamEmails",
"jurisdictionSummaryReportNotificationEmails",
"licenseeRegistrationEnabled",
"postalAbbreviation"
],
Expand Down Expand Up @@ -2573,14 +2572,6 @@
"jurisdictionName": {
"type": "string",
"description": "The name of the jurisdiction"
},
"jurisdictionSummaryReportNotificationEmails": {
"type": "array",
"description": "List of email addresses for summary report notifications",
"items": {
"type": "string",
"format": "email"
}
}
}
},
Expand Down Expand Up @@ -2745,7 +2736,6 @@
"required": [
"jurisdictionAdverseActionsNotificationEmails",
"jurisdictionOperationsTeamEmails",
"jurisdictionSummaryReportNotificationEmails",
"licenseeRegistrationEnabled"
],
"type": "object",
Expand Down Expand Up @@ -2775,17 +2765,6 @@
"licenseeRegistrationEnabled": {
"type": "boolean",
"description": "Denotes whether licensee registration is enabled"
},
"jurisdictionSummaryReportNotificationEmails": {
"maxItems": 10,
"minItems": 1,
"uniqueItems": true,
"type": "array",
"description": "List of email addresses for summary report notifications",
"items": {
"type": "string",
"format": "email"
}
}
},
"additionalProperties": false
Expand Down Expand Up @@ -4443,7 +4422,6 @@
"compactAdverseActionsNotificationEmails",
"compactName",
"compactOperationsTeamEmails",
"compactSummaryReportNotificationEmails",
"configuredStates",
"licenseeRegistrationEnabled"
],
Expand Down Expand Up @@ -4482,14 +4460,6 @@
}
}
},
"compactSummaryReportNotificationEmails": {
"type": "array",
"description": "List of email addresses for summary report notifications",
"items": {
"type": "string",
"format": "email"
}
},
"compactAdverseActionsNotificationEmails": {
"type": "array",
"description": "List of email addresses for adverse actions notifications",
Expand Down Expand Up @@ -4524,7 +4494,6 @@
"required": [
"compactAdverseActionsNotificationEmails",
"compactOperationsTeamEmails",
"compactSummaryReportNotificationEmails",
"configuredStates",
"licenseeRegistrationEnabled"
],
Expand Down Expand Up @@ -4564,17 +4533,6 @@
"additionalProperties": false
}
},
"compactSummaryReportNotificationEmails": {
"maxItems": 10,
"minItems": 1,
"uniqueItems": true,
"type": "array",
"description": "List of email addresses for summary report notifications",
"items": {
"type": "string",
"format": "email"
}
},
"compactAdverseActionsNotificationEmails": {
"maxItems": 10,
"minItems": 1,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -444,7 +444,7 @@
"response": [
{
"_postman_previewlanguage": "json",
"body": "{\n \"compactAbbr\": \"<string>\",\n \"compactAdverseActionsNotificationEmails\": [\n \"<email>\",\n \"<email>\"\n ],\n \"compactName\": \"<string>\",\n \"compactOperationsTeamEmails\": [\n \"<email>\",\n \"<email>\"\n ],\n \"compactSummaryReportNotificationEmails\": [\n \"<email>\",\n \"<email>\"\n ],\n \"configuredStates\": [\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"wa\"\n },\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"va\"\n }\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\"\n}",
"body": "{\n \"compactAbbr\": \"<string>\",\n \"compactAdverseActionsNotificationEmails\": [\n \"<email>\",\n \"<email>\"\n ],\n \"compactName\": \"<string>\",\n \"compactOperationsTeamEmails\": [\n \"<email>\",\n \"<email>\"\n ],\n \"configuredStates\": [\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"wa\"\n },\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"va\"\n }\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\"\n}",
"code": 200,
"cookie": [],
"header": [
Expand Down Expand Up @@ -505,7 +505,7 @@
"language": "json"
}
},
"raw": "{\n \"compactAdverseActionsNotificationEmails\": [\n \"<email>\"\n ],\n \"compactOperationsTeamEmails\": [\n \"<email>\"\n ],\n \"compactSummaryReportNotificationEmails\": [\n \"<email>\"\n ],\n \"configuredStates\": [\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"wa\"\n },\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"ks\"\n }\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\"\n}"
"raw": "{\n \"compactAdverseActionsNotificationEmails\": [\n \"<email>\"\n ],\n \"compactOperationsTeamEmails\": [\n \"<email>\"\n ],\n \"configuredStates\": [\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"wa\"\n },\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"ks\"\n }\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\"\n}"
},
"description": {},
"header": [
Expand Down Expand Up @@ -567,7 +567,7 @@
"language": "json"
}
},
"raw": "{\n \"compactAdverseActionsNotificationEmails\": [\n \"<email>\"\n ],\n \"compactOperationsTeamEmails\": [\n \"<email>\"\n ],\n \"compactSummaryReportNotificationEmails\": [\n \"<email>\"\n ],\n \"configuredStates\": [\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"wa\"\n },\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"ks\"\n }\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\"\n}"
"raw": "{\n \"compactAdverseActionsNotificationEmails\": [\n \"<email>\"\n ],\n \"compactOperationsTeamEmails\": [\n \"<email>\"\n ],\n \"configuredStates\": [\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"wa\"\n },\n {\n \"isLive\": \"<boolean>\",\n \"postalAbbreviation\": \"ks\"\n }\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\"\n}"
},
"header": [
{
Expand Down Expand Up @@ -760,7 +760,7 @@
"response": [
{
"_postman_previewlanguage": "json",
"body": "{\n \"compact\": \"cosm\",\n \"jurisdictionAdverseActionsNotificationEmails\": [\n \"<email>\",\n \"<email>\"\n ],\n \"jurisdictionName\": \"<string>\",\n \"jurisdictionOperationsTeamEmails\": [\n \"<email>\",\n \"<email>\"\n ],\n \"jurisdictionSummaryReportNotificationEmails\": [\n \"<email>\",\n \"<email>\"\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\",\n \"postalAbbreviation\": \"<string>\"\n}",
"body": "{\n \"compact\": \"cosm\",\n \"jurisdictionAdverseActionsNotificationEmails\": [\n \"<email>\",\n \"<email>\"\n ],\n \"jurisdictionName\": \"<string>\",\n \"jurisdictionOperationsTeamEmails\": [\n \"<email>\",\n \"<email>\"\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\",\n \"postalAbbreviation\": \"<string>\"\n}",
"code": 200,
"cookie": [],
"header": [
Expand Down Expand Up @@ -823,7 +823,7 @@
"language": "json"
}
},
"raw": "{\n \"jurisdictionAdverseActionsNotificationEmails\": [\n \"<email>\"\n ],\n \"jurisdictionOperationsTeamEmails\": [\n \"<email>\"\n ],\n \"jurisdictionSummaryReportNotificationEmails\": [\n \"<email>\"\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\"\n}"
"raw": "{\n \"jurisdictionAdverseActionsNotificationEmails\": [\n \"<email>\"\n ],\n \"jurisdictionOperationsTeamEmails\": [\n \"<email>\"\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\"\n}"
},
"description": {},
"header": [
Expand Down Expand Up @@ -897,7 +897,7 @@
"language": "json"
}
},
"raw": "{\n \"jurisdictionAdverseActionsNotificationEmails\": [\n \"<email>\"\n ],\n \"jurisdictionOperationsTeamEmails\": [\n \"<email>\"\n ],\n \"jurisdictionSummaryReportNotificationEmails\": [\n \"<email>\"\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\"\n}"
"raw": "{\n \"jurisdictionAdverseActionsNotificationEmails\": [\n \"<email>\"\n ],\n \"jurisdictionOperationsTeamEmails\": [\n \"<email>\"\n ],\n \"licenseeRegistrationEnabled\": \"<boolean>\"\n}"
},
"header": [
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,8 @@ The lambda is intended to be invoked directly, rather than through an API endpoi
recipientType: // must be one of the following
| 'COMPACT_OPERATIONS_TEAM' // compactOperationsTeamEmails
| 'COMPACT_ADVERSE_ACTIONS' // compactAdverseActionsNotificationEmails
| 'COMPACT_SUMMARY_REPORT' // compactSummaryReportNotificationEmails
| 'JURISDICTION_OPERATIONS_TEAM' // jurisdictionOperationsTeamEmails
| 'JURISDICTION_ADVERSE_ACTIONS' // jurisdictionAdverseActionsNotificationEmails
| 'JURISDICTION_SUMMARY_REPORT' // jurisdictionSummaryReportNotificationEmails
| 'SPECIFIC'; // specificEmails provided in payload
compact: string; // Compact identifier
jurisdiction?: string; // Optional jurisdiction identifier, must be specified if sending to a Jurisdiction based email list
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ export interface Compact {
compactAbbr: string;
compactName: string;
compactOperationsTeamEmails: string[];
compactSummaryReportNotificationEmails: string[];
dateOfUpdate: string;
type: string;
}
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
export type RecipientType =
| 'COMPACT_OPERATIONS_TEAM'
| 'COMPACT_ADVERSE_ACTIONS'
| 'COMPACT_SUMMARY_REPORT'
| 'JURISDICTION_OPERATIONS_TEAM'
| 'JURISDICTION_ADVERSE_ACTIONS'
| 'JURISDICTION_SUMMARY_REPORT'
| 'SPECIFIC';

export interface EmailNotificationEvent {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,4 @@ export interface IJurisdiction {
compact: string;
jurisdictionOperationsTeamEmails: string[];
jurisdictionAdverseActionsNotificationEmails: string[];
jurisdictionSummaryReportNotificationEmails: string[];
}
4 changes: 2 additions & 2 deletions backend/cosmetology-app/lambdas/nodejs/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"type": "commonjs",
"description": "NodeJS lambdas for Compact Connect",
"resolutions": {
"fast-xml-parser": "5.3.6"
"fast-xml-parser": "5.5.7"
},
"scripts": {
"build": "tsc",
Expand Down Expand Up @@ -48,7 +48,7 @@
"@aws-sdk/client-s3": "^3.901.0",
"@aws-sdk/client-sesv2": "^3.901.0",
"@aws-sdk/util-dynamodb": "^3.901.0",
"@csg-org/email-builder": "^0.0.9-alpha.4",
"@csg-org/email-builder": "^0.0.12",
"nodemailer": "^7.0.11",
"zod": "^3.23.8"
}
Expand Down
Loading
Loading