CHI-3719: Create script for importing PRN iCarol contacts to HRM by stephenhand · Pull Request #1100 · techmatters/hrm

stephenhand · 2026-06-29T15:55:48Z

Description

Adds an admin CLI command (import icarol contacts) that imports PRN iCarol "CallReports" CSV exports into HRM as contacts.

Loads the CSV export from an S3 URI (@tech-matters/s3-client) and parses it into typed ICarolContactRecord rows using csv-parse. iCarol exports prefix the file with a title row and a blank row, so parsing starts at line 3.
Maps iCarol fields onto the Aselo contact payload per the PRN field mapping spreadsheet: "Contact > Support Seeker" → rawJson.childInformation, "Contact > Summary" → rawJson.caseInformation, and the "Eight Dimensions of Wellness" → rawJson.categories (each dimension mapped to its category and an "Unspecified/Other" subcategory). Yes/No values are coerced to booleans and blank columns are omitted. Target field names are taken from the Aselo usnc/v1 form definitions (e.g. friendlyName, referral988, wasTheCallerSatisfiedWithTheSupportProvided, doWeHaveTheirPermissionToCallBack, referrals); the unsupported city column is dropped.
Derives the call type from the iCarol "Call Information - Call Type" field: Crisis/Non-Crisis map to the data callType Child calling about self with a boolean caseInformation.isCrisis, and when that field is empty the callType is inferred from the iCarol boolean flag columns (WasRealCall, WasSilentCall, WasHangup, WasWrongNumber, WasPrankCall, WasSexCall) using the PRN CallTypeButtons labels.
Computes conversationDuration (seconds) from the CallDateAndTimeStart/CallDateAndTimeEnd timestamps, defaulting to 0 when either is missing or unparseable.
Attributes each contact to the counsellor named in the iCarol "PhoneWorkerName" column. buildWorkerSidMap (modelled on the Flex populateCounselors lambda) lists the account's TaskRouter workers and builds an in-memory full_name → worker SID map, reading the Twilio auth token and workspace SID from SSM (/${environment}/twilio/${accountSid}/auth_token and .../workspace_sid) and creating the client via @tech-matters/twilio-client's getClient. When a PhoneWorkerName resolves to a worker SID, that SID populates twilioWorkerId, createdBy, and the contactlessTask.createdOnBehalfOf field in rawJson. Records whose worker name has no match fall back to a required default worker SID passed via the --fallback-worker-sid CLI parameter, with a warning logged that includes the CallReportNum and PhoneWorkerName.
The mapping/parsing logic lives in a dedicated contactMapper.ts module (parseICarolBoolean, parseS3Uri, mapCategories, mapCallType, calculateConversationDuration, resolveWorkerSid, mapContact), keeping it isolated from the S3/HTTP/Twilio I/O in contacts.ts so it can be unit tested. parseS3Uri strips the leading slash from the S3 key with an explicit startsWith/slice check rather than a regex replace.
Posts each mapped contact to the admin contacts endpoint, logging any per-record failures.
Adds the @tech-matters/s3-client dependency, moves csv-parse to runtime dependencies, and moves @tech-matters/twilio-client to runtime dependencies.
Adds unit tests (contactMapper.test.ts) covering the mapping/importing logic (including resolveWorkerSid and worker attribution), plus a test:unit script on the hrm-service package so they run under npm run test:unit.

Note: the original TODO referenced rawJson.caseSummary, but ContactRawJson has no such field — the Summary tab is stored under caseInformation, so it is mapped there.

Checklist

Corresponding issue has been opened
New tests added
Feature flags / configuration added

Other Related Issues

None

Verification steps

Run the admin CLI command against an environment, pointing at an iCarol CSV export in S3:

npm run admin-cli -- import icarol contacts -e <environment> -r <region> -a <accountSid> -l s3://<bucket>/<path-to-export>.csv -f <fallback-worker-sid>

Confirm the contacts are created in HRM with the expected childInformation, caseInformation, categories, callType and conversationDuration values, and that contacts whose PhoneWorkerName matches a Twilio worker are attributed to that worker (twilioWorkerId, createdBy, and rawJson.contactlessTask.createdOnBehalfOf all set to the worker SID). Contacts whose PhoneWorkerName does not match a worker should be attributed to the fallback worker SID, with warnings logged including the CallReportNum and PhoneWorkerName.

The mapping logic can be exercised in isolation with the unit tests via npm run test:unit in hrm-domain/hrm-service.

AFTER YOU MERGE

Cut a release tag using the GitHub workflow. Wait for it to complete and notify in the #aselo-deploys Slack channel.
Comment on the ticket with the release tag version AND any additional instructions required to configure an environment to test the changes.
Only then move the ticket into the QA column in JIRA

You are responsible for ensuring the above steps are completed. If you move a ticket into QA without advising what version to test, the QA team will assume the latest tag has the changes. If it does not, the following confusion is on you! :-P

stephenhand · 2026-06-30T06:42:20Z

@copilot
[external] Aselo __ PRN Field Mapping.xlsx
iCarolExport-PromiseResourceNetwo-CallReports-Anon.csv

Using the mapping spreadsheet and example CSV attached, complete the TODO items in the comments that were added in this PR

Copilot · 2026-06-30T06:56:37Z

@copilot
[external] Aselo __ PRN Field Mapping.xlsx
[iCarolExport-PromiseResourceNetwo...

Done in d3596de. Both TODOs are now implemented in contacts.ts:

Load & parse the CSV — added an ICarolContactRecord type for the export rows, parse the file from S3 (@tech-matters/s3-client) with csv-parse. iCarol exports prefix the file with a title row and a blank row, so parsing starts at line 3 (from_line: 3).
Field mapping — per the mapping spreadsheet, "Contact > Support Seeker" fields map onto rawJson.childInformation, "Contact > Summary" fields onto rawJson.caseInformation, and the "Eight Dimensions of Wellness" onto rawJson.categories (each dimension → its category + "Unspecified/Other" subcategory). Yes/No values are coerced to booleans and blank columns are omitted.

One deviation to flag: the TODO referenced rawJson.caseSummary, but ContactRawJson has no such field — the Summary tab is stored under caseInformation (e.g. caseInformation.callSummary), so I mapped it there. A couple of value translations (e.g. demographic option labels that differ between iCarol and Aselo) are passed through as-is since the spreadsheet's Lexicon didn't include the iCarol→Aselo value maps; happy to add those if you can share them.

stephenhand

@copilot - write a set of unit tests for the mapping / importing

stephenhand · 2026-06-30T07:12:40Z

+  if (protocol !== 's3:') {
+    throw new Error(`location must be an S3 URI (s3://bucket/key), got: ${location}`);
+  }
+  return { bucket: hostname, key: decodeURIComponent(pathname.replace(/^\//, '')) };


@copilot Can we use a more explicit method to exclude a leading slash, the 'replace' method behaviour can be confusing

Done in f6de203. parseS3Uri now strips the leading slash explicitly: const key = pathname.startsWith('/') ? pathname.slice(1) : pathname; instead of the regex replace.

…arse

Copilot · 2026-06-30T07:29:42Z

@copilot - write a set of unit tests for the mapping / importing

Added in f6de203. I extracted the pure mapping/parsing logic into contactMapper.ts and added contactMapper.test.ts with 30 cases covering parseICarolBoolean, parseS3Uri, mapCategories and mapContact (childInformation/caseInformation mapping, boolean coercion, blank-column omission, wellness categories, call-length conversion, etc.). They run via a new test:unit script on hrm-service (npm run test:unit).

stephenhand

@copilot Here is a link to the Aselo form definitions that map display names used in the mapping spreadsheet to field names: https://github.com/techmatters/flex-plugins/tree/master/lambdas/packages/hrm-form-definitions/form-definitions/usnc/v1

Correct any target field names using this information~

Address the other 2 comments in this review

stephenhand · 2026-06-30T08:12:37Z

+  );
+
+  const rawJson: ContactRawJson = {
+    callType: record['Call Information - Call Type'] || '',


As described in the spreadsheet, if this field is empty you need to fall back to inferring from the boolean fields described in row 18

If the value here is crisis or non crisis, substitute for 'Child calling about self' and record crisis / non crisis as a boolean 'isCrisis' field under caseInformation

Done in 3e38310 (mapCallType). When "Call Information - Call Type" is Crisis/Non-Crisis, the callType is set to Child calling about self and caseInformation.isCrisis records the crisis/non-crisis boolean. When that field is empty, the callType is inferred from the row-18 boolean flags (WasRealCall→Child calling about self, WasSilentCall→Silent, WasHangup→Hang up, WasWrongNumber→Wrong Number, WasPrankCall→Prank Call, WasSexCall→Sexual Gratifier, using the PRN CallTypeButtons labels).

stephenhand · 2026-06-30T08:15:49Z

+    ),
+  };
+
+  const callLengthMinutes = Number.parseInt(record.CallLength, 10);


Calculate this from CallDateAndTimeStart and CallDateAndTimeEnd rather than using CallLength

Done in 3e38310. conversationDuration is now computed by calculateConversationDuration from CallDateAndTimeStart/CallDateAndTimeEnd (seconds), falling back to 0 when either timestamp is missing or unparseable. CallLength is no longer used.

… map crisis callType

… label

Copilot · 2026-06-30T09:13:07Z

@copilot Here is a link to the Aselo form definitions that map display names used in the mapping spreadsheet to field names: https://github.com/techmatters/flex-plugins/tree/master/lambdas/packages/h...

Done in 3e38310. Corrected the target field names against the usnc/v1 form definitions:

childInformation: name→friendlyName, referralTo988→referral988; dropped city (no such field in the form — county/state remain).
caseInformation: callerSatisfied→wasTheCallerSatisfiedWithTheSupportProvided, permissionToCallBack→doWeHaveTheirPermissionToCallBack, typeOfResourceProvided→referrals.
categories: the eight wellness category/Unspecified/Other subcategory names already matched IssueCategorizationTab.json, so unchanged.

The other two review comments are addressed in the same commit (see thread replies).

stephenhand · 2026-06-30T09:33:25Z

@copilot - we need to look up Twilio worker SIDs from PhoneWorkerName to populate the twilioWorkerId and createdBy fields, and the createdOnBehalfOf field in the contactlessTask rawJson object

https://github.com/techmatters/flex-plugins/blob/master/lambdas/account-scoped/src/worker/populateCounselors.ts - use this code as a guide to looking up twilio workers on the specified account, creating an in memory mapping and using that mapping to go from PhoneWorkerName (FullName) -> SID

https://github.com/techmatters/flex-plugins/blob/47234b5d29dd26fed9fe13f90673259fc7f6f501/lambdas/packages/twilio-configuration/src/twilioConfiguration.ts#L48 - look up the auth token in our SSM store like this

…e imported contacts

…Sid for missing-worker warning

Copilot · 2026-06-30T09:45:01Z

@copilot - we need to look up Twilio worker SIDs from PhoneWorkerName to populate the twilioWorkerId and createdBy fields, and the createdOnBehalfOf field in the contactlessTask rawJso...

Done. iCarol contacts are now attributed to the counsellor named in PhoneWorkerName:

buildWorkerSidMap (in contacts.ts) lists the account's TaskRouter workers and builds an in-memory full_name → SID map, modelled on the Flex populateCounselors lambda. The Twilio auth token and workspace SID are read from SSM (/${environment}/twilio/${accountSid}/auth_token and .../workspace_sid), and the client is created via @tech-matters/twilio-client's getClient.
When PhoneWorkerName resolves to a worker SID, that SID populates twilioWorkerId, createdBy, and rawJson.contactlessTask.createdOnBehalfOf. Records whose worker name has no match are imported unattributed with a warning logged.
Added unit tests for resolveWorkerSid and the worker-attribution branch of mapContact.

stephenhand · 2026-06-30T11:51:26Z

@copilot - If a worker cannot be looked up from the PhoneWorkerName, fall back to a default passed in as a CLI parameter rather than omitting it, the worker sid for these fields should be considered required. Log a warning for all records where the lookup fails, including the CallReportNum and the PhoneWorkerName the lookup was attempted on

- Add --fallback-worker-sid (-f) CLI parameter (required) - Use fallback worker SID when PhoneWorkerName lookup fails - Log warnings with CallReportNum and PhoneWorkerName when lookup fails - Update mapContact to require worker SID parameter instead of optional lookup map - Update all tests to pass default worker SID

Copilot · 2026-06-30T11:59:54Z

@copilot - If a worker cannot be looked up from the PhoneWorkerName, fall back to a default passed in as a CLI parameter rather than omitting it, the worker sid for these fields shou...

Done. Added a required --fallback-worker-sid CLI parameter. When a PhoneWorkerName doesn't resolve to a Twilio worker, the import now falls back to the provided worker SID and logs a warning that includes the CallReportNum and PhoneWorkerName (e.g., No Twilio worker found for PhoneWorkerName "Jane Doe" (call report 12345); falling back to default worker WKxxx).

stephenhand · 2026-07-27T10:10:36Z

@mckinneybock - in case this work comes your way, this is a copilot assisted, untested first stab at an importing PRN contacts, if it's of any value to you

Create script for importing PRN iCarol contacts to HRM

6d0856c

Copilot started work on behalf of stephenhand June 30, 2026 06:42 View session

CHI-3719: Implement iCarol CSV load, parse and field mapping

d3596de

Copilot finished work on behalf of stephenhand June 30, 2026 06:57

stephenhand commented Jun 30, 2026

View reviewed changes

Copilot started work on behalf of stephenhand June 30, 2026 07:21 View session

CHI-3719: Add unit tests for iCarol contact mapping; clearer S3 key p…

f6de203

…arse

Copilot finished work on behalf of stephenhand June 30, 2026 07:30

stephenhand commented Jun 30, 2026

View reviewed changes

Copilot started work on behalf of stephenhand June 30, 2026 09:01 View session

Copilot AI added 2 commits June 30, 2026 09:10

CHI-3719: Correct Aselo field names, derive duration from timestamps,…

3e38310

… map crisis callType

CHI-3719: Clarify intentional double space in 'Wrong Number' callType…

0960210

… label

Copilot finished work on behalf of stephenhand June 30, 2026 09:14

Copilot started work on behalf of stephenhand June 30, 2026 09:34 View session

Copilot AI added 2 commits June 30, 2026 09:42

CHI-3719: Look up Twilio worker SIDs from PhoneWorkerName to attribut…

3205c9c

…e imported contacts

CHI-3719: Clarify worker attribute parse warning, reuse resolveWorker…

1bfd4de

…Sid for missing-worker warning

Copilot finished work on behalf of stephenhand June 30, 2026 09:46

Change default channel for iCarol imports

add0a73

Copilot started work on behalf of stephenhand June 30, 2026 11:52 View session

Copilot finished work on behalf of stephenhand June 30, 2026 12:00

Uh oh!

Conversation

stephenhand commented Jun 29, 2026 • edited by Copilot AI Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Other Related Issues

Verification steps

AFTER YOU MERGE

Uh oh!

stephenhand commented Jun 30, 2026

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

stephenhand left a comment

Choose a reason for hiding this comment

Uh oh!

stephenhand Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

stephenhand left a comment

Choose a reason for hiding this comment

Uh oh!

stephenhand Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

stephenhand Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

stephenhand commented Jun 30, 2026

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

stephenhand commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

stephenhand commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stephenhand commented Jun 29, 2026 •

edited by Copilot AI

Loading

stephenhand commented Jun 30, 2026 •

edited

Loading