alumni_lookup

Phase 30 — Unified CRM Imports

Status: ✅ Complete — September 24, 2026 (30.1, 30.2, 30.3). Model handoff: the user chose to continue implementation on the planning model (“proceed”), September 24, 2026. Surface: Alumni Lookup (staff settings). Nothing touches cp/ or alumni_network/.

Completion Summary

Contacts, Educations, Areas of Study and Employment now run one flow at /settings/imports/:type: upload → scan → review → import → results on one status-driven batch page, with one poller, one controller (Settings::ImportsController), one state machine (ImportBatch) and a registry for the per-type differences (Imports::Type). Every type has a Commit confirm, a working Cancel, drag-and-drop upload, streamed manifests and heartbeat stale detection. The legacy Alumni CSV, Degrees CSV, Banner and /people importers are deleted.

Bugs fixed on the way: the scan page that needed a manual refresh (batches created pending; Contacts’ inline-script pollers under Turbo), a finished import redirecting staff who had left the page, Contacts results mixing batch #12 with #123, and the phone rake task and contact import undoing each other.

Deferred (BACKLOG → Unified CRM Imports): Giving/engagement imports onto the framework; the orphan report reading the legacy degrees table; merging the batch tables (won’t build).

Deploys: three pushes to staging (969f573a, 5ceaa6a7, 79a0b7a8); no migrations and no post-deploy rake tasks, so no launch guide. Suite at wrap: 6071 runs, 0 failures. Details per sub-phase in §9.

1. The Question This Phase Answers

Four imports do the same job: upload a CRM export, scan it in the background, review what will change, apply it in the background, and report the results. They are Contacts (BruinQuest), Educations, Areas of Study and Employment. They share a job pipeline but not a UI, so each one behaves slightly differently, and two of those differences are bugs:

• The scan page sometimes needs a manual refresh (Educations, Areas of Study, Employment). The upload creates the batch as pending and then enqueues the scan. The batch page only mounts its progress poller while the status is scanning or running, so if it renders before the worker picks up the job, it shows a bare “pending” and never updates. The same bug was fixed for the commit step in v1.0.64 (“Flip Status Before Enqueueing” in the debug skill), but the fix was never applied to upload.
• Contacts polls with inline <script> blocks on its scanning and processing pages. After a Turbo navigation, re-rendering one of those pages redeclares a top-level const, which throws, so polling never starts. And the polling loop outlives the page: after staff navigate away mid-import (which the page says is safe), the finished loop redirects them to the results page from wherever they are.

The rest are plain inconsistencies (§2). The phase makes all four imports one flow with one UI, moves them under /settings/imports/:type, and removes the legacy importers that the CRM imports replaced.

2. Current State

	Contacts	Educations / Areas of Study	Employment
Pages	4: scanning → preview → processing → results	1 status-driven page	1 status-driven page
Polling	Inline scripts (two different loops)	import-progress Stimulus controller	same
Batch created as	scanning ✅	pending ❌	pending ❌
Scan progress	None (spinner, “worker may not be running” after 2 min)	Live bar	Live bar
Commit confirm	None	None	None, even though commit deletes rows
Cancel	Scan + apply	Route exists, no UI links to it; apply never checks for a cancel	same
Stale auto-fail	Apply only (v1.0.92)	Heartbeat, 10 min	Heartbeat, 10 min
Manifest	ManifestStore, one JSON document	StreamingManifest (v1.0.92)	StreamingManifest
Results	Separate page with conflicts and CRM changes	Inline	Inline
History	“View All” page	Recent-batches partial	Its own recent-batches partial
URL	/settings/bruinquest/…	/settings/alumni/…	/settings/alumni/…

Contacts’ apply still parses its whole manifest (ManifestStore.fetch), which is the same out-of-memory risk that killed a 66K-row education import on staging in September 2026. It has succeeded so far only because its rows are smaller.

3. Decisions

Decision A — One page per batch, driven by status

Every import uses a single batch page that renders the section for its current state:

Scanning ──► Review ──► Importing ──► Results
   │            │           │
   └────────────┴───────────┴──► Failed / Cancelled

Contacts’ four pages collapse into this. Its preview content (ID conflicts, recency conflicts, field-change stats) becomes the Review section, and its results page (conflicts and CRM changes from the batch) becomes the Results section.

Decision B — /settings/imports/:type

Route	Purpose
GET /settings/imports	Hub: one card per type, with its last run and status
GET /settings/imports/:type	Upload form + that type’s batch history
POST /settings/imports/:type	Upload → create batch as scanning → enqueue scan → redirect
GET /settings/imports/:type/batches/:id	The single batch page
GET /settings/imports/:type/batches/:id/status	JSON for the poller
POST /settings/imports/:type/batches/:id/commit	Flip to running, enqueue apply
POST /settings/imports/:type/batches/:id/cancel	Cancel during scan, review or apply
GET /settings/imports/:type/batches/:id/issues.csv	Gap/issue CSV where the type has one

:type is one of contacts, educations, areas-of-study or employment. Batch ids are per table, and Educations and Areas of Study share education_import_batches, so a batch requested under the wrong type redirects to the right one rather than 404ing.

Old URLs redirect permanently (301) to their new equivalents, including batch pages. Before removing an old route, grep for persisted paths (StaffNotification#url, ActionItemsService). The debug skill has this happening twice on the 23.x renames.

Decision C — BruinQuest’s Conflicts and CRM Changes pages stay at /settings/bruinquest

They are CRM data tools that outlive any single import, not steps of the import flow. The Contacts Results section links to them, filtered to the batch.

Decision D — One “Imports” entry in the Settings sidebar

It replaces the four separate links and opens the hub. All four upload pages share one set of type tabs.

Decision E — One implementation, with a per-type adapter

One controller (Settings::ImportsController) serves every type. The differences between types live in a registry that maps each type slug to:

• its batch model
• its scan job and apply job
• its manifest store
• its Review and Results partials
• its commit-confirm copy

Shared pieces:

• Model concern: status helpers, stale, record_progress! (the heartbeat), and start/complete/fail transitions under one set of names.
• Status JSON: one shape for all types, including the stale auto-fail.
• Stimulus controller: one poller, import-progress. It stops on disconnect (so no loop outlives its page) and reloads on any state change.
• Partials: status header, progress panel, action bar (Commit / Cancel), results summary and history table.

The batch tables stay separate. Merging them is a migration with no user-facing benefit.

Decision F — Every import gets the same controls

• Commit confirm on every type, using data-turbo-confirm (not data-confirm, which nothing reads; see the debug skill). Employment’s confirm names the deletion count.
• Cancel is visible while scanning, in review, and while importing. Apply jobs check for a cancel at every chunk boundary (Contacts already does, the others don’t), so a cancel stops within one chunk. Rows committed in earlier chunks stay; a re-import scans them as no-ops.
• Contacts’ scan reports progress, so its bar moves and the heartbeat covers scanning too. That removes the started_at exception from AffinaquestImportBatch.stale.
• Contacts’ manifest moves to StreamingManifest, and its apply streams changes the same way the other three stream rows.

Decision G — Legacy importers are deleted, not hidden

Flow	Entry point	Replaced by
Alumni CSV	settings/alumni#upload_alumni, #import; Csv::AlumniImporter	Contacts
Degrees CSV	#upload_degrees, #import_degrees_preview/commit, #import_degrees	Educations
Banner (JH-0026c)	#upload_banner, #import_banner_preview/commit; Csv::BannerImporter	Educations
People	PeopleController, app/views/people/, resources :people	Contacts

The routes, actions, views, services and tests are all deleted, with no redirects: nothing should keep calling these. Three things are kept:

• The degrees table and the Degree model. Dropping them is already its own BACKLOG item, gated on the education data.
• alumni:deduplicate and phone:reformat, moved to a small service. Revised in the 30.3 interview: alumni:deduplicate is deleted (superseded by alumni:cleanup_duplicate_buids), and alumni:reformat_phones calls the shared PhoneNumber.format that the contact import writes with. See §9, 30.3.
• settings/alumni#orphaned, the orphan report. It is not an import. Revised: #index was deleted, and /settings/alumni redirects to the report.

4. Sub-Phases

#	Sub-phase	Ships
30.1	Shared framework + Educations, Areas of Study, Employment	Registry, controller, routes and redirects, concern, partials, poller; the scan-refresh fix; commit confirm; cancel UI and cancellation in the apply jobs; hub and sidebar
30.2	Contacts onto the framework	Single-page Contacts, inline scripts deleted, scan progress, StreamingManifest for Contacts, old /settings/bruinquest flow URLs redirected
30.3	Remove legacy importers	Decision G

Each sub-phase deploys on its own. None has a migration.

5. Acceptance Tests

Framework (run for every type)

• Upload creates the batch as scanning, and the first render of the batch page mounts the poller. This is the regression test for the manual-refresh bug.
• The status JSON has the same keys for all four types.
• Wrong type for a batch → redirect to the correct type (Educations ↔ Areas of Study).
• The Commit button carries data-turbo-confirm, asserted on the element. Employment’s text includes the deletion count.
• Cancel is visible while scanning, in review and while importing. A cancelled apply stops at the next chunk, and earlier chunks stay committed.
• A batch that’s stale by heartbeat is failed on poll, for every type and in both scanning and running.
• Every old URL returns a 301 to the new path, including batch pages. The legacy route helper names are gone (assert_not respond_to?).
• The poller stops polling when it disconnects. Covered by a JS test if the suite has one, otherwise by removing module-level state.

Contacts

• The whole flow happens on one page; no Contacts view contains a <script> tag.
• Scan writes progress.
• Apply never calls fetch on the manifest (asserted structurally, as in v1.0.92).
• Review shows ID conflicts, recency conflicts and change stats. Results lists the batch’s own conflicts and field changes and links to Conflicts and CRM Changes. Those two pages have no batch filter, so the links are unfiltered; the batch-specific rows are on the batch page itself.

Legacy removal

• Removed routes 404, and their helpers are undefined.
• alumni:deduplicate and phone:reformat still run through the new service. Revised: alumni:deduplicate is gone and alumni:reformat_phones uses PhoneNumber.format (test/tasks/alumni_rake_test.rb).
• bin/test passes with the importer tests deleted, and the architecture test baseline has not grown.

6. Activity Event Tracking

None. This is a staff-only admin tool. Each batch row is already its own audit record (imported_by, counts, timestamps, error).

7. Documentation Updates (required before wrap)

• docs/features/CSV_IMPORTERS.md: rewrite around the shared flow; delete the legacy importer sections
• docs/CHANGELOG.md
• docs/planning/BACKLOG.md: retire “Rename people_controller” (the controller is deleted); add “Giving/engagement imports onto the framework” (§8)
• .claude/skills/debug/SKILL.md: the inline-script poller bug
• config/faq.yml Skipped: it is the held portal surface’s help file and does not mention the import pages
• Phases index row and roadmap_controller.rb entry

8. Out of Scope

• Giving and the engagement-activity imports. They use a different synchronous preview/commit flow with a Tempfile manifest; the Phase 29 BACKLOG entries already track that. The registry is built so they can join later as another type.
• Merging the batch tables (Decision E).
• Dropping the degrees table (existing BACKLOG item).

9. What Was Implemented

30.1 — Shared framework ✅ (September 24, 2026)

Files

• app/services/imports/type.rb: the registry. Educations, Areas of Study and Employment; Contacts joins in 30.2.
• app/models/concerns/import_batch.rb: statuses, stale, record_progress! (heartbeat), fail!, cancel!, failed_in_db?, progress and duration display. EducationImportBatch and EmploymentImportBatch keep only their type-specific transitions.
• app/controllers/settings/imports_controller.rb, replacing 17 actions in Settings::AlumniController.
• app/views/settings/imports/: hub, type page, batch page, and the _tabs, _history, _progress and _status_badge partials. Per-type _instructions and _review live in educations/, areas_of_study/ and employment/; the review tables were moved from settings/alumni/, not rewritten.
• import_progress_controller.js: reloads on any status change (it used to reload only on leaving in-progress), reads counters generically, treats a non-JSON response as a retry, and holds all state on the instance.
• Csv::ChunkedCommit.each_row(stop_if:), threaded through the three importers’ commit as cancelled:.

Spec deviations

• Cancel is a POST, not the old DELETE. Nothing was deleted; the batch is kept as a record of the attempt.
• failed_in_db?, not “cancelled”. Jobs stop on any failed status, which also covers a stale sweep. A job that was swept while actually still alive stops at its next chunk instead of racing the sweep.
• The gap CSV is now called the “issues” CSV in the URL and the UI. “Gap” was internal vocabulary.
• The BruinQuest “Alumni Contacts” sidebar link stays until 30.2 moves Contacts. The hub already links to the other three.

Tests: imports_controller_test.rb (17 tests: framework behavior across every type, legacy redirects, retired helpers); the two old controller test files were ported to imports_educations_test.rb and imports_employment_test.rb; cancellation tests for ChunkedCommit, the apply job and the scan job. The regression test for the manual-refresh bug was sabotage-verified by reverting the upload to pending.

30.2 — Contacts onto the framework ✅ (September 24, 2026)

Files

• Imports::Type registry gains contacts (AffinaquestImportBatch, AffinaquestScanJob, AffinaquestApplyJob, ManifestStore), plus counter_label (“ID conflicts”) and an optional _results partial rendered once a batch finishes.
• app/views/settings/imports/contacts/: _instructions (format and links to Conflicts and CRM Changes), _review (ported from preview.html.erb, reading the streamed changes in one capped pass) and _results (ported from show_batch.html.erb).
• AffinaquestImportBatch includes ImportBatch; field_changes moved onto the model.
• ManifestStore now uses StreamingManifest (changes as rows, everything else as the header summary) and gains header/rows; StreamingManifest accepts a legacy_rows_key, so a pre-deploy changes-keyed blob still reads.
• Csv::AffinaquestContactImporter#scan(progress:); #apply_manifest_to_batch(batch:, changes:, summary:) streams.
• Deleted: six settings/affinaquest views, ten Settings::AffinaquestController actions, AffinaquestImportJob and #import! (unused).

Spec deviations

• Drag-and-drop upload on every type. Contacts had it and the others did not; the shared form took the richer version rather than dropping it.
• The sidebar’s “Alumni Contacts” entry became “CRM Changes & Conflicts”, pointing at /settings/bruinquest/crm_changes, rather than being removed: Decision C keeps those pages, and they needed an entry of their own.
• Contacts has no issues CSV. Its table has no gap_* columns; ImportBatch#issues_csv? guards the link, and the batch page hides “Issue rows” for types without the column.
• Removed the “worker may not be running” warning the Contacts scanning page showed after two minutes. The scan now reports progress, so a silent worker is caught by the stale heartbeat and failed after 10 minutes with a message, rather than warned about.

Bugs found while porting

• The old results page’s notes LIKE '%batch #12%' also matched batch #123. Now '%batch #12)%', pinned by a model test.
• The old results page ran Alumni.find_by once per field change (up to 100 queries).

Tests: imports_contacts_test.rb (10, including a full upload → scan → commit round trip on one URL, the review reading without fetch, and every old URL’s redirect). The framework tests in imports_controller_test.rb now run across all four types. Tests for deleted actions were removed from affinaquest_controller_test.rb; model and ManifestStore tests updated and extended.

30.3 — Remove legacy importers ✅ (September 24, 2026)

Deleted: Csv::AlumniImporter, Csv::BannerImporter, PeopleController and its views, nine Settings::AlumniController actions (index, the three upload pages, import, the three degree import actions and the two banner actions) and their six views, the routes, lib/tasks/phone_formatting.rake, and the importers’ test files.

Spec deviations (approved in the 30.3 interview, September 24, 2026)

• alumni:deduplicate was deleted, not preserved. It duplicated alumni:cleanup_duplicate_buids, which is the one the docs and the debug skill point to and is the safer of the two (dry run by default; keeps the row with more data rather than the newest).
• alumni:reformat_phones now shares the contact import’s formatter. The old task’s parse_and_format_phone disagreed with AffinaquestContactImporter#normalize_phone for non-standard lengths (original text vs digits only), so each run of one undid the other. Both now call the shared PhoneNumber.format, and a test pins that they agree. reformat_phones and reformat_phones_batch are one task with an optional batch size.
• /settings/alumni redirects to the orphan report instead of staying as a page holding a single link. The sidebar’s “Archived Imports → Alumni (Banner)” entry is now “Alumni Without Degrees” under Data Imports.

Found, not fixed (BACKLOG): the orphan report reads the legacy degrees table.

Tests: legacy_importers_removed_test.rb (the endpoints 404, helpers and classes gone, the redirect, the sidebar), phone_number_test.rb, test/tasks/alumni_rake_test.rb.