alumni_lookup

Deployment Pipeline

Last Updated: December 2, 2025
Purpose: Document deployment workflows, smoke tests, rollback procedures, and migration handling.

Table of Contents

  1. Deployment Overview
  2. Deployment Flows
  3. Database Migrations
  4. Smoke Test Checklist
  5. Staging Testing Workflow
  6. Database Backups
  7. Rollback Procedures
  8. Troubleshooting

Deployment Overview

Pipeline Architecture

┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Local      │────▶│    GitHub    │────▶│   Staging    │────▶│  Production  │
│   Dev        │     │    (main)    │     │   (Heroku)   │     │   (Heroku)   │
└──────────────┘     └──────────────┘     └──────────────┘     └──────────────┘
       │                    │                    │                    │
       │                    │                    │                    │
   bin/dev              CI tests           Smoke tests          Verification
   Manual test          Auto-deploy        Manual QA            Monitoring

Environment URLs

Environment Lookup Portal Champions Portal
Development https://lookup.bualum.dev:3000 https://champions.bualum.dev:3000
Staging https://staging-lookup.bualum.co https://staging-champions.bualum.co
Production https://lookup.bualum.co https://champions.bualum.co

Deployment Flows

Flow 1: Feature Development → Staging

# 1. Create feature branch
git checkout -b feature/my-feature

# 2. Develop and test locally
bin/dev
rails test

# 3. Push and create PR
git push -u origin feature/my-feature
# Create PR on GitHub

# 4. CI runs automatically
# Wait for green checkmark

# 5. Merge to main
# Staging deploy triggers automatically

# 6. Verify on staging
# Run smoke tests (see checklist below)

Flow 2: Staging → Production

Option A: Tagged Release (Recommended)

# 1. Ensure staging is verified
# Run full smoke test checklist

# 2. Create release tag
git checkout main
git pull origin main
git tag -a v1.2.3 -m "Release v1.2.3: Description of changes"
git push origin v1.2.3

# 3. Production deploy triggers automatically
# Or requires approval (if configured)

# 4. Verify production
# Run smoke tests on production URLs

Option B: Manual Deploy

# 1. Trigger workflow manually on GitHub
# Actions → Deploy to Production → Run workflow

# 2. Approve (if required)
# 3. Monitor deploy logs

Flow 3: Hotfix

# 1. Create hotfix branch from main
git checkout main
git pull origin main
git checkout -b hotfix/critical-fix

# 2. Make minimal fix
# Write test for the bug first if possible
rails test

# 3. Push and merge (expedited review)
git push -u origin hotfix/critical-fix
# Create PR, get quick review, merge

# 4. Immediately tag for production
git checkout main
git pull origin main
git tag -a v1.2.4 -m "Hotfix: Critical bug fix"
git push origin v1.2.4

# 5. Verify production immediately

Database Migrations

Migration Strategy by Environment

Environment When Migrations Run How
Development Manually rails db:migrate
Test Automatically Schema loaded on test run
Staging On deploy heroku run rails db:migrate
Production On deploy heroku run rails db:migrate

Pre-Deployment Migration Checks

Before deploying migrations:

# 1. Check migration is reversible
rails db:migrate
rails db:rollback
rails db:migrate

# 2. Check migration is safe
# Review for:
# - [ ] Removing columns in use
# - [ ] Changing column types with data
# - [ ] Adding NOT NULL without default
# - [ ] Long-running operations on large tables

Reversible vs Irreversible Migrations

Migration Type Reversible? Example Precaution
Add column ✅ Yes add_column :alumni, :foo, :string None needed
Remove column ✅ Yes (with down) remove_column :alumni, :foo Backup data first
Add index ✅ Yes add_index :alumni, :buid Use algorithm: :concurrently for large tables
Change column type ⚠️ Careful change_column :alumni, :buid, :text May lose data
Drop table ❌ No drop_table :old_table Full backup required

Zero-Downtime Migration Pattern

For production changes to critical tables:

# Step 1: Add new column (deploy 1)
class AddNewFieldToAlumni < ActiveRecord::Migration[7.1]
  def change
    add_column :alumni, :new_field, :string
  end
end

# Step 2: Backfill data (deploy 2 or rake task)
# Alumni.find_each { |a| a.update(new_field: derive_value(a)) }

# Step 3: Switch code to use new field (deploy 3)

# Step 4: Remove old column (deploy 4)
class RemoveOldFieldFromAlumni < ActiveRecord::Migration[7.1]
  def change
    remove_column :alumni, :old_field
  end
end

Champion Portal Setup Tasks

Geographic Data (Regions, Districts, ZIP Codes)

The Champion Portal requires geographic seed data for district assignment. This data comes from the CSV file db/seeds/data/zip_district_region.csv.

Initial Setup (First Deploy)

After deploying the Champion Portal foundation migrations, run the seed task:

# Staging
heroku run rails champion_portal:seed_geographic_data --app alumni-lookup-staging

# Production
heroku run rails champion_portal:seed_geographic_data --app alumni-lookup

Verify Data Loaded

heroku run rails champion_portal:stats --app alumni-lookup-staging

Expected output:

=== Champion Portal Geographic Data ===
Regions: 7
Districts: 820 (8 highlighted)
ZIP codes: 39305

Re-seeding (Destructive)

If you need to update geographic data:

  1. Ensure no champions have district assignments (or they’ll be orphaned)
  2. Run the reseed task:
heroku run rails champion_portal:reseed_geographic_data --app alumni-lookup-staging

CSV File Location

The seed data CSV is committed to the repo at:

If the CSV is updated:

  1. Replace the CSV file locally
  2. Test with rails champion_portal:reseed_geographic_data
  3. Commit and deploy
  4. Run seed task on staging/production

Smoke Test Checklist

Quick Smoke Test (Post-Deploy)

Run immediately after each deploy:

Lookup Portal (Internal)

Champions Portal (External)

Full Smoke Test (Before Production)

Run before tagging production release:

Authentication & Roles

Core Features

Feature Test Expected Result
Alumni Search Search “Smith” Results with “Smith” in name
Alumni Profile View a profile Photo, degrees, engagement display
Engagement Stats Click stats tab Analytics load, no timeout
Top Engaged View top engaged Ranked list appears
Champion Signups View signups list Table loads with signups

Data Operations

Champion Signup Flow

Staging Testing Workflow

Overview

Each production release cycle uses a GitHub Issue as the QA checklist. This provides:

Workflow Diagram

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Push to        │────▶│  Agent creates  │────▶│  QA tests on    │────▶│  Tag production │
│  Staging        │     │  GitHub Issue   │     │  Staging        │     │  & close Issue  │
└─────────────────┘     └─────────────────┘     └─────────────────┘     └─────────────────┘
                              │                        │
                              │                        │
                        Auto-generated            Check off items
                        QA checklist              in GitHub Issue

Step 1: Create QA Issue (After Push to Staging)

When code is pushed to staging, ask the agent to generate a QA issue:

Prompt to use:

Generate a QA checklist for staging. Create a GitHub Issue with click-through 
testing instructions for all changes since the last production tag.

The agent will:

  1. Identify the last production tag
  2. Analyze commits and file changes since that tag
  3. Generate a QA checklist based on what changed
  4. Create a GitHub Issue titled: QA Checklist: v1.0.x → v1.0.y

Note: If the agent cannot create issues directly, it will output the issue content for you to paste into GitHub → Issues → New Issue.

Step 2: QA Issue Template

The generated issue should follow this format:

## QA Checklist: v[LAST_TAG] → v[NEXT_TAG]

**Staging URL:** https://staging-lookup.bualum.co  
**Champions Staging:** https://staging-champions.bualum.co  
**Last Production Tag:** v1.0.x  
**Commits Since Tag:** [count]  

---

### Pre-Flight Checks
- [ ] Staging deployment completed successfully
- [ ] No console errors on page load
- [ ] Database migrations ran (if any)

---

### Quick Smoke Test
- [ ] Login works on Lookup Portal
- [ ] Alumni search returns results
- [ ] Engagement Stats page loads without timeout

---

### Feature-Specific Tests

#### [Feature/Fix Name]
**Files Changed:** `app/controllers/example.rb`, `app/views/example/show.html.erb`

**Test Steps:**
- [ ] Navigate to [URL]
- [ ] Verify [expected behavior]
- [ ] Test edge case: [description]

[Repeat for each feature/fix]

---

### Sign-Off

| Role | Name | Date | Approved |
|------|------|------|----------|
| QA Tester | | | ☐ |
| Developer | | | ☐ |

---

### Ready for Production?
- [ ] **All tests passed**
- [ ] **No blocking issues found**
- [ ] **Approved for production tag**

**Blocking Issues:** (list any issues that must be fixed before production)

Step 3: Execute QA Testing

  1. Open the GitHub Issue in your browser
  2. Open staging in another tab
  3. Work through the checklist, checking boxes as you go
  4. Add comments to the issue for any questions or findings
  5. If issues found:
    • Add them as comments with reproduction steps
    • Create separate bug issues if needed
    • Link blocking issues in the “Blocking Issues” section

Step 4: Production Release

When all checklist items are complete:

  1. Verify sign-off section is complete
  2. Create the production tag: 
    git checkout main
git pull origin main
git tag -a v1.0.x -m "Release v1.0.x: Description"
git push origin v1.0.x
  3. Close the QA Issue with a comment: 
    Released as v1.0.x - closing this QA cycle.
  4. The next staging push starts a new QA cycle with a new issue

Step 5: Starting the Next Cycle

After production deployment, the cycle resets:

QA Tester Quick Reference

Your Workflow

  1. Receive notification that staging has new changes
  2. Find the QA Issue in GitHub (search: is:issue is:open QA Checklist)
  3. Open staging URLs in browser
  4. Execute each test step, checking boxes as you complete them
  5. Document any issues in the issue comments
  6. Complete sign-off when all tests pass
  7. Notify developer that QA is complete

Testing URLs

Environment Lookup Portal Champions Portal
Staging https://staging-lookup.bualum.co https://staging-champions.bualum.co
Production https://lookup.bualum.co https://champions.bualum.co

If You Find a Bug

  1. Don’t just uncheck the box - document it!
  2. Add a comment to the QA issue with:
    • Steps to reproduce
    • Expected vs actual behavior
    • Screenshot if helpful
    • Severity: Blocker / Major / Minor
  3. If it’s a blocker, add it to the “Blocking Issues” section
  4. Developer will fix and notify you to re-test

Test Credentials

Contact the development team for staging test credentials.

Agent Commands for QA Workflow

Generate QA Checklist

Generate a QA checklist for staging based on changes since the last production tag.
Include click-through testing instructions for each change.

Check What Changed

What files have changed since the last production tag? Summarize the features
and fixes that need testing.

Create Release Notes

Generate release notes for the upcoming production tag based on all changes
since the last tag.

Database Backups

Backup Configuration

Environment Plan Auto Backup Retention
Production Essential 0 ✅ Daily 7 days
Staging Mini ❌ Manual only N/A

Check Backup Status

# List backups
heroku pg:backups --app alumni-lookup

# View schedule
heroku pg:backups:schedules --app alumni-lookup

Manual Backup (Before Risky Changes)

# Capture backup before major migration
heroku pg:backups:capture --app alumni-lookup

Restore from Backup

# Restore most recent backup
heroku pg:backups:restore --app alumni-lookup

# Restore specific backup (get ID from pg:backups list)
heroku pg:backups:restore b001 DATABASE_URL --app alumni-lookup

Download Backup Locally

# Get download URL (expires after 1 hour)
heroku pg:backups:url --app alumni-lookup

# Download and restore to local dev
curl -o latest.dump "$(heroku pg:backups:url --app alumni-lookup)"
pg_restore --verbose --clean --no-acl --no-owner -d alumni_lookup_development latest.dump

Rollback Procedures

Heroku Release Rollback

If a deploy breaks production:

# 1. List recent releases
heroku releases --app alumni-lookup

# Output:
# v42 Deploy abc1234 user@example.com 2025-12-02T10:00:00Z
# v41 Deploy def5678 user@example.com 2025-12-01T15:00:00Z
# ...

# 2. Rollback to previous release
heroku rollback v41 --app alumni-lookup

# 3. Verify site is working
curl -f https://lookup.bualum.co/users/sign_in

Git Revert + Redeploy

For a cleaner history:

# 1. Identify bad commit
git log --oneline -10

# 2. Revert the commit
git revert <commit-sha>

# 3. Push to main
git push origin main

# 4. Let CI/CD redeploy
# Or manually trigger if needed

Database Rollback

If migration caused issues:

# 1. Rollback migration on production
heroku run rails db:rollback --app alumni-lookup

# 2. If multiple migrations, rollback multiple steps
heroku run rails db:rollback STEP=2 --app alumni-lookup

# 3. Then roll back the release
heroku rollback v41 --app alumni-lookup

⚠️ Warning: Some migrations are not reversible. Check migration file before attempting rollback.

Rollback Decision Tree

┌────────────────────────────────────────────────────────────┐
│                  Site is broken after deploy               │
└────────────────────────────────────────────────────────────┘
                              │
                              ▼
                    ┌─────────────────┐
                    │ Was there a DB  │
                    │   migration?    │
                    └─────────────────┘
                     │              │
                   No               Yes
                     │              │
                     ▼              ▼
           ┌─────────────┐  ┌─────────────────┐
           │   Rollback  │  │  Is migration   │
           │   release   │  │   reversible?   │
           └─────────────┘  └─────────────────┘
                             │              │
                           Yes              No
                             │              │
                             ▼              ▼
                   ┌─────────────┐  ┌─────────────┐
                   │  Rollback   │  │   Fix       │
                   │  migration  │  │   forward   │
                   │  then       │  │   (hotfix)  │
                   │  release    │  └─────────────┘
                   └─────────────┘

Troubleshooting

Common Deploy Issues

1. Build Fails

# Check build logs
heroku builds:info --app alumni-lookup

# Common causes:
# - Missing gem in Gemfile
# - Node package issues
# - Asset compilation failure

Fix: Check error message, fix locally, push again.

2. Migrations Fail

# Check migration status
heroku run rails db:migrate:status --app alumni-lookup

# Run pending migrations manually
heroku run rails db:migrate --app alumni-lookup

Fix: If migration is bad, rollback migration first, then release.

3. App Crashes on Start

# Check logs
heroku logs --tail --app alumni-lookup

# Common causes:
# - Missing environment variable
# - Bad initializer code
# - Memory issues

Fix: Check for missing env vars, rollback if needed.

4. 500 Errors After Deploy

# Check logs for errors
heroku logs --tail --app alumni-lookup | grep "Error\|Exception"

# Check specific request
heroku logs --tail --app alumni-lookup --source heroku --dyno web

Pre-Flight Checklist

Before deploying, verify:

Check Command Expected
Tests pass rails test 0 failures
No pending migrations locally rails db:migrate:status All “up”
Git is clean git status No uncommitted changes
On correct branch git branch * main
Remote is up to date git pull origin main Already up to date

Post-Deploy Verification

Check Command/Action Expected
App responds curl https://lookup.bualum.co/users/sign_in 200 OK
Logs clean heroku logs --tail No errors
Migrations complete heroku run rails db:migrate:status All “up”
Key features work Manual smoke test All pass

Monitoring & Alerts

Current Setup

Aspect Tool Status
Logs Heroku Logs ✅ Available
Metrics Heroku Metrics ✅ Basic
Alerts None ❌ Not configured
Tool Purpose Cost
Heroku Logentries Log search, alerts Free tier
Uptime Robot Uptime monitoring Free
Sentry/Rollbar Error tracking Free tier

Key Metrics to Monitor

Metric Warning Critical
Response time > 1000ms > 3000ms
Error rate > 1% > 5%
Memory usage > 80% > 95%
Dyno restarts > 1/hour > 3/hour