alumni_lookup

Security & Rate Limiting

Last Updated: January 31, 2026
Version: 1.0.31+

This document covers the security measures protecting the Alumni Lookup and Alumni Portal applications, including rate limiting, bot blocking, and incident troubleshooting.

---

Table of Contents

1. Overview
2. Rack::Attack Rate Limiting
3. Request Logging Middleware
4. Security Rake Tasks
5. Incident Troubleshooting
6. Log Analysis Commands
7. Common Attack Patterns
8. Configuration Reference

---

Overview

The application uses a layered security approach:

Layer	Component	Purpose
Rate Limiting	Rack::Attack	Throttle abusive requests, block known bad actors
Request Logging	RequestLogger Middleware	Log slow requests, detect suspicious patterns
Monitoring	Security Rake Tasks	Quick status checks and log analysis tips

Middleware Stack Order

Rack::Runtime              # Request timing
Middleware::RequestLogger  # Custom logging (early for visibility)
...
Rack::Attack               # Rate limiting (before routing)

---

Rack::Attack Rate Limiting

File: config/initializers/rack_attack.rb

Safelists (Always Allow)

Rule	Description
allow-localhost	Requests from 127.0.0.1 (development)
allow-health-checks	/up endpoint for Heroku health checks

Blocklists (Always Block)

Rule	Trigger	Response
block-wordpress-scanners	Paths like /wp-admin, /xmlrpc.php, /.env	403 Forbidden
block-screenshot-services	Query string contains screenshotCacheBust	403 Forbidden
fail2ban/throttle-abuse	10+ throttle violations in 10 minutes	403 Forbidden (1 hour)

Throttles (Rate Limits)

Rule	Limit	Scope	Applies To
req/ip	300 per 5 min	IP address	All requests
assets/ip	500 per 1 min	IP address	/assets/* requests
logins/ip	5 per 20 sec	IP address	Login attempts
logins/email	5 per 20 sec	Email address	Login attempts
champion-logins/ip	5 per 20 sec	IP address	Alumni Portal login
password-resets/ip	3 per 30 min	IP address	Password reset requests
api/ip	60 per 1 min	IP address	/api/* endpoints

Response Codes

Code	Meaning	Rack::Attack Action
403	Forbidden	Blocklist match
429	Too Many Requests	Throttle exceeded

Logging

All Rack::Attack events are logged with the [Rack::Attack] tag:

[Rack::Attack] Blocked wordpress-scanner: 192.168.1.1
[Rack::Attack] Throttled req/ip: 192.168.1.1
[Rack::Attack] Fail2ban triggered for: 192.168.1.1

---

Request Logging Middleware

File: lib/middleware/request_logger.rb

What Gets Logged

Pattern	Tag	Condition
Slow requests	[SLOW REQUEST]	Duration > 5000ms (configurable)
Screenshot services	[SCREENSHOT SERVICE]	Query contains screenshotCacheBust
Vulnerability scanners	[SCANNER DETECTED]	Path matches known scanner patterns
Slow assets	[SLOW ASSET]	Asset requests > 3000ms (when enabled)

Environment Variables

Variable	Default	Description
SLOW_REQUEST_THRESHOLD_MS	5000	Threshold for slow request logging (milliseconds)
LOG_ASSET_REQUESTS	false	Enable logging of slow asset requests

Example Log Output

[SLOW REQUEST] GET /alumni/search took 7234ms from 192.168.1.1 (Mozilla/5.0...)
[SCREENSHOT SERVICE] GET /events/123?screenshotCacheBust=abc from 8.36.86.54
[SCANNER DETECTED] GET /wp-admin/admin-ajax.php from 45.33.32.156

---

Security Rake Tasks

File: lib/tasks/security.rake

Check Configuration Status

# Local
bin/rake security:status

# Production
heroku run bin/rake security:status --app alumni-lookup

Output:

🛡️  SECURITY STATUS
====================
📋 Rack::Attack Configuration:
   ✅ Rack::Attack is loaded
   • Safelists: 2
   • Blocklists: 3
   • Throttles: 7

🔧 Environment Configuration:
   • SLOW_REQUEST_THRESHOLD_MS: 5000 (default)
   • LOG_ASSET_REQUESTS: false (default)

💾 Cache Backend:
   • Rails.cache: ActiveSupport::Cache::MemoryStore

List All Rules

bin/rake security:rules

Output:

📜 RACK::ATTACK RULES
=====================
🟢 SAFELISTS (Always Allow):
   • allow-localhost
   • allow-health-checks

🔴 BLOCKLISTS (Always Block):
   • block-wordpress-scanners
   • block-screenshot-services
   • fail2ban/throttle-abuse

🟡 THROTTLES (Rate Limits):
   • req/ip: 300 requests per 300 seconds
   • assets/ip: 500 requests per 60 seconds
   • logins/ip: 5 requests per 20 seconds
   ...

Get Log Analysis Tips

bin/rake security:analyze_tips

Outputs ready-to-use Heroku CLI commands for log analysis.

---

Incident Troubleshooting

Step 1: Identify the Problem

Symptoms → Likely Cause:

Symptom	Possible Cause
H27 errors (client disconnect)	Slow asset delivery, possibly from bot/crawler
Spike in 429 responses	Rate limiting triggered, legitimate or attack
Spike in 403 responses	Blocked requests (scanners, screenshot services)
Memory/CPU spike	Heavy traffic, possibly DDoS

Step 2: Check Recent Blocks

# Find Rack::Attack events in last 1000 lines
heroku logs -n 1000 --app alumni-lookup | grep "\[Rack::Attack\]"

# Count blocks by IP
heroku logs -n 2000 --app alumni-lookup | grep "\[Rack::Attack\]" | \
  awk '{print $NF}' | sort | uniq -c | sort -rn | head -20

Step 3: Check Slow Requests

# Find slow requests
heroku logs -n 2000 --app alumni-lookup | grep "\[SLOW REQUEST\]"

# Find requests taking >10 seconds
heroku logs -n 3000 --app alumni-lookup | grep -E "service=[0-9]{5,}ms"

Step 4: Identify Suspicious Patterns

# Screenshot service requests
heroku logs -n 1000 --app alumni-lookup | grep "screenshotCacheBust"

# Scanner detection
heroku logs -n 1000 --app alumni-lookup | grep "\[SCANNER DETECTED\]"

# H27 errors (client disconnected)
heroku logs -n 2000 --app alumni-lookup | grep "code=H27"

Step 5: Time-Window Analysis

# Search specific time window (replace timestamp)
heroku logs -n 5000 --app alumni-lookup | grep "2026-01-31T15:0"

# Real-time monitoring
heroku logs --tail --app alumni-lookup | grep -E "(Rack::Attack|SLOW REQUEST|H27)"

---

Log Analysis Commands

Quick Reference

# === RACK::ATTACK EVENTS ===
# All blocks/throttles
heroku logs -n 1000 --app alumni-lookup | grep "\[Rack::Attack\]"

# Count by IP
heroku logs -n 2000 --app alumni-lookup | grep "\[Rack::Attack\]" | \
  awk '{print $NF}' | sort | uniq -c | sort -rn

# === SLOW REQUESTS ===
# Custom slow request logging
heroku logs -n 2000 --app alumni-lookup | grep "\[SLOW REQUEST\]"

# Service time >10 seconds
heroku logs -n 3000 --app alumni-lookup | grep -E "service=[0-9]{5,}ms"

# === SPECIFIC THREATS ===
# Screenshot services
heroku logs -n 1000 --app alumni-lookup | grep "screenshotCacheBust"

# Vulnerability scanners
heroku logs -n 1000 --app alumni-lookup | grep "\[SCANNER DETECTED\]"

# WordPress scanner attempts
heroku logs -n 1000 --app alumni-lookup | grep -E "(wp-admin|xmlrpc|wp-login)"

# === ERRORS ===
# H27 (client disconnect)
heroku logs -n 2000 --app alumni-lookup | grep "code=H27"

# H12 (request timeout)
heroku logs -n 2000 --app alumni-lookup | grep "code=H12"

# R14 (memory quota exceeded)
heroku logs -n 2000 --app alumni-lookup | grep "code=R14"

# === REAL-TIME MONITORING ===
# Watch for security events
heroku logs --tail --app alumni-lookup | grep -E "(Rack::Attack|SLOW|SCANNER|H27|H12)"

---

Common Attack Patterns

1. WordPress Scanners

Pattern: Automated bots scanning for WordPress vulnerabilities
Paths: /wp-admin, /wp-login.php, /xmlrpc.php, /.env, /config.php
Response: Blocked (403) via block-wordpress-scanners

2. Screenshot Services

Pattern: Services that render pages for screenshots/thumbnails
Signature: ?screenshotCacheBust= query parameter
Problem: Can overwhelm asset delivery, causing H27 errors
Response: Blocked (403) via block-screenshot-services

Real Incident (Jan 31, 2026): IP 8.36.86.54 with User-Agent containing “HeadlessChrome” made rapid requests with screenshotCacheBust parameter, causing assets to take 10-12 seconds to serve and triggering H27 errors.

3. Credential Stuffing

Pattern: Rapid login attempts with different credentials
Protection: logins/ip and logins/email throttles (5 per 20 seconds)
Escalation: Fail2ban triggers after 10 throttle violations

4. API Abuse

Pattern: Excessive API calls (scraping, automation)
Protection: api/ip throttle (60 per minute)

---

Configuration Reference

Files

File	Purpose
config/initializers/rack_attack.rb	Rate limiting rules, blocklists, throttles
lib/middleware/request_logger.rb	Custom request logging middleware
lib/tasks/security.rake	Security monitoring rake tasks

Adding a New Blocklist

Edit config/initializers/rack_attack.rb:

# Block requests matching a pattern
Rack.Attack.blocklist("block-my-pattern") do |req|
  req.path.include?("/suspicious-path") ||
    req.query_string.include?("bad-param")
end

Adding a New Throttle

# Throttle specific endpoint
Rack::Attack.throttle("my-endpoint/ip", limit: 10, period: 60) do |req|
  if req.path == "/my-endpoint"
    req.ip
  end
end

Adjusting Thresholds

Rate limits: Edit the limit and period values in rack_attack.rb

Slow request threshold: Set SLOW_REQUEST_THRESHOLD_MS environment variable

# Heroku
heroku config:set SLOW_REQUEST_THRESHOLD_MS=3000 --app alumni-lookup

# Local (.env)
SLOW_REQUEST_THRESHOLD_MS=3000

Testing Locally

# Verify Rack::Attack is loaded
bin/rake security:status

# View all rules
bin/rake security:rules

# Test a throttle (requires multiple rapid requests)
for i in {1..10}; do curl -s http://localhost:3000/users/sign_in > /dev/null; done
# Check logs for throttle messages

---

Related Documentation

• DEPLOYMENT_PIPELINE.md — CI/CD and deployment process
• ENVIRONMENTS_CURRENT.md — Environment configuration
• ../development/ARCHITECTURE.md — Application architecture

---

Changelog

Date	Version	Changes
2026-01-31	1.0.31	Initial security implementation