docuseal/docs/architecture/rollback-strategy.md

Rollback Strategy for Institution Management

Overview

This document outlines the rollback procedure for the institution management migrations (20250103000001-20250103000006).

Migration Sequence

1. 20250103000001 - Add institution_id to account_access (nullable)
2. 20250103000002 - Create institutions table
3. 20250103000003 - Create cohort_admin_invitations table
4. 20250103000004 - Update account_access roles
5. 20250103000005 - Backfill institution data (makes institution_id non-nullable)
6. 20250103000006 - Create security_events table

Rollback Procedure

Step 1: Test Rollback on Staging/Development

# Rollback last migration
bin/rails db:rollback STEP=1

# Rollback all institution migrations
bin/rails db:rollback STEP=6

Step 2: Production Rollback (If Needed)

⚠️ CRITICAL: Always backup database before rollback

# 1. Create database backup
pg_dump -Fc docuseal_production > docuseal_backup_$(date +%Y%m%d_%H%M%S).dump

# 2. Check current migration status
bin/rails db:migrate:status

# 3. Rollback in reverse order
bin/rails db:rollback STEP=1  # Rollback security_events
bin/rails db:rollback STEP=1  # Rollback backfill
bin/rails db:rollback STEP=1  # Rollback account_access roles
bin/rails db:rollback STEP=1  # Rollback cohort_admin_invitations
bin/rails db:rollback STEP=1  # Rollback institutions
bin/rails db:rollback STEP=1  # Rollback institution_id addition

# 4. Verify rollback
bin/rails db:migrate:status

Step 3: Data Safety Verification

After rollback, verify:

1. Existing data intact:

   SELECT COUNT(*) FROM users;
   SELECT COUNT(*) FROM accounts;
   SELECT COUNT(*) FROM account_accesses;
   SELECT COUNT(*) FROM templates;
   SELECT COUNT(*) FROM submissions;
   

2. No orphaned records:

   -- Check for orphaned records
   SELECT * FROM account_accesses WHERE account_id NOT IN (SELECT id FROM accounts);
   SELECT * FROM users WHERE account_id NOT IN (SELECT id FROM accounts);
   

3. Existing functionality works:

   • User login
   • Template creation
   • Submission workflows
   • API access

Rollback Risks and Mitigations

Risk 1: Data Loss

Impact: High Mitigation:

• Always backup before rollback
• Test rollback on staging first
• Verify data integrity after rollback

Risk 2: Downtime

Impact: Medium Mitigation:

• Schedule rollback during maintenance window
• Have rollback plan ready
• Test procedure beforehand

Risk 3: Application Errors

Impact: High Mitigation:

• Keep application version compatible with database schema
• Have emergency rollback to previous app version ready
• Monitor error logs during rollback

Emergency Rollback

If critical issues arise during deployment:

1. Immediate rollback:

   git revert HEAD
   bin/rails db:rollback STEP=6
   

2. Restore from backup if needed:

   pg_restore -d docuseal_production docuseal_backup_YYYYMMDD_HHMMSS.dump
   

3. Notify stakeholders of rollback and reason

Post-Rollback Verification

After rollback, verify these critical paths:

• User authentication works
• Existing templates accessible
• Submissions can be created
• API endpoints return correct data
• No database constraint violations
• Email notifications work
• Webhook delivery works

Rollback Decision Matrix

Rollback if:

• Data corruption detected
• Critical security vulnerabilities found
• Major performance degradation (>50%)
• Application crashes on startup
• Cannot fix issues within 2 hours

Do NOT rollback if:

• Minor bugs that can be hotfixed
• Performance issues within acceptable range (<10%)
• UI/UX issues only
• Non-critical feature failures

Contact Information

Emergency Contacts:

• Database Administrator: [To be filled]
• DevOps Engineer: [To be filled]
• Security Team: [To be filled]

Rollback Window: [To be scheduled] Estimated Downtime: 15-30 minutes