Operations and recovery
Environments, monitoring, backups and tested recovery. Targets: recovery point objective 15 minutes, recovery time objective 4 hours, availability 99.9% for department views.
Environments
| Environment | Purpose | Compute | Data | Encryption | Network | Sign in |
|---|---|---|---|---|---|---|
| Development | Build and unit test pipelines | X-Small, auto suspend 1 min | Synthetic and masked samples | Encrypted at rest (platform keys) | Private endpoint | City SSO with MFA |
| Test | Integration, user testing, restore tests | Small, auto suspend 5 min | Masked copy refreshed weekly | Encrypted at rest (platform keys) | Private endpoint | City SSO with MFA |
| Production | Department views and reporting | Per workload class, auto suspend 5 to 10 min | Live | Encrypted at rest (City managed keys) | Private endpoint, no public access | City SSO with MFA, conditional access |
Run duration and rows (monitoring)
Recovery tests
| Test | Date | Scenario | RPO target / actual | RTO target / actual | Result | Witness |
|---|---|---|---|---|---|---|
| DR-03 | 2026-10-22 | Restore production snapshot to paired region; run reconciliation suite | 15 / 11 min | 240 / 104 min | Passed | IT Infrastructure Manager |
| DR-02 | 2026-07-16 | Point in time restore of the cleansed layer after a bad load | 15 / 6 min | 120 / 38 min | Passed | Kevin Tran |
| DR-01 | 2026-04-09 | Failover of the test environment to the paired region | 60 / 22 min | 480 / 212 min | Passed with findings | IT Infrastructure Manager |
Backups: nightly snapshot after each run, copied to the paired region, plus continuous log backup for point in time restore (7 days). One snapshot a week is restored and verified.
Backups
| Backup | Taken | Environment | Kind | Size (GB) | Location | Status | Restore verified |
|---|---|---|---|---|---|---|---|
| BK-20261111 | 11/11/2026 02:25 | Production | Nightly snapshot | 188.0 | South Central US (paired region) | Completed | |
| BK-20261110 | 11/10/2026 02:25 | Production | Nightly snapshot | 187.4 | South Central US (paired region) | Completed | |
| BK-20261109 | 11/09/2026 02:25 | Production | Nightly snapshot | 186.8 | South Central US (paired region) | Completed | |
| BK-20261108 | 11/08/2026 02:25 | Production | Nightly snapshot | 186.2 | South Central US (paired region) | Completed | |
| BK-20261107 | 11/07/2026 02:25 | Production | Nightly snapshot | 185.6 | South Central US (paired region) | Completed | Yes |
| BK-20261106 | 11/06/2026 02:25 | Production | Nightly snapshot | 185.0 | South Central US (paired region) | Completed | |
| BK-20261105 | 11/05/2026 02:25 | Production | Nightly snapshot | 184.4 | South Central US (paired region) | Completed | |
| BK-20261104 | 11/04/2026 02:25 | Production | Nightly snapshot | 183.8 | South Central US (paired region) | Completed | |
| BK-20261103 | 11/03/2026 02:25 | Production | Nightly snapshot | 183.2 | South Central US (paired region) | Completed | |
| BK-20261102 | 11/02/2026 02:25 | Production | Nightly snapshot | 182.6 | South Central US (paired region) | Completed | |
| BK-20261101 | 11/01/2026 01:25 | Production | Nightly snapshot | 182.0 | South Central US (paired region) | Completed |
Data as of run R-20261111-0200, finished 11/11/2026 02:13 Central. Curated layer: curated schema.