Summary

On July 20, 2026, between 9:07 AM and 9:21 AM (Brasília time), the selfie capture and authentication flows were unavailable for approximately 14 minutes. The incident was caused by a query without an adequate index running as part of the database migration flow, causing database CPU to spike to 53%, dependent service latency to jump 78 times above normal, and the circuit breaker mechanism to fully block calls to the affected service. The causing behavior was reverted and the service recovered in approximately 1 minute.

Impact

Users of the selfie, authentication, and web-based proof-of-life flows received open-circuit errors during the period, with 743 and 125 requests blocked respectively in the most affected flows. The service returned to normal at 9:21 AM after the causing behavior was reverted

Root Cause

The root cause was identified in the database migration flow: a query introduced as part of the migration lacked the index needed to run efficiently on the new database. When the new behavior was activated in production, the query began performing full scans per tenant, with the number of rows read jumping from ~5 million to ~25 million per execution, saturating the database CPU and causing response latencies of up to 10 seconds.

Resolution

The team identified the database CPU spike as the source of the problem and correlated it with the recently introduced behavior change approximately 7 minutes after the first alert. The change was reverted at 9:20 AM, and the service recovered immediately.

Lessons Learned

The incident showed that volume differences between validation and production environments can mask performance issues that only manifest at real-world scale. Silent error handling in the application worsened the situation by making the degradation invisible to monitoring. As follow-up actions, the team prioritized creating the missing index and removing the silent error suppression pattern.