Alarm investigations

Prod ALB p95 latency alarm fired after a few low-traffic request bursts (13 req/min) at 15:39 and 15:41 PDT took 6–11s. There were no 5xx errors, both hosts stayed healthy, latency was back to ~0.04s by 15:47, and the cause is unconfirmed.

Next: Confirm the alarm returns to OK, and use ALB access logs or the real app logs to find which endpoints were slow at 15:17–15:19 and 15:39–15:41 PDT.

Prod mermaid-api ALB p95 latency hit 28s around 18:20 UTC on Oct 17 while API CPU peaked at 89% and memory jumped from 23% to 71%. It recovered within a minute with no 5xx errors, but the root cause is unconfirmed and spikes recur.

Next: Check whether prod API task memory is still around 71%. Enable ALB access logs or app logging to find the slow endpoint behind the recurring spikes.

Prod ALB p95 latency alarm went off for about 1 minute (02:42–02:43 UTC, Oct 17). A few slow requests during low traffic caused it. No errors, unhealthy hosts or DB strain were seen. Cause not proven; the alarm is OK now.

Next: Find the slow endpoints via ALB access logs or the real app log location, and tune the alarm so a few slow requests at low traffic don't trigger it.

Prod RDS (db.t3.medium) CPU stayed above 90% from about 20:40Z to 22:30Z on Oct 17, during high API traffic and back-to-back summary-cache rebuilds. The alarm returned to OK at 22:34Z, and the root cause is suspected, not proven.

Next: Review Performance Insights top SQL for Oct 17 20:30-22:35Z, check the SummaryCacheQueue backlog, and watch for repeat spikes.