AKServices Incident Note — 02 Nov 2025
=====================================

Title: Recharge operator queue spike
Severity: Moderate (delays but no financial loss)
Start: 02 Nov 2025, 10:05 IST
End: 02 Nov 2025, 14:32 IST
Duration: 4 hours 27 minutes of intermittent delay

Summary
-------
MTNL's recharge switch began throttling callbacks which caused our aggregator queue to back up. Transactions were accepted but confirmations were delayed and the front-end showed "processing" for longer than usual.

Customer Impact
---------------
- ~2,800 recharge requests experienced 2–5 minute completion delays.
- No duplicate charges reported; wallet balances stayed accurate.
- Retailers escalated via helpdesk due to longer counter wait times.

Root Cause
----------
Our shared worker pool for MTNL ran at a static concurrency of 5. When MTNL slowed responses, our own queue length grew past 1,200 entries, yet autoscaling never triggered because CPU usage stayed under threshold.

Fix & Mitigations
-----------------
1. Temporarily rerouted new MTNL traffic to backup aggregator while draining queue.
2. Raised worker pool to 15 concurrent slots for MTNL profiles and added queue-length metric.
3. Implemented alert if average operator wait exceeds 2 minutes for 5 consecutive checks.

Follow-up Actions
-----------------
- Include operator-specific autoscaling logic by 10 Nov 2025 sprint.
- Publish transparency update to distributors and archive copy in incident log.
- Reviewed settlement reconciliations to confirm no pending credits.