Aerialytic Incident Framework

2026-07-15 Kafka file-download consumer-group replay and Nearmap throttling

A production rollout changed kafka-file-download from the established FILE-DOWNLOAD-group consumer identity to FILE-DOWNLOAD-rmuqop-group while retaining fromBeginning:true. The new group had no committed offsets and replayed retained historical download requests during a 66-second old/new rolling overlap. Duplicate traffic caused Nearmap HTTP 429 throttling, file-download DLQ growth, and customer-facing iframe design-generation failures. Production was rolled back to version 2.0.0 and is currently healthy, but permanent remediation, exact impact enumeration, and recovery of failed jobs remain pending.

SEV2Partial OutageMonitoring / Remediation PendingINC-2026-07150002

Current State

Production is rolled back to version 2.0.0 and the latest runtime sample is healthy. Permanent remediation, exact impact inventory, failed-job recovery, customer follow-up, and urgent adjacent security actions remain open. PR #4241 is partial candidate mitigation only and is not live.

2222bad-release DLQ outcomes
1267HTTP 429 outcomes
955HTTP 400 outcomes
Pendingunique customers and jobs

Reports And Data

Canonical public-safe JSONAction register CSVEvidence register CSVAction RegisterCustomer Impact SummaryDeployment Remediation ReportEvidence RegisterExecutive BriefPostmortemRollback ReportRoot Cause AnalysisSecurity Reliability AssessmentTechnical Incident ReportTimelinePrivate Slack incident recordSHA-256 manifestSafe incident package

Action Register

IDPriorityStatusActionOwnerDue
ACT-0001P0in_progressComplete, merge, release, and verify the #4241 remediation setKafka File Download Service Owners2026-07-16
ACT-0002P0acceptedRecover failed jobs with deduplication and provider-wide rate boundsAerialytic Data Recovery and Kafka Platform2026-07-16
ACT-0003P0in_progressProduce exact impacted customer and job inventoryAerialytic Incident Response and Data Engineering2026-07-16
ACT-0004P0acceptedComplete customer recovery and approved communicationsAerialytic Customer Success and Support2026-07-16
ACT-0005P1acceptedAdd offset-continuity and cancellation deployment gatesAerialytic Release Engineering2026-07-17
ACT-0006P1in_progressShip 429, DLQ, replay-pressure, and group-drift alerts with runbook #4244Aerialytic SRE and Observability2026-07-17
ACT-0007P1acceptedRestore production workload log retentionAerialytic Cloud Platform and Security2026-07-22
ACT-0008P1acceptedAdd customer-facing error correlation and actionable design failure statesAerialytic Iframe Product and Frontend2026-07-22
ACT-0009P0acceptedAdd provider-wide header-aware rate control and circuit breakingKafka File Download Service Owners and SRE2026-07-17
ACT-0010P0acceptedHarden Kafka consumer ownership, buffering, and crash consistencyKafka Platform and File Download Service Owners2026-07-17
ACT-0011P0acceptedRemove file path traversal and root-container exposureKafka File Download Service Owners and Application Security2026-07-16
ACT-0012P0in_progressRemove credential-bearing startup logs and rotate exposed Kafka/TLS materialAerialytic Security and Cloud Platform2026-07-16
ACT-0013P1acceptedMake incident contracts hard release gates on the actual artifactAerialytic Release Engineering2026-07-17
ACT-0014P1acceptedMake health checks reflect the live consumer run loopKafka File Download Service Owners2026-07-18

Timeline

UTCEventEvidence
PR #4171 merged; commit 16ab67a changed the production consumer-group identity.EV-0003
PR #4205 promoted the consumer-group change into main.EV-0003
kafka-file-download version 2.0.2 was published.EV-0003
mainenv commit b37d2d6 changed the image from 2.0.0 to 2.0.2. CD was cancelled 23 seconds later, but the GitOps commit already existed.EV-0003
Production was patched to version 2.0.2 and ReplicaSet 7656f5446f appeared.EV-0003
Old and new consumer groups overlapped for 66 seconds through 19:39:47Z, creating concurrent processing from separate offset lineages.EV-0003
First observed HTTP 429 response in the measured incident window.EV-0003
The 19:47 minute peaked at 1,013 HTTP 429 DLQ records.EV-0003
An external customer reported 'Aerialytic error (code undefined)' while generating designs; other iframe failures continued.EV-0001, EV-0002
mainenv revert 16e8f8e restored image version 2.0.0.EV-0003
Rollback old/new overlap lasted 45 seconds; the final version 2.0.2 pod stopped at 20:01:56Z.EV-0003
Last observed HTTP 429 in the measured window. Provider throttling and queued work had continued after rollback.EV-0003
Full CI for PR #4241 was queued on the self-hosted runner; it was not green at incident-publication time.EV-0005