Incident volume, triage, auto-resolution and handoff, since table inception
Time-saved heuristic (same one the weekly handoff report uses): 15 min per KB auto-resolution, 20 min per LLM root-cause analysis, vs. a human manually digging through Airflow logs — applied once per distinct diagnostic event. 388 KB + 653 LLM first-occurrences × (15/20 min) + 1,731 duplicate rows × 2 min = 22,342 min = 372.4 hrs. This is cumulative time avoided across the whole team over 27 weeks (~13.6 hrs/week average) — not hours worked by one person in one day.
Methodology: incident volume has been rising all year (see section 3), so a flat extrapolation likely understates the total. Both scenarios are built on de-duplicated diagnostic events, not raw rows. Low end applies the full 27‑week average rate (372.4 hrs ÷ 192 days ≈ 1.94 hrs/day) to the 128 remaining days to Jan 31, 2027. High end applies the last three full weeks' rate (Aug 31–Sep 20: 188 distinct diagnostic events / 21 days ≈ 9.0 events/day ≈ 3.2 hrs/day) to the same 128 days. Actual will depend on whether that recent pace holds and whether flood frequency stays constant — both scenarios assume no material change to KB coverage, DAG failure rates, or the rate of flood-style outages; treat as a directional range, not a forecast commitment.
The bot sits between an Airflow task failure and a human. Every failure is filtered for noise before it ever becomes a logged "incident", then routed to either an instant fix or an AI-authored root-cause analysis:
Volume has grown sharply since May — largely LLM-handled failures (e.g. sfdc_etl_load*, jira_load Jira-API rate limiting). KB coverage hasn't kept pace with growth: KB accounted for ~40–60% of weekly volume through April, but has dropped to single-digit percentages most weeks since June as new/unclassified failure types outrun the known-issue library.
Of 2,771 incidents, only 54 (1.9%) escalated to actually paging a human — everything else was handled silently via Slack/Jira. Outages are almost entirely one recurring pattern:
| DAG | Paged incidents | Notes |
|---|---|---|
syslog2_daily_sensor | 49 | Known-Issue KB rule — per-cell syslog volume anomaly vs. historical average. Not transient; requires manual pipeline check. |
okta_s3_cellside_iga_aurora_incremental | 2 | LLM-flagged, escalated |
federal_datashare | 2 | LLM-flagged, escalated |
jira_load | 1 | LLM-flagged, escalated |
syslog2_daily_sensor (13 → 25). Worth a closer look at that sensor's health/thresholds before next quarter, since it's now the dominant source of real on-call pages.
OK19/OK20 page this quarter:
OK19 baseline window this week, (2) assign someone to root-cause the OK20 volatility before further tuning its threshold, (3) revisit nightly instead of quarterly given it's the #1 page source.
syslog2_daily_sensor has failed this exact way before — twice. A 2025-04-03 RCA found 3 weeks of missing OK5/OK10 syslog data in the federal cell because the syslog2-gov DAG silently dropped out of Airflow's dagbag (zombie tasks) — its own action item was "add OK5/OK10 cell data check to syslog2_sensor DAG." A 2025-12-09 P0 then lost OK5/OK10 data again, this time because a hotfix stripped those cells from a shared OKTA_CELL_LIST parameter used by multiple DAGs. The sensor now paging most in Sept 2026 is the same one both prior RCAs were about — this looks like a chronically under-fixed pipeline, not a fresh problem.
Section 4's "54 paged incidents" comes entirely from ONCALL_ANALYSIS_HISTORY, which only contains bot-logged Airflow DAG/task failures. Reading #eng-data-alerts and #eng-data-trex-alerts directly (Jan 1 – Sep 25, 2026) surfaces a second, structurally invisible population of pages: infra-level ECS/CloudWatch alarms that page a human the same way but never touch a DAG, so they can't appear in that count.
| Alert type | Channel / tenant | First → last seen (2026) | Frequency | In ONCALL_ANALYSIS_HISTORY? |
|---|---|---|---|---|
SYSLOG2_DAILY_SENSOR anomaly check | Both (prod + trex) | Predates 2026 → ongoing | ~1/night, escalating to 2–3/night in Sept | Yes |
jira_load / federal_datashare escalations | Both | Scattered all year | Low, sporadic (few/month) | Yes |
biwms-trex: Desired > running for <service> | #eng-data-alerts (despite the trex name) | 2026-09-08 → 09-23 | 7 alerts total | No — infra |
ECS deployment-stuck (worker/api_server/dag_processor/triggerer/scheduler "failing to start tasks, has NOT converged") | #eng-data-alerts (bi-wms-prod) | 2026-09-08 → 09-22 | 3 bursts: 9/8 (7 alerts), 9/17 (6), 9/22 (9, during biwms prod release 09/22) | No — infra |
A distinct pattern from the ECS-restart noise filter in section 2: a single DAG occasionally throws 10–20+ near-identical task-failure messages within the same minute or two, flooding the channel. Each one still gets logged as its own incident (inflating the DAG counts in section 7), but they share one root cause and one fix. Reading the bot's own thread replies (not just the alert headers) shows at least four distinct causes hiding behind that pattern:
| Cause | Example | Bot's diagnosis | Real KB-expansion candidate? |
|---|---|---|---|
| Airflow scheduler ECS restart (self-resolving) | 2026-09-22 ~13:11 EDT — 15+ sfdc_etl_load_v2 bronze-load tasks failed in the same minute | Correctly auto-detected: "Airflow Scheduler Restart Detected... not a code issue" — same event as the Sept ECS PagerDuty bursts above | No — already handled well |
| Code/schema defect fanning out | PR #10832 renamed a Snowflake schema (BASE_SALESFORCE→SALESFORCE) but missed a stored proc; broke ~15 sfdc_bronze_v2 objects identically | Correctly identified the PR, severity high, "not safe to retry" | No — one-off code bug, already root-caused |
| External API rate limiting | jira_load recon task hit Jira HTTP 429s 9 times in 3 weeks (Sept) | Flagged escalating trend, "not self-resolving," recommended a rate-limit exemption | Partial — bot sees it, no structural fix shipped yet |
| External SaaS outage during auth | 2026-05-21 ~17:20 EDT — 14+ sfdc_iceberg objects failed at once on Salesforce SERVER_UNAVAILABLE during SOAP login | Correctly distinguished transient outage from an actual expired/invalid credential | No — correctly triaged as transient |
| Genuine permission/access gaps | Compass metadata columns "not accessible for our user in prod" (Sept); Snowflake login rejected on IP allowlist (Jul) | Escalated to a human — bot has no fix path | Yes |
| Schema drift from an upstream source | SFDC silently renamed MDF_Budgeted_Cost_of_Campaign__c→MDF_Budgeted_Cost__c, breaking sfdc_etl_load with no notice (2026-04-24 RCA) | Surfaced the exact INVALID_FIELD error, but required a human to chase SFDC support for a fix | Yes |
| DAG | Total incidents | KB | LLM | KB % |
|---|---|---|---|---|
sfdc_etl_load | 525 | 251 | 274 | 48% |
compass_metadata_load_v2 | 289 | 185 | 104 | 64% |
sfdc_etl_load_v2 | 253 | 10 | 243 | 4% |
jira_load | 232 | 10 | 222 | 4% |
tableau_cloud_federal_copy | 193 | 1 | 192 | 1% |
sfdc_iceberg_load | 145 | 1 | 144 | 1% |
workflows_load | 79 | 43 | 36 | 54% |
databricks_ml_run | 77 | 10 | 67 | 13% |
syslog2_daily_sensor | 56 | 55 | 1 | 98% |
greenhouse_load_v1 | 54 | 53 | 1 | 98% |
These 10 DAGs account for ~1,900 of 2,771 incidents (69%) across only 87 distinct DAGs total — failure volume is highly concentrated. sfdc_etl_load_v2, jira_load, tableau_cloud_federal_copy and sfdc_iceberg_load are the best KB-expansion candidates: high volume, almost entirely LLM-handled (1–4% KB coverage), meaning the bot re-diagnoses the same/similar failures from scratch every time instead of applying a saved fix.
"Distinct diagnoses" and "Hours saved" are de-duplicated per section 1's methodology — one credited diagnosis per DAG + scheduled run, not per failed task row.
| Tenant | Total incidents logged | Distinct diagnoses | KB | LLM | Hours saved | % of diagnoses |
|---|---|---|---|---|---|---|
| bi-wms-trex | 1,415 | 443 | 170 | 273 | 165.9 | 42.6% |
| bi-wms-prod | 1,032 | 344 | 163 | 181 | 124.0 | 33.0% |
| bi-regional-prod | 218 | 192 | 16 | 176 | 63.5 | 18.4% |
| bi-wms | 94 | 54 | 34 | 20 | 16.5 | 5.2% |
| bi-regional | 12 | 8 | 5 | 3 | 2.4 | 0.8% |
bi-regional-prod barely floods at all — 218 logged incidents collapse to only 192 distinct diagnoses (88% ratio), vs. bi-wms-trex's 1,415→443 (31% ratio). Trex's raw incident count looks 4x bigger than bi-regional-prod's, but its real diagnostic-effort footprint is only ~2.3x bigger — most of that gap is flood duplication, not genuinely more distinct problems.
bi-wms-trex (the pre-prod/Trex cell) generates more incidents than bi-wms-prod — expected for a lower environment catching issues before they promote, but worth confirming that's the intended balance and Trex isn't just noisier due to under-provisioned resources.
Every Monday, the bot aggregates the trailing 7 days per tenant, creates one Jira ticket (project DA) summarizing metrics and top failing DAGs, tags the outgoing/incoming on-call Guardian (via PagerDuty schedule lookup), and posts the same summary to Slack. All bot analyses for that week get appended to the ticket as comments automatically, so the ticket becomes the week's incident log for whoever's on call. This has run continuously for 43 consecutive tenant-weeks with no gaps.
Engineers can react 👍/👎 to any bot analysis; 15 of 18 recorded reactions to date are positive (thumbsup, or confirming the bot correctly linked a failure to a PR), versus 3 negative (thumbsdown or "PR unrelated" corrections) — a rough proxy for perceived analysis quality, though the sample size (18) is small relative to 2,771 incidents.
ONCALL_RESOLUTION_EVENTS — the table designed to hold ALERTED_AT → RESOLVED_AT and MTTR_MINUTES — has 0 rows despite existing since mid-June. We can report incident count and resolution method confidently, but not how long incidents stayed open. Closing this gap (wiring up whatever marks an alert "resolved" to insert here) would let a future version of this report show real time-to-resolution, not just the flat 15/20-min proxy used for "time saved."
syslog2_daily_sensor is now the #1 source of real pages (25 of 27 paged incidents in September) and its page rate is accelerating month over month. This is at least the third documented occurrence of OK5/OK10 syslog trouble since April 2025 (section 4) — worth treating as a standing reliability project, not a one-off investigation.