The scheduler console — 79 jobs, and what each one stops if it stops

A stopped scheduler is the quietest outage this platform can have. The site keeps serving, every page loads, the system health screen stays green — and no withdrawal is processed, no price is written, no investment settles, no temporary block expires. There is nothing in the ordinary admin panel that says so, because the only evidence of a scheduler is work that silently stops.

The console at Admin → System → Scheduled Tasks (/admin/system/cron) exists for exactly that. It is shaped around "what is not working" first and "what is working" second.

Processes and ports covers the cron process itself. Scheduled jobs reference lists all 79 jobs with their cadence. This page is how to read the screen.

Three signals, deliberately not merged

The page keeps three independent things apart, and any one of them can be down while the other two look fine. There is no single green tick, on purpose.

The job list is polled over HTTP every 20 seconds and the heartbeat every 15, underneath the socket. The socket only carries changes, so a page opened between two ticks of a six-hourly job would otherwise show nothing about it; and on a split deployment the browser's socket is attached to the web process while the state is written by the cron one, so a dropped relay would leave a page that looks live and is frozen. Polling underneath means the worst case is 20 seconds stale, not forever.

The heartbeat banner

The scheduling process rewrites one Redis key (cron:scheduler) every 15 seconds, with a 90-second TTL. The banner reports one of six states.

unknown is deliberately a warning and never destructive: "I cannot tell" and "nothing is scheduling" are different answers, and conflating them would put a red NO SCHEDULER banner on a healthy deployment every time Redis blinked.

The banner also names this process (mode and PID), because "the scheduler is elsewhere" is only meaningful relative to who is answering the request.

The five tiles

Health excludes both refused and failed jobs, so a refusing scheduler cannot read 100%. Last activity is the cheapest read on the whole page: on a healthy install the 15-second and 5-second jobs make this "a few seconds ago" permanently. A figure in minutes means something is wrong before you have read anything else.

The job list

Each row carries the job name, its category, and four figures under a sticky column legend:

Above the list: a text search, a category filter (the addon buckets plus Normal), and five tabs — All, Needs attention, Running, Completed, Idle. The Needs attention chip turns red when it carries a number; it gathers refused, degraded and failed together, and it is the reason to be on this page.

Clicking a row opens the detail panel. Clicking the row's Run now button fires the job by hand.

Why the list shows run state at all

Run bookkeeping — lastRun, status, successRate, lastExecutions, the sticky refusal — exists only in the process that actually schedules. Two deployment shapes serve this admin route from somewhere else, and a freshly-built registry has lastRun: null, status: "idle" hard-coded on every entry:

• The split (CRON_MODE=off on the web tier). This is the shape where the console is the only window onto a scheduler in another process, and a refusal recorded there could not otherwise reach this route at all.
• The threaded entry point (pnpm start:thread), where a worker thread also serves the admin routes — an operator polling the page saw two thirds of the responses claim every job had never run.

getCronJobsWithRunState() fixes both by hydrating those fields from a snapshot the scheduling process publishes to Redis after every completed run. If Redis cannot be read, the route reports the registry as it stands rather than inventing a state.

The REFUSED state

A job can decline to run in this process and still return normally. That is the whole point: throwing would be recorded as a failure, and a refusal is not one. But a normal return used to be stamped completed — so on this very table, a job that had not executed a single line of its work in a week was byte-identical to one succeeding every 30 seconds.

backend/src/cron/refusal.ts is the one sanctioned way for a job to decline, and it is never quiet. Every refusal lands in four places:

1. Sticky refusal state on the job row. The only one of the four that is still there tomorrow, and the only one that stops a refused job being reported as completed. It records since (when this refusal started, not when it was last seen) so the row can say "this has been skipped for six hours".
2. logger.error, for the process log and whatever ships it.
3. The live log on this page, so a refusal is visible on the screen that lists the job as scheduled.
4. An URGENT in-app and email alert to every Admin and Super Admin, linking back to /admin/system/cron. The only channel that reaches a human who is not already looking.

The state is written on every refused tick. The three announcements are rate-limited to one per job per fifteen minutes, because the jobs that refuse tick every 5 to 60 seconds and an alert arriving 1,440 times a day is one nobody reads. A message is an event; a state is a fact, and the fact has to hold between announcements.

Nothing here expires on a timer. A refusal is cleared only by a run that did not refuse, and a refused job never returns to idle — idle means "scheduled, waiting its turn", which is precisely the impression a refused job must not give. successRate counts only completed runs, so it drops rather than sitting at 100%.

What a refusal notice tells you

degraded is not refused

One variant means "running, but achieving nothing" rather than "not running". The handler executed, no scheduled tick was dropped — but the work has nowhere to land. The AI market maker engine is the case: with ecosystem disabled nothing in the deployment boots a matcher, so its bots have no market to quote, and saying "not running" would be both stronger and untrue.

Degraded rows count toward Not working and toward Needs attention, and are painted in warning colour rather than destructive. The three gates that can produce a refusal or a degradation, and what you change to clear each, are in Scheduled jobs reference.

Why refusals replaced exit code 78

Until the cron split became the default, an addon whose job could not run on a dedicated scheduler made the whole cron process print a banner and process.exit(78) — and 78 is in stop_exit_codes, so PM2 stopped the app rather than restart-looping it. That was defensible while the split was opt-in and an operator was necessarily watching.

As the default it inverts. The web process runs CRON_MODE=off, so a cron process that refuses to boot leaves the deployment with no scheduler at all — no withdrawals, no price updates, no settlement, nothing in the admin panel to say so — for an operator who never asked for the split and never read a banner that scrolled past at boot. One addon's limitation would take out all 79 jobs.

So the cron process now always starts, always runs every job that is safe, and the specific jobs that are not refuse themselves per run — loudly. What still exits 78 is deliberately narrow and unchanged: an unreachable Redis, and the native-module and Node-ABI preflight. Neither is addon-specific and neither is survivable; the web process fails on both identically, so exiting loses no scheduling the deployment still had.

The live log

The right-hand pane has two tabs, Live log and Activity.

The log is severity-coloured and severity-labelled, because cron traffic is overwhelmingly routine and the two lines that matter — a refusal and a failure — both arrive as error and both have to be findable while the stream is moving. There is a per-job filter (roughly 65 to 79 jobs share one socket, several ticking every five seconds) and autoscroll that pauses when you scroll up, with a resume button that says how many lines arrived meanwhile.

Two things about the stream that are not obvious:

• Opening the page is what starts it. The browser sends SUBSCRIBE on open, which puts it in the relay's audience set. Ordinary log lines are only built and published when somebody is in that set — the scheduler holds no sockets of its own, so it asks the web process whether anyone is looking before it serialises anything. With no dashboard open there is nothing to send, and roughly 500 call sites would otherwise fire into an empty room.
• Refusal lines skip that gate. Presence is heard over the bus and is therefore a slightly stale fact — a cron process that has just booted may believe nobody is watching for up to a heartbeat, and "this job is not running" is the one message that must not fall into that window. They are rate-limited to one per job per fifteen minutes, so always publishing them costs nothing.

Anything that must survive an unwatched run is written to logger at the same call site, so pm2 logs cron loses nothing when the page is closed.

Run now, and when it refuses

Run now calls POST /api/admin/system/cron/trigger with the job name. The button explains itself before you press it: the scheduler endpoint returns a canTrigger flag that mirrors the two refusals inside the manager, and the button is disabled with a tooltip when it is false.

canTrigger is false in two cases:

• This process is not the main thread. Under pnpm start:thread, worker threads serve this admin route too and their job map is permanently empty, so the single-flight guard would be a no-op and the handler would run beside the main thread's scheduled tick against the same rows. Retrying may land you on the right thread.
• Cron is delegated (CRON_MODE=off on the process answering you — the default production shape). Every web process is the wrong one, so this refusal is terminal. The 503 says to wait for the next scheduled run, and does not offer to trigger it on the cron process: that process registers no application routes at all, so there is no admin endpoint there to call, on a port nothing should connect to.

Run now shares its single-flight guard with the scheduler, so it returns 409 rather than running concurrently when a scheduled tick is already in flight. On a successful trigger the page re-reads the job list immediately, so the row settles without waiting for the next poll.

Which jobs appear on your install

The registry is built per install. 17 jobs are registered unconditionally; the other 62 are grouped into 17 extension buckets and only pushed into the registry when that extension is enabled. An install with no addons shows 17 rows, and that is correct.

Gating is re-evaluated every 60 seconds against the extension table directly, so toggling an extension on Admin → System → Extensions takes effect without a restart. The scheduler logs each change:

Extension staking enabled — scheduled cron processStakingPositions
Extension nft disabled — stopped cron settleAuctions

A disabled extension's jobs are torn down properly — the repeatable is removed from BullMQ, the fallback interval is cleared, and any job with a teardown hook (the background deposit scanner and the BTC scanner both arm their own internal loops) is asked to stop. Deregistering the supervisor tick alone would not have stopped those.

Permissions

A role with access.cron and no view.cron gets the page frame and no data. Super Admin short-circuits all three.

A five-minute check