Notification delivery operations

Running the notification service day to day — the queue's counters and what "degraded" means, proving each channel still delivers, and working the "customers stopped getting emails" incident.

12 min readUpdated 6 August 2026notifications, queue, email, sms, incident

"Customers stopped getting emails" is one of the highest-frequency incidents on any install, and the platform gives you twenty admin endpoints to answer it with. This page is how to use them under pressure.

The configuration side — the four mail transports, every SMS, push and VAPID variable, and what each tab contains — is The notification service. Read that once. This page assumes it is already configured and something has changed.

The console

System → Communication Tools → Notification Service, /admin/system/notification, seven tabs — six of them over one API surface, and PWA over a different one. The active tab is in the URL as ?tab=, so a link to a specific tab is shareable.

Tab	Use it when
overview	First look — service status, Redis, registered channels, totals
health	Is the service itself broken
test	Prove a channel delivers, right now, with a real message
queue	Email is backing up or failing
metrics	How much did we send, and how much failed
PWA	Editing the installable-app manifest (not a delivery surface)
settings	What is actually configured — read-only

Permissions: every notification endpoint on this console gates on access.notification.settings, which covers six of the seven tabs. PWA is the exception — it is not on the notification API at all. It reads and saves /api/admin/system/pwa (view.settings to read, edit.settings to save), adds and removes screenshots at /api/admin/system/pwa/screenshot (edit.settings), and uploads icons through PUT /api/admin/system/settings/logo (edit.settings). The two child screens have their own: /admin/system/notification/template needs view.notification.template to read and edit.notification.template to change anything; /admin/system/notification/sms reuses access.notification.settings.

/admin/system/notification has no entry in the URL permission map, so it opens for anyone holding the catch-all access.admin. Every request the six notification tabs make is refused without access.notification.settings — so an under-privileged operator sees an empty console rather than a locked door, and may report the service as "down".

The PWA tab splits the other way. An operator holding only access.notification.settings gets a working console with a dead PWA tab; one holding only view.settings / edit.settings gets the reverse — a live PWA tab in a console where nothing else loads. Granting both is what makes the screen whole. Check the role before you check the service.

The queue is the email channel, and only the email channel

This is the single most useful thing to know before an incident.

EmailChannel enqueues; SMSChannel, PushChannel and InAppChannel deliver inline, inside the request that triggered them. So:

A queue that is empty and healthy tells you nothing about SMS, push or in-app delivery.
An SMS that never arrived will never appear as a failed job. Its evidence is in the backend log and in the per-channel metrics, not the queue.
Conversely, mail is asynchronous: "the platform sent it" and "the customer received it" are separated by this queue, and that gap is where an outage hides.

The queue is a Bull queue named notification-emails, and it lives in Redis — not in MySQL, and not in the backend's memory.

Redis is a hard boot dependency. An unreachable Redis at startup stops the backend with exit 78 rather than degrading delivery — PM2 shows the app as stopped, not errored, with a boxed explanation still on screen. There is no partial mode where the API runs and mail quietly stops.

Redis going away after boot behaves differently: the queue cannot be created, so addEmailJob falls back to sending each message inline rather than dropping it. Mail still goes out, more slowly, on the request's own thread — and queue/stats reports all zeros and queue/items returns an empty list, because there is no queue to inspect. Zeroes across the board with mail still arriving is that state, not a quiet system.

Three defaults worth knowing because they explain numbers you will see:

Retries: 5 attempts (MAIL_QUEUE_MAX_ATTEMPTS), with a backoff that honours what the provider asked for — a Gmail 454 4.7.0 Too many login attempts comes back in minutes, not in two seconds.
Send budget: 10 messages per 60 seconds across the whole deployment (MAIL_QUEUE_RATE_MAX, MAIL_QUEUE_RATE_WINDOW_MS). The limiter is Redis-backed, so it counts across the backend and cron processes together. Over-budget jobs move to delayed and go out later; nothing is dropped. Raise it only once you know your provider's real ceiling.
Retention: Bull keeps the last 100 completed and 500 failed jobs. Older ones are gone whether or not you ever clean the queue.

Reading the queue

Counters, failure rate and a health verdict

Returns a queue block — waiting, active, completed, failed, delayed — and a derived health block with totalJobs, failureRate and a status.

The two derived numbers are computed differently, and the difference matters:

failureRate = failed / (completed + failed) × 100, to two decimals.
status is degraded as soon as failed exceeds 10% of completed, and healthy otherwise.

Those are not the same test. Early in a process's life — few completions, a handful of failures — the verdict flips to degraded on a very small number of messages. Read the raw counters, not the badge.

The endpoint's published response schema lists a paused boolean. The handler does not return one, and there is no admin control that pauses this queue. Do not build a monitor on that field.

The Queue tab refreshes these every 5 seconds.

What each shape means:

What you see	What it is
`waiting` climbing, `active` at 0	Nothing is draining it. Check the backend is up and Redis is reachable
`failed` climbing alongside `completed`	The transport is rejecting sends. Open an item, read its provider and attempt count, then reproduce with the Test tab
`delayed` large, everything else calm	Retries backing off, or the 10-per-minute budget. Resolves itself
Everything zero, but mail is arriving	Redis went away after boot and sends are going inline
Everything zero, and mail is not arriving	Nothing is being enqueued at all — look at `MAIL_DISABLED` and the sending code path, not at the queue

The jobs that have not been delivered yet

Up to 200, 50 by default, ordered active first (mid-delivery) then the backlog in the order it will drain. Each row carries the user id, notification id, title, type, channels, template, provider, attemptsMade, when it was queued, a human-readable age, and a status of pending or processing.

It deliberately excludes completed and failed jobs, because Bull's retained 100 and 500 would bury the handful still waiting. This answers what is stuck, not what went wrong — for that, read the backend log for the module the transport logs under.

Removes old completed and failed jobs

Takes olderThan in milliseconds, defaulting to 86400000 (24 hours), and returns the number removed with the first ten ids.

It removes completed and failed jobs older than the grace period, and nothing else. It does not cancel a waiting job, does not retry a failed one, and does not touch delayed. Clearing the failures does not fix the cause — it destroys the evidence of it.

Read queue/items and the backend log first, then clean. There is no admin button for this: no screen in the panel calls the endpoint, so it is reached deliberately, by API, which is the right amount of friction for an action that cannot be undone.

Proving a channel still delivers

Four endpoints, and every one of them sends a real message. This is not a dry run — it is the point. You want to discover a broken channel before a customer does.

Sends a real in-app notification

Sends a real email through the configured transport

Sends a real SMS

Sends a real push notification

Each takes a userId, defaulting to your own account, and each response names the channels delivered and the channels that failed with the channel's own error string. The email and SMS tests accept an override recipient but refuse to relay it anywhere except your own account's address or phone — an unguarded test endpoint is an open relay.

The push test is the most informative of the four: before sending anything it reports fcmAvailable, webPushAvailable and userHasTokens, so it can tell you "no provider is configured" apart from "this customer never accepted the browser prompt", which are the two completely different causes of a silent push channel.

There is a fifth, older test worth knowing:

Queues the EmailTest template to your own account address

It ignores any address you pass and always sends to the calling administrator's own account email. Its value is that it goes through the queue rather than sending inline, so a message that arrives proves the queue is draining as well as that the transport works. An optional name parameter fills the template's first-name placeholder.

It proves the platform handed the message to the transport and the transport accepted it. Rejection at the recipient's side — SPF, DKIM, DMARC, a spam folder, a provider blocklist — is invisible from here. If tests pass and customers still report nothing, the next place to look is your mail provider's own delivery log, not this console.

Health, metrics and analytics answer different questions

Endpoint	Answers
`GET .../notification/health`	Is it broken? Redis connectivity and cache hit rate, the registered channels and their count, the queue's five counters, and the backend's `uptime`
`GET .../notification/metrics`	How is it performing? Sent, failed and success rate, overall and per channel, over a `period` of `hour`, `day` (default), `week` or `month`
`GET .../notification/analytics`	How much did we send? Time-series KPIs and chart data over `24h`, `7d` (default), `30d` or `90d`

Only the first is diagnostic. Reach for health during an incident; the other two are for reporting, and a healthy-looking chart of last week's volume says nothing about the last ten minutes.

One detail on health worth using: uptime is the backend process's own uptime, which tells you when channels were last registered. A channel that is missing from components.channels.available after you fixed its credentials means the process has not restarted — channel registration reads the environment at boot.

Templates: edit and enable, never create or delete

System → Communication Tools → Notification Templates, /admin/system/notification/template.

Each template carries a subject, an emailBody, an smsBody and a pushBody, plus three independent booleans — email, sms, push — deciding which channels use it, and the list of short codes it can substitute.

There are only five endpoints behind the screen: list, read one, update one, toggle its status, and GET /api/admin/system/notification/template/wrapper, which returns the email wrapper's HTML — the screen fetches it to preview a template inside the frame the customer will actually receive. That wrapper is read-only from here; it is a file on disk (email/templates/generalTemplate.html), with no endpoint to change it. There is no create and no delete, and no write endpoint exists to add one. That is not an oversight: the set is fixed by what the platform actually sends, and a template with no sending code behind it would never fire.

New templates therefore arrive with a release, through the seeder backend/seeders/20240402234702-notificationTemplates.js, which pnpm seed runs — and pnpm seed is part of every update. If a release adds a notification and you do not see its template, you have not finished the update; see Updating.

package.json defines seed:notification as node scripts/update-notification-templates.js, and that script is not present in the repository. Running it fails with a module-not-found error. Use pnpm seed, which runs the notification template seeder along with the rest.

SMS providers

System → Communication Tools → SMS Providers, /admin/system/notification/sms, compares Twilio and MSG91 side by side — coverage, cost, setup effort, the India DLT position, and which required and optional credentials are currently present.

Lists both providers with credential status

Validates candidate credentials against the vendor without saving or sending

The test endpoint is the one to use before you write a key into .env and restart: it builds a throwaway provider from the values you supply, checks them against the vendor, and discards it. Nothing is persisted and no SMS is sent. Any field you omit falls back to the configured environment, so you can test the one value you are about to change.

MSG91's send endpoint returns {"type":"success"} with no authkey, and accepts an OTP-Widget tokenAuth as readily as a real server authkey. A presence check passes, a send-based test passes, and nothing is ever queued at the telco. Only a call to an endpoint that genuinely authenticates tells the difference, which is what this route and the SMS health probe both do.

Twilio fails loudly by comparison, rejecting a bad credential synchronously. If you are debugging silent SMS on MSG91, assume the credential is wrong until this endpoint says otherwise.

The credential test is deliberately kept off the admin audit trail — it checks values that have not been saved, so there is no record for an audit row to point at — and credential values never reach the log context; only the field names appear.

When customers stop getting emails

Confirm the backend is up and Redis is reachable. pm2 list, then redis-cli ping. An app sitting at stopped right after a start has exit 78 in its log with the reason in a box — see Logs.
Open the Queue tab and read the raw counters, not the badge. Use the shape table above. Zeroes with mail still arriving is the Redis-after-boot case; zeroes with no mail at all means nothing is being enqueued.
Check MAIL_DISABLED. Set to true, 1 or yes, it drops every outbound message before it reaches the queue, logs MAIL_DISABLED: dropping … and reports success upstream. It exists so a test run cannot burn the production mail account's login quota, and it is the single most common reason a working install stops sending.
```
grep -i "MAIL_DISABLED\|APP_EMAILER" .env
pm2 logs backend --lines 200 --nostream | grep -i "MAIL_DISABLED\|EMAIL"
```
Send a real test. Test tab → Email. Then the older GET .../notification/email/test, which goes through the queue and therefore proves it is draining. If the first passes and the second never arrives, the transport is fine and the queue is stuck.
If it failed, read the failure, not the count. queue/items for what is stuck; the backend log for why. A 454 4.7.0 Too many login attempts means the provider account is throttled and no configuration change helps until it lifts — leave MAIL_DISABLED=true on while you test anything that emits notifications.
If the transport itself is wrong, fix .env and restart the backend. Mail settings are read from the environment, not the settings table, so a restart is required — see The notification service for the exact variables per transport.
Only then clean the queue, if the failed count is noise you no longer need. It is irreversible.

Two failures that look like this one and are not: a customer whose address ends .invalid, .test, .example or .localhost is dropped before the queue by design, and a customer who never accepted the browser prompt has no push subscription however healthy the push channel is.

Routine

Weekly — open the Test tab and send yourself one message on each channel you actually use. Delivery breaks silently; nothing else will tell you.
Weekly — glance at the Queue tab. A failed count that grows week on week is a transport degrading, not noise.
After every update — confirm new templates arrived (pnpm seed runs as part of pnpm updator), and re-run the channel tests, because a restart re-registers channels from the environment.
After any .env mail, SMS or push change — restart the backend, then test that channel. The console reports what is configured, and a restart is what makes a configuration real.

The notification service — every transport, variable and tab in detail.
Environment variables — the mail, SMS and push variables, and the duplicate names that configure nothing.
The health screen — the platform-wide Email Service probe, which is a configuration check and not a delivery check.
Logs — where a transport's own error message is written.
Troubleshooting — "Emails are not being sent", from the infrastructure side.