MMashDiv

An order did not settle

Four order books, four reconcilers and four ways to strand a row — which screen shows what, which job owns each book, and what a stuck order means on each one.

10 min readUpdated 6 August 2026orders, settlement, reconciler, futures, ecosystem

"My order filled and my balance is wrong", or "my order has been open for hours". Before anything else, work out which book it is on, because the four share almost nothing: different stores, different settlement paths, different reconcilers, different failure shapes.

canCreate, canEdit and canDelete are all off on Exchange, Ecosystem, Futures and Binary orders. There is no admin action that cancels, settles or edits an order. You diagnose here and fix somewhere else — by getting the owning job to run, or by adjusting a wallet at the end. The one exception is the ecosystem Cleanup corrupted orders button, described below, and it deletes ghost rows rather than settling real ones.

Which book, and who owns it

Screen Store Settled by Cadence
/admin/finance/order/exchange MySQL exchangeOrder processPendingSpotOrders — reconciles against the external exchange 60s
/admin/finance/order/ecosystem ScyllaDB, ecosystem keyspace The matching engine in the leaseholder process. No cron settles these continuous
/admin/finance/order/futures ScyllaDB, futures keyspace The futures matching engine; reconcileFuturesOrders replays lost writes 5 min
/admin/finance/order/binary MySQL binaryOrder An in-process expiry timer, with processPendingOrders as the backstop 15s

Two more screens belong to the same investigation:

  • /admin/futures/position — the positions behind futures orders. OPEN, CLOSED, CANCELLED, LIQUIDATED. Also read-only, also ScyllaDB.
  • /admin/system/cron — where you find out whether the owning job ran at all.

Permissions, if the screen will not open: access.exchange.order, access.ecosystem.order, access.futures.order, access.binary.order, each with its matching view. key, plus access.futures.position for positions.

The order to work in

  1. Find the order on its screen and read its status. The status vocabulary differs per book — the tables below.

  2. Check the owning job actually ran. /admin/system/cron. A stopped scheduler does not stop the website, so nothing else makes it obvious. See The scheduler console.

  3. Trigger it by hand and re-read the row. The Run now control needs manage.cron; view.cron alone shows the job but cannot fire it. If the row settles on a manual run, the order was never the problem — the schedule was.

  4. Only then escalate. If the job ran, succeeded and the row is unchanged, the failure is inside the book, and the per-book sections below say what that means.

Spot — /admin/finance/order/exchange

Statuses: OPEN, CLOSED, CANCELED, EXPIRED, REJECTED. CLOSED means filled.

The order lives on your external exchange provider, not here. This row is a local mirror of it, keyed by referenceId — the provider's own order id. Settlement is asking the provider what happened and applying it: credit on fill, refund on cancel or expiry, and the partial-fill arithmetic in between.

Two paths do that, and only one of them is reliable:

  • The WebSocket poller runs only while the customer has the orders page open. Close the tab and it stops.
  • processPendingSpotOrders runs every 60 seconds regardless. It loads every OPEN row that has a referenceId, asks the exchange for each one, and settles it. This is the safety net, and for a customer who placed a limit order and closed the tab it is the only path.

Three things stop it, in order of likelihood:

  1. The job is not running. Check it on /admin/system/cron.
  2. The exchange provider is throttling you. The reconciler checks the ban status first and skips the whole tick rather than spamming a rate-limited exchange, logging Exchange is banned; skipping spot reconciliation tick. That is the provider's problem, not the order's — see The exchange provider is degraded.
  3. The provider is unavailable. Exchange unavailable; skipping tick.

One deliberate outcome that reads like a fault: an order the exchange has archived — cancelled or expired with no executions, more than 90 days old — can no longer be fetched. The reconciler recognises that specific error, marks the row CANCELED locally and refunds. That is correct, not a lost order.

On this book the fee currency is the base asset for a BUY and the quote asset for a SELL — for BTC/USDT, a buy is charged in BTC and a sell in USDT. The feeCurrency column on the row says which, so read it before answering a customer who is comparing their filled amount to their debit. The ecosystem book does the opposite; see below.

Ecosystem — /admin/finance/order/ecosystem

Statuses: OPEN, CLOSED, CANCELLED. Rows live in ScyllaDB, not MySQL.

Funds are held before matching. A BUY locks cost + fee on the quote wallet at placement; a SELL locks the full amount on the base wallet. Those funds sit in the wallet's inOrder and are drained per fill. So an ecosystem order stuck OPEN is not merely an unfilled order — it is money the customer cannot spend, for as long as the row rests.

The matching engine is a deployment-wide singleton. Exactly one process holds the ecosystem-matching lease and does the matching; every other process is a follower. A follower that accepts an order writes it to Scylla and publishes the symbol to a Redis dirty set; the leaseholder drains that set every second and re-reads the symbol's open orders, with a full resync sweep every 60 seconds as the backstop.

If no process holds that lease, ecosystem orders do not settle at all — no amount of triggering crons will help, because no cron settles this book. That is a process-role problem, not an order problem: see Processes and ports. The symptom is platform-wide and unmistakable: every ecosystem order rests, nothing crosses, and funds accumulate in inOrder across the whole user base.

Cancelling releases held funds. The customer-facing cancel path is guarded — it claims the order out of the matching engine under the engine lock first, then releases release-only, so it can return held funds and never mint new ones. Anything that goes around that guard, or any bulk cancel run against orders that were never properly funded, is a way to create balance that never existed. There is no admin cancel in the panel for exactly this reason.

Both sides are charged in quote here: the seller is paid cost - fee and the platform collects both legs' fees in the quote asset. That is why a BUY holds cost + fee rather than cost. A customer comparing their filled base amount to their quote debit will always see the fee in the difference, and that is not a bug.

Cleanup corrupted orders

The one button on this screen. It scans the Scylla orders table for rows whose essential fields — symbol, amount, price, cost, side — are null. Those are ghosts produced by Scylla's upsert behaviour when an UPDATE was issued against a primary key that did not exist. They are not orders and they hold nothing.

It needs manage.ecosystem.order, and it takes Scan first before it will offer Clean up orders — always run the dry-run scan and read the counts before deleting. It scans up to 10,000 rows by default and is expensive (ALLOW FILTERING over the table), so run it in a quiet period. The action is recorded on the audit trail under module ADMIN_ECO, title Cleanup corrupted orders.

Futures — /admin/finance/order/futures and /admin/futures/position

Statuses: orders OPEN, CLOSED, CANCELLED; positions add LIQUIDATED.

This book is the one that can be internally inconsistent, and it has a distinct signature worth memorising.

Futures money is in MySQL; futures orders and positions are in ScyllaDB. Closing and liquidating both credit the wallet inside a MySQL transaction and then flip the Scylla status. If the process dies, or the Scylla write fails after its retries, the result is:

The wallet is correct. The position still reads OPEN in Scylla.

The customer has been paid and the screen says the position is still open. That is the failure shape, and it is not a money loss — it is a display and book-keeping divergence, and the money is already right.

reconcileFuturesPositions exists for exactly this. Every 5 minutes it looks for wallet credits whose idempotency key starts futures_close_ or futures_liquidation_ and replays the Scylla status update they were supposed to produce. It never touches wallets; the money is already correct in MySQL and the Scylla write is idempotent, so replaying is safe. It stamps the rows it handled so later ticks do not rescan them, and it works through a rolling window — 24 hours by default (FUTURES_RECONCILER_WINDOW_HOURS) — 500 rows at a time (FUTURES_RECONCILER_BATCH_SIZE), oldest first, so a backlog drains across ticks.

reconcileFuturesOrders is the same idea for the other write ordering. An order is written to Scylla first and the wallet is debited second, with key futures_order_<orderId>. A debit that committed with no matching Scylla order is an orphaned debit — funds locked with no open position — and the reconciler logs it loudly and marks the breadcrumb handled rather than retrying forever. The mirror direction (order present, debit missing) is deliberately out of scope and needs manual investigation.

Only the position reconciler reads the switch. reconcileFuturesPositions checks SCYLLA_ENABLED itself and returns an empty summary when it is false or when the ecosystem module is absent — it does nothing at all, and still reports a clean run. If positions are diverging and the job shows green, check that Scylla is actually enabled and reachable before reading anything into the job's status.

reconcileFuturesOrders has no such check. Its only guard is whether the ecosystem Scylla client module could be loaded — and that module exports a constructed (never connected) client plus a keyspace name defaulting to futures, so neither goes falsy on a disabled install. With SCYLLA_ENABLED=false it therefore keeps scanning MySQL every tick and firing CQL at a client that was never connected. That one does not look clean: it logs Failed to check Scylla order per row and its failed count climbs. Transport errors are treated as transient, so those breadcrumbs are left unstamped and re-tried on every tick until Scylla is back.

There is also sweepFuturesPositions, every 60 seconds. It is not a settlement reconciler — it re-marks open positions against the live ticker and enforces stop-loss, take-profit and liquidation. It is the safety net for a realm whose matching engine never booted; the engine itself sweeps every two seconds on the process holding the futures-matching lease. A minute is deliberate: it must not become the primary liquidation clock.

Binary — /admin/finance/order/binary

Statuses: PENDING, WIN, LOSS, DRAW, CANCELED.

A binary contract settles when it expires, from an in-process timer. That timer lives in memory, so a deploy, a crash or a restart loses it — and processPendingOrders, every 15 seconds, is the backstop that settles orders whose timer is gone. It skips orders an in-process timer still owns and takes a lock per order, so the tight cadence cannot double-settle.

Fifteen seconds rather than sixty is deliberate: a slow backstop settles past the point where the engine may still publish into the expiry candle, which makes a deliberate refusal indistinguishable from a late settlement.

Binary has enough distinct causes — the feature switch, the health endpoint, the per-process settings cache, a frozen price feed — that it has its own page: Binary orders that did not settle. Go there rather than working it from this one.

Two things that look like missing money and are not

An untriggered stop order holds funds and appears on no order screen

Ecosystem stop orders are hold-first. The reservation is taken at placement, before the trigger is ever crossed: a SELL reserves the base amount, a BUY LIMIT reserves amount × limitPrice + fee in quote, and a BUY MARKET reserves a worst-case estimate with 10% headroom, recomputed exactly when it triggers. Only when the stop price is crossed does the reservation get released and a real order placed.

Until then the order lives in a separate stop_orders table and is not on /admin/finance/order/ecosystem, which reads the orders table. So a customer with a resting stop sees a balance smaller than they expect, and you see no order explaining it. That is working as designed. Ask them whether they have a conditional order set.

The fee is in a currency they were not watching

Covered above per book, and it is the commonest "the numbers do not add up" ticket: spot charges a BUY in the base asset, ecosystem charges both sides in the quote asset. Read the feeCurrency column on the row before you conclude anything.

When the row really is stuck

If the owning job ran, succeeded, and the row is unchanged:

  • Spot — the provider's own record is authoritative. Check the order at your exchange provider directly. If it does not exist there, the row was never placed and holds nothing.
  • Futures — check whether the wallet was already credited (the customer's transaction ledger, filtered to their futures wallet) before doing anything else. If it was, the money is right and the reconciler is what you are waiting for.
  • Ecosystem — confirm a process actually holds the matching lease before treating it as a single-order fault.
  • All four — a hand adjustment is the last resort, not the first move, and it does not close the order row. See Customer wallets and balance adjustments for what an adjustment does and does not do.