The order lifecycle

Three support questions account for most of the traffic an operator gets about this addon: why did my customer's order never fill, why is their balance locked with no open order, and what does CLOSED mean. All three are answered by the same sequence, so this page walks it once, end to end.

Everything here concerns ecosystem markets only — orders placed through POST /api/ecosystem/order and stored in the trading keyspace. Exchange (SPOT) orders are a passthrough to a third-party venue and follow none of it.

Two gates before anything is written

KYC feature trade. The handler asserts it before it branches on order type, so it applies to conditional orders too. If your levels do not grant the trade feature, no ecosystem order can be placed by anyone the policy applies to. The feature toggles live in the level builder at Users → Compliance & Verification → Verification Levels (/admin/crm/kyc/level) — see KYC: levels, features and the application queue.

A per-user throttle of 120 orders per minute. This route uses the trade-grade limiter, not the checkout-grade one that governs e-commerce and ICO purchases (5 per 15 minutes) — a limit ladder or a quick re-quote would otherwise lock a real trader out after a handful of orders. It fails open: if Redis is unavailable the throttle is skipped rather than halting all trading, because the money-safety guard here is the per-order balance check, not the rate limit.

Four types, two engines

The body's type is accepted in four values, and they split immediately.

Anything else is rejected with 422 before a token is touched.

The split is deliberate and structural. The matching engine has no trigger mechanism, and its match loop only branches on LIMIT and MARKET — an unrecognised type used to fall into a branch where neither side of the loop could advance, spinning the engine at 100% CPU. Stop orders are therefore diverted into a separate Scylla table the matcher never reads, which is the subject of Stop orders. Nothing on the rest of this page applies to them until they trigger.

Both branches are also refused when the market's status is off. A disabled market stops accepting new orders on the way in; cancels and refunds stay open, which is the right way round for a delisting.

The three statuses, and the one that does not exist

trading.orders.status only ever holds three values.

Two columns that look like they should move on a fill and do not:

• cost is the placement-time figure and is never rewritten. For a market BUY it is a deliberately conservative worst-case, not what the customer paid.
• average is never persisted at all. The ledger is inserted without it and no update path sets it, so it reads null on every order. Per-fill prices live in the trades JSON on the order itself, which is the permanent record of what executed.

CANCELED is spelled with one L. The admin orders table offers a Cancelled status filter spelled with two, and that value is passed straight into the CQL WHERE, so filtering on it returns nothing. Filter on OPEN or CLOSED and read the rest.

Price-time priority is the physical row order

open_orders_by_market is clustered (price, "createdAt", id), which is exactly what matchmaking.sortOrders derives priority from, in exactly that order. The database therefore returns rows already in matching order and the engine's per-cycle sort has nothing to do — it scans for a violation in O(n) and only sorts when it finds one. price is safe as a key column because nothing ever updates a placed order's price; a re-price would have to be a cancel and a new order.

Practical consequence: two customers resting at the same price fill oldest first, and that holds across a restart, because the ordering is a property of the table rather than of the engine's memory.

One caveat worth knowing before you read a book dump. Cassandra can only serve the full reverse of a declared clustering order, so reading the bid side ORDER BY price DESC also reverses time within each price level. Everything in the platform re-sorts, so this is invisible in behaviour — but a raw SELECT ... ORDER BY price DESC you run yourself will show bids newest-first inside a level.

The hold: what leaves the balance and when

Placement moves funds from wallet.balance into wallet.inOrder on the same row. Nothing is debited; a hold is a move between two columns of one wallet.

A market BUY does not hold amount x bestAsk. It walks the real asks, level by level, until amount is covered — but what it then holds is not the cost of that sweep. It holds amount multiplied by the deepest price the sweep touched, quantised at the market's quote precision and rounded up, plus the fee — strictly more than the sweep itself would cost whenever more than one level is crossed. The sweep total is computed on that walk, but it is used as the fee basis and for the cost-limit check, never as the hold. If the book cannot cover the amount, placement is refused with 422 ("Order book has insufficient liquidity...") rather than being partly held. The book it walks is the real one — AI market-maker display levels carry a TTL and no backing order, so sizing a hold from them would under-hold against the actual fill.

The order's stored price for a market BUY is then the deepest price the sweep touched, not the average. That looks wrong on the orders screen and is load-bearing: settlement releases each fill's share of the hold pro-rata by quantity, which only covers the fill when every execution price is at or below the stored one. Holding at the average let cheap top-of-book levels release more than they consumed, and the deeper levels then failed with "Buyer has insufficient locked funds" — leaving the sweep half done and the rest of the hold stranded in inOrder.

The extra hold is not a charge. Every fill refunds its own price improvement to the customer's available balance as it settles, so the net cost is the same. The fee, though, is quoted from the realistic average sweep price rather than the worst-case cap, so a market BUY crossing several levels is not overcharged.

Placement is atomic by construction: if the wallet hold fails the order row is rolled back, and if the enqueue fails the hold is released and the row rolled back, in that order. A crash between the two leaves the recoverable state (funds held, order OPEN and cancellable) rather than funds held with nothing to cancel.

Cancellation

The timestamp query parameter is required — it is the order's createdAt, part of the Scylla primary key.

The sequence that matters is the first step. claimOrderForCancel splices the order out of the matching queue under the engine lock and returns the remaining the engine sees. That is what makes the refund fill-consistent: any fill that was in flight has already completed, and no later cycle can touch the order after the claim. Reading remaining from a Scylla snapshot instead races concurrent fills, and the stale, too-large refund was partly minted.

The refund is then (cost + fee) x remaining / amount in quote for a BUY, or remaining in base for a SELL, released with an idempotency key and marked release-only so a retry can never credit funds that were not held.

Order of operations after that is chosen so every failure is recoverable: the funds are released before the row is marked CANCELED, and if the status write fails the claimed order is pushed back into the matching queue. An order left claimed-but-not-cancelled would be OPEN in Scylla, holding funds, resting in the displayed book, and invisible to the engine — permanently stuck, because a claim requires queue presence.

Cancel-all claims every order under one lock acquisition rather than one per order. Interleaving engine-lock and wallet-row locks is what produced multi-second lock convoys and MySQL "Lock wait timeout" storms when several cancel-alls ran at once.

Placing an order from a process that does not hold the matcher

Exactly one process may own the matching lease (ecosystem-matching); see Operations. Placement no longer requires it. The order row and the wallet hold are durable wherever they are written, and the engine's in-memory queue is a cache of the orders table — so a process without the lease persists the order and tells the leaseholder that a symbol changed.

Four moving parts, all in Redis:

Only a symbol string crosses the process boundary. The leaseholder re-reads that symbol's open orders and ingests what it does not already hold, which makes the whole path idempotent: a nudge delivered twice resyncs twice and the second finds nothing to do.

Cancellation needs its own channel because a resync cannot notice it — the order still looks perfectly open, so there is no row for a re-read to see. And it cannot be served on the follower at all, because the claim under the engine lock is the thing that makes the refund safe.

Two log lines are worth knowing. On the leaseholder, Ingested N order(s) for SYMBOL placed by another process is the only external evidence that cross-process placement is working end to end. Served a cross-process cancel for order <id> is its counterpart for cancels.

When an order rests and nothing fills it

Work down this list; each item is a different fault with a different fix.

The market has no counterparty. The ordinary case. Check the book on /trade?symbol=BTC-USDT&type=spot-eco.

The order is outside the resident window. The engine holds the orders nearest the market, per side, per symbol — 25,000 a side by default. Orders beyond that are safe in the ledger and are loaded as price reaches them. The log line is SYMBOL: resident window is N bid(s) / M ask(s) out of a deeper book. Nothing is wrong; see ScyllaDB schema and engine tuning.

The open-orders index is short of the ledger. If the index is missing a row for an order the ledger says is OPEN, the matcher never sees it and it rests forever on the customer's money. Run pnpm eco:index:check — it exits non-zero on drift and prints exactly what disagrees — then pnpm eco:index:repair and restart. The engine also re-checks one market an hour on rotation and falls back to reading the ledger if it catches a shortfall.

A ghost level. The aggregated orderbook table shows depth with no backing order, so the displayed book lies while nothing can match at that price. The signature in the log is:

Orderbook level missing on cancel: order <id> BTC/USDT ASKS @ 65000 —
status will be CANCELED but no level was decremented (ghost-level suspect)

The engine reconciles the aggregated book against real open orders every five minutes; pnpm rebuild:eco-orderbook is the offline repair.

Funds held with no open order. Usually a resting stop order — those reserve at placement, before any trigger. Otherwise run pnpm reconcile:eco-inorder, which only ever releases and never raises a hold.

The residue failure worth recognising

There is one historical fault that produced a genuinely immortal order, and its signature is distinctive enough to name.

orders.id and orders."userId" are declared UUID in Scylla, so the driver returns Uuid objects, while the ids reaching the cancel path come from HTTP params as strings. object === string is always false, so the cancel marked the row CANCELED in Scylla while silently failing to remove the order from the engine's in-memory queue. That left a copy resident with status: "OPEN" and remaining > 0 forever, which the five-minute reconciler then wrote back into the aggregated book as an immortal ghost level. It broke ecosystem Cancel-All, the Hummingbot batch cancel and the operator kill switch at once.

The ids are coerced to strings at the read boundary now. If you are looking at an older install and see a level that survives every cancel and reappears after each reconciliation, this is the shape of it — and a backend restart clears the in-memory copy, because the queue is rebuilt from the ledger at boot.

Where orders appear in the admin

Admin → Finance → Orders → Ecosystem (/admin/finance/order/ecosystem), gated by access.ecosystem.order for the screen and view.ecosystem.order for the data.

The screen also carries a cleanup action for corrupted rows — orders with a valid primary key and null symbol, amount, price or side. Those are an artefact of Scylla's upsert behaviour: an UPDATE against a primary key with no row behind it creates the row with nulls everywhere else. The list endpoint filters them out of the display, so the count in the cleanup dialog is the only place you will see them.

Run it with dryRun: true first.

Related

• Stop orders — the other half of the placement endpoint
• Trading fees and where the revenue lands — what the fee on each fill becomes
• ScyllaDB schema and engine tuning — the tables and the five knobs
• Operations — engine placement, repair scripts, backups
• Tokens and markets — the limits and precision every order is checked against