Deterministic replay and queue-position research for passive execution.
A passive order’s fill depends on queue priority the public feed does not contain. queue_positioning replays a price-level book, places hypothetical orders under explicit latency and queue assumptions, and records where those assumptions change the answer. It does not send orders and it is not a trading system.
01 — The research problem
A limit order that does not cross rests behind displayed size. Whether it fills depends on who cancels, who trades, and where it sat. Public Binance spot depth is market-by-price: each update is an absolute quantity at a price, not an order identity. A decrease can be a cancel, a trade the feed has not yet paired, or both. The slot of an order that was never sent is not in the file.
The quantities that matter for a passive decision — fill probability, implementation shortfall against the decision mid, and later adverse selection — are therefore estimates under a queue rule, not observations. Changing the rule, or the delay before the order arrives, can change the estimate. That sensitivity is the object of study.
Displayed size at a price, best bid and ask, spread, mid, and prints with price, quantity, and a maker flag.
Which resting order was cancelled. Hidden size. The queue slot of a hypothetical order. One-way latency. A historical L2 book from the public trade dumps — those dumps do not include diff depth.
02 — System architecture
Stages below are the modules that exist. Simulated orders never enter the public book. Python reads files the OCaml engine already wrote.
scripts/capture_binance.py or generate-fixture. Capture is public streams, no API key, no orders.
binance_decoder turns REST snapshots and stream payloads into canonical events. Exchange timestamps are ms stored as µs.
Manifest checksum, mixed-symbol checks, then a replay. An experiment refuses a dataset that fails.
Spot diff rules. Total order is exchange time, then kind, then seq_index.
Three models share join-at-back. Trades are the only fills. Ledger is integer average cost.
Latency grid or explicit intents. CSV and JSON under reports/. Figures are a later Python step.
Saved state is the depth-sync record, not the book alone. Suffix replay must match the full hash.
Wall-clock of this binary, this machine. Not a portable throughput claim.
03 — How it works
Canonical events are snapshot, depth, trade, aggregate trade, and gap. Prices and quantities are integers at the instrument scale. Extra fractional digits are an error. A dataset is a manifest.json plus events.jsonl. Validation recomputes the SHA-256 and fails on a mismatch, a gap, a broken sync, a negative quantity, or mixed symbols.
The book is a map from price to displayed quantity. Zero deletes. Replay orders by exchange time when it is non-zero, otherwise receive time. On a tie the order is snapshot, trade, aggregate trade, depth, gap, then source index. Trades sit before a same-timestamp depth update so a print is not also counted as a cancel. That is a documented tie policy, not a claim of exchange causality. Two replays of the same file produce the same book hash.
Every model joins at the back of displayed size, after subtracting prints at that price the book update has not yet absorbed. Earlier simulated orders at the same price are known and sit in ahead_sim. Unexplained decrease C is displayed size that disappeared and was not a print. Conservative takes C from behind first. Optimistic takes it from ahead first. Probabilistic draws a seeded binomial with probability ahead_public / (ahead_public + behind), exact at or below 4096, normal approximation above. Adds join behind. A depth decrease is not itself a fill.
A trade at the limit consumes public ahead, then simulated ahead, then the order. A trade strictly through the limit fills the residual. Aggregate trades are ignored for fills when raw trades exist. A cancel that is effective at or before arrival is rejected and never rests. A fill during an in-flight cancel stands. Spread capture on a buy is mid_at_fill − price. Implementation shortfall on a buy is price − decision_mid. Fees hit the ledger immediately and are not subtracted again in those columns. Markouts use a later mid and are ex-post only.
The session grid varies submit delay and queue model on one synthetic book. The five-event divergence fixture is built so the cancellation rules can disagree. Report both. Do not average the models into a true queue.
dune exec queue_positioning -- experiment \
--config configs/queue_sensitivity.json
dune exec queue_positioning -- experiment \
--config configs/divergence.json04 — Results and experiments
Every number on this page is read from a file under reports/. The session is data/processed/synthetic/session, seed 7, 4000 steps, symbol field BTCUSDT, synthetic: true. It is not a Binance recording. Prices are integer price units. On this fixture price_scale is 2, so one unit is 0.01 of the quote currency. Quantities use qty_scale 4. Those scales are a scenario assumption, not a live exchangeInfo response.
Loading the session table…
Fill rate is the fraction of accepted orders with filled_qty > 0. Intervals are circular block bootstrap, block floor(sqrt n), percentile 2.5/97.5, 200 resamples. They are not a claim of correct coverage under every dependence structure.
configs/baseline.json is the same session, one model, quantity 20, 40 orders. It is a second synthetic cell, not a live comparison.
05 — Validation and reproducibility
dune runtest covers checksums, depth sync, duplicate and gap handling, replay determinism, checkpoint resume, queue-model disagreement on the divergence events, no double-count of a print, cancel/fill races, ledger identities, fee arithmetic, Binance JSON decode, and a QCheck random book update. The property test prints a seed. Frame encode/decode for the capture socket is python3 scripts/capture_binance.py --self-test.
Passing those tests means the stated rules are implemented and deterministic. It does not mean a modeled fill would have been the exchange’s fill. The session’s equal fill rates across models are an allowed result of the generator. They are not allowed to be silently forced apart.
| Check | What it is |
|---|---|
| Checksum | Event-file SHA-256 matches the manifest. |
| Sync | Drop u ≤ lastUpdateId, bridge on the next id, gap if the chain breaks. No invented pu. |
| Replay hash | Two full replays match. Checkpoint plus suffix matches the full hash. |
| Book | Off-tick, off-lot, and negative sizes rejected. Zero deletes. |
| Divergence | Same events: conservative fill 0, optimistic fill 1. |
| No double count | A print and the depth update that reflects it do not fill twice. |
| Ledger | equity = cash + mark·position = realized + unrealized. |
| Isolation | Simulated orders do not change the book hash. |
Source: docs/validation_protocol.md. Read reports/divergence/trace.csv before claiming a fill.
06 — Limitations
Public Binance depth is an absolute quantity at a price. Individual order identity and true queue position are not in the feed and must be inferred.
The committed experiments validate controlled scenarios. They are not evidence of live-market fill probability or of profitable execution.
A matching book hash means the engine applied the file. It does not mean the queue model is the exchange, or that a decision using it would have made money.
No live Binance depth is in the committed dataset. A capture attempt in this build environment failed with HTTP 451 on the first REST call. No raw file was written and no depth was invented. The offline protocol fixture in test/fixtures/protocol_example did normalize to a live book. A capture with network access to Binance is a different run and is not redistributed here.
Latency values in the grid are scenario assumptions, not measured round-trips. The 100 ppm fee is not a Binance fee-tier lookup. Int64 multiplication in the ledger wraps on overflow. The probabilistic normal tail for n > 4096 is not an exact binomial. Block bootstrap does not remove all dependence. Fuller list: docs/assumptions_and_limitations.md.
07 — Repository and usage
opam install . --deps-only --with-test
dune build
dune runtest
python3 scripts/capture_binance.py --self-testdune exec queue_positioning -- generate-fixture \
--out data/processed/synthetic/session --steps 4000 --seed 7
dune exec queue_positioning -- validate \
--dataset data/processed/synthetic/session
dune exec queue_positioning -- replay \
--dataset data/processed/synthetic/sessiondune exec queue_positioning -- normalize \
--raw test/fixtures/protocol_example \
--out /tmp/queue_positioning-protocol --symbol BTCUSDT
dune exec queue_positioning -- capture \
--symbol BTCUSDT --duration-seconds 30 \
--out data/raw/btcusdt-30
QUEUE_POSITIONING_PROFILE=release dune exec --profile release \
queue_positioning -- benchmark \
--dataset data/processed/synthetic/session \
--synthetic-events 200000 --out reports/bench.jsonExit codes: 0 success, 1 failed validation or a gap, 2 bad usage. simulate and experiment are the same runner. Do not commit raw captures.
| Path | What it is |
|---|---|
| lib/ | Replay, depth sync, queue models, fill engine, ledger, experiment. |
| bin/main.ml | CLI: validate, replay, generate-fixture, experiment, benchmark, normalize, capture. |
| configs/ | Session grid, divergence, baseline. |
| test/ | Engine tests and the hand-built protocol fixture. |
| docs/ | Architecture, queue models, exchange semantics, validation, limits. |
| reports/ | Committed synthetic outputs, benchmark, research note, figures. |
08 — Research notes
Rendered from reports/execution_research.md. If the CSVs change, regenerate that file with python3 scripts/write_report.py. Do not hand-edit the numbers.