trading_bot_v4/docs/ORDERBOOK_SHADOW_LOGGING.md

Orderbook Shadow Logging System

Status: ✅ DEPLOYED (Dec 19, 2025)
Purpose: Track orderbook metrics for all trades to prepare for real orderbook integration (Phase 2)

Overview

Phase 1 implementation: Oracle-based estimates with 2bps spread assumption. Captures orderbook snapshots at trade execution only (not periodic).

Critical Scope Note: Orderbook data is captured ONLY when trades execute via /api/trading/execute, NOT during 1-minute market data feed updates (/api/trading/market-data). The 1-minute feed captures price/ATR/ADX/RSI metrics but does NOT capture orderbook data. This is intentional to minimize API overhead - we only need orderbook state at the moment of trade execution.

Database Schema

Table: Trade
New Columns (all nullable DOUBLE PRECISION):

Column	Description	Expected Range
spreadBps	Bid-ask spread in basis points	2-100 bps typical
imbalanceRatio	Order flow imbalance (-1 to 1)	Negative=sell pressure, Positive=buy
depthBid1Usd	Bid depth USD (1% from mid)	>0
depthAsk1Usd	Ask depth USD (1% from mid)	>0
priceImpact1Usd	Price impact for $1 trade	Small decimal
bidWall	Largest bid wall detected	NULL or >0
askWall	Largest ask wall detected	NULL or >0

ENV Configuration

# Enable/disable orderbook shadow logging
ENABLE_ORDERBOOK_LOGGING=true  # Default: true (omitting = defaults to true)

How to toggle:

# Disable logging
echo "ENABLE_ORDERBOOK_LOGGING=false" >> .env
docker compose restart trading-bot

# Re-enable
sed -i 's/ENABLE_ORDERBOOK_LOGGING=false/ENABLE_ORDERBOOK_LOGGING=true/' .env
docker compose restart trading-bot

Implementation Details

Files Modified:

• prisma/schema.prisma - Added 7 orderbook fields to Trade model
• config/trading.ts - Added enableOrderbookLogging boolean flag (line 127)
• types/trading.ts - Added orderbook fields to CreateTradeParams interface
• lib/database/trades.ts - Updated createTrade() to save orderbook data
• app/api/trading/execute/route.ts - Added ENV flag wrapper (lines 1037-1053)

Data Source (Phase 1): Oracle prices with 2bps spread estimates

• No real orderbook API integration yet
• Estimates sufficient for pattern analysis
• Phase 2 will integrate Hyperliquid/Jupiter for real data

CRITICAL: Trade Execution Only:

• Orderbook capture happens in /api/trading/execute endpoint (lines 1037-1053)
• Triggered ONLY when trade signal executes (not periodic)
• /api/trading/market-data endpoint (1-minute feed) does NOT capture orderbook data
• 1-minute feed captures: ATR, ADX, RSI, volume, price position (stored in MarketData table)
• Orderbook data stored in: Trade table (7 columns added via ALTER TABLE)
• Frequency: 3-5 orderbook snapshots per day (when trades execute) vs. 1,440 market data updates per day

Verification Queries

Check Latest Trade with Orderbook Data

SELECT 
  id, 
  symbol, 
  direction, 
  spreadBps, 
  imbalanceRatio, 
  depthBid1Usd, 
  depthAsk1Usd,
  priceImpact1Usd,
  bidWall,
  askWall,
  "createdAt"
FROM "Trade" 
WHERE spreadBps IS NOT NULL 
ORDER BY "createdAt" DESC 
LIMIT 1;

Count Trades with Orderbook Data

SELECT 
  COUNT(*) as total_trades,
  COUNT(spreadBps) as trades_with_orderbook,
  ROUND(100.0 * COUNT(spreadBps) / COUNT(*), 1) as pct_coverage
FROM "Trade";

Average Orderbook Metrics by Symbol

SELECT 
  symbol,
  COUNT(*) as trades,
  ROUND(AVG(spreadBps)::numeric, 2) as avg_spread_bps,
  ROUND(AVG(imbalanceRatio)::numeric, 3) as avg_imbalance,
  ROUND(AVG(depthBid1Usd)::numeric, 2) as avg_bid_depth,
  ROUND(AVG(depthAsk1Usd)::numeric, 2) as avg_ask_depth
FROM "Trade"
WHERE spreadBps IS NOT NULL
GROUP BY symbol
ORDER BY trades DESC;

Spread vs Win Rate Analysis

SELECT 
  CASE 
    WHEN spreadBps < 5 THEN '0-5 bps (tight)'
    WHEN spreadBps < 10 THEN '5-10 bps (normal)'
    WHEN spreadBps < 20 THEN '10-20 bps (wide)'
    ELSE '20+ bps (very wide)'
  END as spread_bucket,
  COUNT(*) as trades,
  ROUND(100.0 * COUNT(CASE WHEN "realizedPnL" > 0 THEN 1 END) / COUNT(*), 1) as win_rate,
  ROUND(AVG("realizedPnL")::numeric, 2) as avg_pnl
FROM "Trade"
WHERE spreadBps IS NOT NULL
  AND "exitReason" IS NOT NULL
GROUP BY spread_bucket
ORDER BY MIN(spreadBps);

Deployment Timeline

Date	Action	Status
Dec 18, 2025	Task 1-3: Code implementation	✅ COMPLETE
Dec 18, 2025	Task 4: ENV flag + syntax fix	✅ COMPLETE
Dec 18, 2025	Task 5: Database migration via direct SQL	✅ COMPLETE
Dec 18, 2025	Container rebuild and restart	✅ COMPLETE
Dec 18, 2025+	Production validation (await next trade)	⏳ PENDING

Production Validation Checklist

After next trade executes, verify:

• Query returns non-NULL orderbook values
• spreadBps in reasonable range (2-100 bps)
• imbalanceRatio between -1 and 1
• depthBid1Usd and depthAsk1Usd > 0
• priceImpact1Usd is small positive decimal
• bidWall/askWall NULL or positive (if detected)

Manual Test (Optional):

# Trigger test trade via Telegram
# Send: "long sol --force" or "short sol --force"
# Then query database for orderbook data

Phase 2 Roadmap (Future Enhancement)

Real Orderbook Integration:

• Hyperliquid API for SOL/ETH orderbook data
• Jupiter aggregator for DEX liquidity depth
• Replace oracle estimates with actual L2 orderbook snapshots
• Add real-time spread/imbalance monitoring

Potential Phase 1.5 (User Declined Dec 19, 2025):

• Periodic orderbook capture via 1-minute market data feed
• Would store orderbook snapshots every 1 minute in separate table (not Trade table)
• Use case: Track spread/imbalance trends over time independent of trades
• Tradeoffs: More API calls (1,440/day vs. 3-5/day), more storage, minimal benefit for Phase 1 validation
• Decision: User confirmed trade-time capture sufficient for now, periodic monitoring not needed yet

Analysis Queries

Troubleshooting

Problem: Next trade has NULL orderbook fields
Solution: Check ENV flag is true, restart container, verify code deployed

Problem: Values seem incorrect (negative depths, >100 spread)
Solution: Oracle estimates may be off - acceptable for Phase 1, will fix in Phase 2

Problem: Want to disable logging temporarily
Solution: Set ENABLE_ORDERBOOK_LOGGING=false in .env, restart container

Database Maintenance

Migration Method: Direct SQL (not Prisma migrate)

• Used due to migration history drift
• Zero data loss approach
• See git commit for ALTER TABLE statement

Prisma Sync:

DATABASE_URL="postgresql://postgres:postgres@localhost:55432/trading_bot_v4?schema=public" \
npx prisma db pull && npx prisma generate

Git History

• Commit: [TBD] "feat: Orderbook shadow logging system (Phase 1)"
• Files changed: 5 files (schema, config, types, database, execute)
• Database changes: 7 columns added via direct SQL
• Container: Rebuilt with sha256:6c88c4543ef8...

Notes

• Phase 1: Oracle-based estimates (2bps spread assumption)
• Phase 2: Real orderbook integration (Hyperliquid/Jupiter APIs)
• Data safety: Preserved production data during migration (avoided destructive reset)
• Defaults: Logging enabled by default (ENABLE_ORDERBOOK_LOGGING=true)
• Performance: Minimal overhead (estimates calculated at trade execution only)

---

Last Updated: Dec 18, 2025
System Status: ✅ PRODUCTION READY (awaiting first trade validation)