trading_bot_v4/docs/ORDERBOOK_SHADOW_LOGGING.md

Orderbook Shadow Logging System

Status: ✅ DEPLOYED (Dec 18, 2025)
Purpose: Track orderbook metrics for all trades to prepare for real orderbook integration (Phase 2)

Overview

Phase 1 implementation: Oracle-based estimates with 2bps spread assumption. Captures orderbook snapshots at trade execution for future analysis.

Database Schema

Table: Trade
New Columns (all nullable DOUBLE PRECISION):

Column	Description	Expected Range
spreadBps	Bid-ask spread in basis points	2-100 bps typical
imbalanceRatio	Order flow imbalance (-1 to 1)	Negative=sell pressure, Positive=buy
depthBid1Usd	Bid depth USD (1% from mid)	>0
depthAsk1Usd	Ask depth USD (1% from mid)	>0
priceImpact1Usd	Price impact for $1 trade	Small decimal
bidWall	Largest bid wall detected	NULL or >0
askWall	Largest ask wall detected	NULL or >0

ENV Configuration

# Enable/disable orderbook shadow logging
ENABLE_ORDERBOOK_LOGGING=true  # Default: true (omitting = defaults to true)

How to toggle:

# Disable logging
echo "ENABLE_ORDERBOOK_LOGGING=false" >> .env
docker compose restart trading-bot

# Re-enable
sed -i 's/ENABLE_ORDERBOOK_LOGGING=false/ENABLE_ORDERBOOK_LOGGING=true/' .env
docker compose restart trading-bot

Implementation Details

Files Modified:

• prisma/schema.prisma - Added 7 orderbook fields to Trade model
• config/trading.ts - Added enableOrderbookLogging boolean flag (line 127)
• types/trading.ts - Added orderbook fields to CreateTradeParams interface
• lib/database/trades.ts - Updated createTrade() to save orderbook data
• app/api/trading/execute/route.ts - Added ENV flag wrapper (lines 1037-1053)

Data Source (Phase 1): Oracle prices with 2bps spread estimates

• No real orderbook API integration yet
• Estimates sufficient for pattern analysis
• Phase 2 will integrate Hyperliquid/Jupiter for real data

Verification Queries

Check Latest Trade with Orderbook Data

SELECT 
  id, 
  symbol, 
  direction, 
  spreadBps, 
  imbalanceRatio, 
  depthBid1Usd, 
  depthAsk1Usd,
  priceImpact1Usd,
  bidWall,
  askWall,
  "createdAt"
FROM "Trade" 
WHERE spreadBps IS NOT NULL 
ORDER BY "createdAt" DESC 
LIMIT 1;

Count Trades with Orderbook Data

SELECT 
  COUNT(*) as total_trades,
  COUNT(spreadBps) as trades_with_orderbook,
  ROUND(100.0 * COUNT(spreadBps) / COUNT(*), 1) as pct_coverage
FROM "Trade";

Average Orderbook Metrics by Symbol

SELECT 
  symbol,
  COUNT(*) as trades,
  ROUND(AVG(spreadBps)::numeric, 2) as avg_spread_bps,
  ROUND(AVG(imbalanceRatio)::numeric, 3) as avg_imbalance,
  ROUND(AVG(depthBid1Usd)::numeric, 2) as avg_bid_depth,
  ROUND(AVG(depthAsk1Usd)::numeric, 2) as avg_ask_depth
FROM "Trade"
WHERE spreadBps IS NOT NULL
GROUP BY symbol
ORDER BY trades DESC;

Spread vs Win Rate Analysis

SELECT 
  CASE 
    WHEN spreadBps < 5 THEN '0-5 bps (tight)'
    WHEN spreadBps < 10 THEN '5-10 bps (normal)'
    WHEN spreadBps < 20 THEN '10-20 bps (wide)'
    ELSE '20+ bps (very wide)'
  END as spread_bucket,
  COUNT(*) as trades,
  ROUND(100.0 * COUNT(CASE WHEN "realizedPnL" > 0 THEN 1 END) / COUNT(*), 1) as win_rate,
  ROUND(AVG("realizedPnL")::numeric, 2) as avg_pnl
FROM "Trade"
WHERE spreadBps IS NOT NULL
  AND "exitReason" IS NOT NULL
GROUP BY spread_bucket
ORDER BY MIN(spreadBps);

Deployment Timeline

Date	Action	Status
Dec 18, 2025	Task 1-3: Code implementation	✅ COMPLETE
Dec 18, 2025	Task 4: ENV flag + syntax fix	✅ COMPLETE
Dec 18, 2025	Task 5: Database migration via direct SQL	✅ COMPLETE
Dec 18, 2025	Container rebuild and restart	✅ COMPLETE
Dec 18, 2025+	Production validation (await next trade)	⏳ PENDING

Production Validation Checklist

After next trade executes, verify:

• Query returns non-NULL orderbook values
• spreadBps in reasonable range (2-100 bps)
• imbalanceRatio between -1 and 1
• depthBid1Usd and depthAsk1Usd > 0
• priceImpact1Usd is small positive decimal
• bidWall/askWall NULL or positive (if detected)

Manual Test (Optional):

# Trigger test trade via Telegram
# Send: "long sol --force" or "short sol --force"
# Then query database for orderbook data

Phase 2 Roadmap

Real Orderbook Integration (Future - after Phase 1 validation):

1. Integrate Hyperliquid API for real-time orderbook snapshots
2. Integrate Jupiter API for Solana DEX orderbook aggregation
3. Replace oracle estimates with actual L2 data
4. Add orderbook depth charts to analytics UI
5. Implement smart entry based on orderbook liquidity
6. Detect and avoid low-liquidity traps
7. Optimize entry timing based on bid/ask pressure

Data Collection Goal: 50-100 trades with shadow logging before Phase 2 implementation

Troubleshooting

Problem: Next trade has NULL orderbook fields
Solution: Check ENV flag is true, restart container, verify code deployed

Problem: Values seem incorrect (negative depths, >100 spread)
Solution: Oracle estimates may be off - acceptable for Phase 1, will fix in Phase 2

Problem: Want to disable logging temporarily
Solution: Set ENABLE_ORDERBOOK_LOGGING=false in .env, restart container

Database Maintenance

Migration Method: Direct SQL (not Prisma migrate)

• Used due to migration history drift
• Zero data loss approach
• See git commit for ALTER TABLE statement

Prisma Sync:

DATABASE_URL="postgresql://postgres:postgres@localhost:55432/trading_bot_v4?schema=public" \
npx prisma db pull && npx prisma generate

Git History

• Commit: [TBD] "feat: Orderbook shadow logging system (Phase 1)"
• Files changed: 5 files (schema, config, types, database, execute)
• Database changes: 7 columns added via direct SQL
• Container: Rebuilt with sha256:6c88c4543ef8...

Notes

• Phase 1: Oracle-based estimates (2bps spread assumption)
• Phase 2: Real orderbook integration (Hyperliquid/Jupiter APIs)
• Data safety: Preserved production data during migration (avoided destructive reset)
• Defaults: Logging enabled by default (ENABLE_ORDERBOOK_LOGGING=true)
• Performance: Minimal overhead (estimates calculated at trade execution only)

---

Last Updated: Dec 18, 2025
System Status: ✅ PRODUCTION READY (awaiting first trade validation)