trading_bot_v4/docs/ORDERBOOK_SHADOW_LOGGING.md

# Orderbook Shadow Logging System

**Status**: ✅ DEPLOYED (Dec 18, 2025)  
**Purpose**: Track orderbook metrics for all trades to prepare for real orderbook integration (Phase 2)

## Overview

Phase 1 implementation: Oracle-based estimates with 2bps spread assumption. Captures orderbook snapshots at trade execution for future analysis.

## Database Schema

**Table**: `Trade`  
**New Columns** (all nullable `DOUBLE PRECISION`):

| Column | Description | Expected Range |
|--------|-------------|----------------|
| `spreadBps` | Bid-ask spread in basis points | 2-100 bps typical |
| `imbalanceRatio` | Order flow imbalance (-1 to 1) | Negative=sell pressure, Positive=buy |
| `depthBid1Usd` | Bid depth USD (1% from mid) | >0 |
| `depthAsk1Usd` | Ask depth USD (1% from mid) | >0 |
| `priceImpact1Usd` | Price impact for $1 trade | Small decimal |
| `bidWall` | Largest bid wall detected | NULL or >0 |
| `askWall` | Largest ask wall detected | NULL or >0 |

## ENV Configuration

```bash
# Enable/disable orderbook shadow logging
ENABLE_ORDERBOOK_LOGGING=true  # Default: true (omitting = defaults to true)
```

**How to toggle**:
```bash
# Disable logging
echo "ENABLE_ORDERBOOK_LOGGING=false" >> .env
docker compose restart trading-bot

# Re-enable
sed -i 's/ENABLE_ORDERBOOK_LOGGING=false/ENABLE_ORDERBOOK_LOGGING=true/' .env
docker compose restart trading-bot
```

## Implementation Details

**Files Modified**:
- `prisma/schema.prisma` - Added 7 orderbook fields to Trade model
- `config/trading.ts` - Added `enableOrderbookLogging` boolean flag (line 127)
- `types/trading.ts` - Added orderbook fields to CreateTradeParams interface
- `lib/database/trades.ts` - Updated createTrade() to save orderbook data
- `app/api/trading/execute/route.ts` - Added ENV flag wrapper (lines 1037-1053)

**Data Source (Phase 1)**: Oracle prices with 2bps spread estimates
- No real orderbook API integration yet
- Estimates sufficient for pattern analysis
- Phase 2 will integrate Hyperliquid/Jupiter for real data

## Verification Queries

### Check Latest Trade with Orderbook Data
```sql
SELECT 
  id, 
  symbol, 
  direction, 
  spreadBps, 
  imbalanceRatio, 
  depthBid1Usd, 
  depthAsk1Usd,
  priceImpact1Usd,
  bidWall,
  askWall,
  "createdAt"
FROM "Trade" 
WHERE spreadBps IS NOT NULL 
ORDER BY "createdAt" DESC 
LIMIT 1;
```

### Count Trades with Orderbook Data
```sql
SELECT 
  COUNT(*) as total_trades,
  COUNT(spreadBps) as trades_with_orderbook,
  ROUND(100.0 * COUNT(spreadBps) / COUNT(*), 1) as pct_coverage
FROM "Trade";
```

### Average Orderbook Metrics by Symbol
```sql
SELECT 
  symbol,
  COUNT(*) as trades,
  ROUND(AVG(spreadBps)::numeric, 2) as avg_spread_bps,
  ROUND(AVG(imbalanceRatio)::numeric, 3) as avg_imbalance,
  ROUND(AVG(depthBid1Usd)::numeric, 2) as avg_bid_depth,
  ROUND(AVG(depthAsk1Usd)::numeric, 2) as avg_ask_depth
FROM "Trade"
WHERE spreadBps IS NOT NULL
GROUP BY symbol
ORDER BY trades DESC;
```

### Spread vs Win Rate Analysis
```sql
SELECT 
  CASE 
    WHEN spreadBps < 5 THEN '0-5 bps (tight)'
    WHEN spreadBps < 10 THEN '5-10 bps (normal)'
    WHEN spreadBps < 20 THEN '10-20 bps (wide)'
    ELSE '20+ bps (very wide)'
  END as spread_bucket,
  COUNT(*) as trades,
  ROUND(100.0 * COUNT(CASE WHEN "realizedPnL" > 0 THEN 1 END) / COUNT(*), 1) as win_rate,
  ROUND(AVG("realizedPnL")::numeric, 2) as avg_pnl
FROM "Trade"
WHERE spreadBps IS NOT NULL
  AND "exitReason" IS NOT NULL
GROUP BY spread_bucket
ORDER BY MIN(spreadBps);
```

## Deployment Timeline

| Date | Action | Status |
|------|--------|--------|
| Dec 18, 2025 | Task 1-3: Code implementation | ✅ COMPLETE |
| Dec 18, 2025 | Task 4: ENV flag + syntax fix | ✅ COMPLETE |
| Dec 18, 2025 | Task 5: Database migration via direct SQL | ✅ COMPLETE |
| Dec 18, 2025 | Container rebuild and restart | ✅ COMPLETE |
| Dec 18, 2025+ | Production validation (await next trade) | ⏳ PENDING |

## Production Validation Checklist

After next trade executes, verify:
- [ ] Query returns non-NULL orderbook values
- [ ] spreadBps in reasonable range (2-100 bps)
- [ ] imbalanceRatio between -1 and 1
- [ ] depthBid1Usd and depthAsk1Usd > 0
- [ ] priceImpact1Usd is small positive decimal
- [ ] bidWall/askWall NULL or positive (if detected)

**Manual Test** (Optional):
```bash
# Trigger test trade via Telegram
# Send: "long sol --force" or "short sol --force"
# Then query database for orderbook data
```

## Phase 2 Roadmap

**Real Orderbook Integration** (Future - after Phase 1 validation):
1. Integrate Hyperliquid API for real-time orderbook snapshots
2. Integrate Jupiter API for Solana DEX orderbook aggregation
3. Replace oracle estimates with actual L2 data
4. Add orderbook depth charts to analytics UI
5. Implement smart entry based on orderbook liquidity
6. Detect and avoid low-liquidity traps
7. Optimize entry timing based on bid/ask pressure

**Data Collection Goal**: 50-100 trades with shadow logging before Phase 2 implementation

## Troubleshooting

**Problem**: Next trade has NULL orderbook fields  
**Solution**: Check ENV flag is true, restart container, verify code deployed

**Problem**: Values seem incorrect (negative depths, >100 spread)  
**Solution**: Oracle estimates may be off - acceptable for Phase 1, will fix in Phase 2

**Problem**: Want to disable logging temporarily  
**Solution**: Set ENABLE_ORDERBOOK_LOGGING=false in .env, restart container

## Database Maintenance

**Migration Method**: Direct SQL (not Prisma migrate)
- Used due to migration history drift
- Zero data loss approach
- See git commit for ALTER TABLE statement

**Prisma Sync**: 
```bash
DATABASE_URL="postgresql://postgres:postgres@localhost:55432/trading_bot_v4?schema=public" \
npx prisma db pull && npx prisma generate
```

## Git History

- **Commit**: [TBD] "feat: Orderbook shadow logging system (Phase 1)"
- **Files changed**: 5 files (schema, config, types, database, execute)
- **Database changes**: 7 columns added via direct SQL
- **Container**: Rebuilt with sha256:6c88c4543ef8...

## Notes

- **Phase 1**: Oracle-based estimates (2bps spread assumption)
- **Phase 2**: Real orderbook integration (Hyperliquid/Jupiter APIs)
- **Data safety**: Preserved production data during migration (avoided destructive reset)
- **Defaults**: Logging enabled by default (ENABLE_ORDERBOOK_LOGGING=true)
- **Performance**: Minimal overhead (estimates calculated at trade execution only)

---

**Last Updated**: Dec 18, 2025  
**System Status**: ✅ PRODUCTION READY (awaiting first trade validation)