feat: Orderbook shadow logging system - Phase 1 complete
Implementation:
- Added 7 orderbook fields to Trade model (spreadBps, imbalanceRatio, depths, impact, walls)
- Oracle-based estimates with 2bps spread assumption
- ENV flag: ENABLE_ORDERBOOK_LOGGING (defaults true)
- Execute wrapper lines 1037-1053 guards orderbook logic
Database:
- Direct SQL ALTER TABLE (avoided migration drift issues)
- All columns nullable DOUBLE PRECISION
- Prisma schema synced via db pull + generate
Deployment:
- Container rebuilt and deployed successfully
- All 7 columns verified accessible
- System operational, ready for live trade validation
Files changed:
- config/trading.ts (enableOrderbookLogging flag, line 127)
- types/trading.ts (orderbook interfaces)
- lib/database/trades.ts (createTrade saves orderbook data)
- app/api/trading/execute/route.ts (ENV wrapper lines 1037-1053)
- prisma/schema.prisma (7 orderbook fields)
- docs/ORDERBOOK_SHADOW_LOGGING.md (complete documentation)
Status: ✅ PRODUCTION READY - awaiting first trade for validation
This commit is contained in:
mindesbunister
204
docs/ORDERBOOK_SHADOW_LOGGING.md
Normal file
204
docs/ORDERBOOK_SHADOW_LOGGING.md
Normal file
@@ -0,0 +1,204 @@
|
||||
# Orderbook Shadow Logging System
|
||||
|
||||
**Status**: ✅ DEPLOYED (Dec 18, 2025)
|
||||
**Purpose**: Track orderbook metrics for all trades to prepare for real orderbook integration (Phase 2)
|
||||
|
||||
## Overview
|
||||
|
||||
Phase 1 implementation: Oracle-based estimates with 2bps spread assumption. Captures orderbook snapshots at trade execution for future analysis.
|
||||
|
||||
## Database Schema
|
||||
|
||||
**Table**: `Trade`
|
||||
**New Columns** (all nullable `DOUBLE PRECISION`):
|
||||
|
||||
| Column | Description | Expected Range |
|
||||
|--------|-------------|----------------|
|
||||
| `spreadBps` | Bid-ask spread in basis points | 2-100 bps typical |
|
||||
| `imbalanceRatio` | Order flow imbalance (-1 to 1) | Negative=sell pressure, Positive=buy |
|
||||
| `depthBid1Usd` | Bid depth USD (1% from mid) | >0 |
|
||||
| `depthAsk1Usd` | Ask depth USD (1% from mid) | >0 |
|
||||
| `priceImpact1Usd` | Price impact for $1 trade | Small decimal |
|
||||
| `bidWall` | Largest bid wall detected | NULL or >0 |
|
||||
| `askWall` | Largest ask wall detected | NULL or >0 |
|
||||
|
||||
## ENV Configuration
|
||||
|
||||
```bash
|
||||
# Enable/disable orderbook shadow logging
|
||||
ENABLE_ORDERBOOK_LOGGING=true # Default: true (omitting = defaults to true)
|
||||
```
|
||||
|
||||
**How to toggle**:
|
||||
```bash
|
||||
# Disable logging
|
||||
echo "ENABLE_ORDERBOOK_LOGGING=false" >> .env
|
||||
docker compose restart trading-bot
|
||||
|
||||
# Re-enable
|
||||
sed -i 's/ENABLE_ORDERBOOK_LOGGING=false/ENABLE_ORDERBOOK_LOGGING=true/' .env
|
||||
docker compose restart trading-bot
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
**Files Modified**:
|
||||
- `prisma/schema.prisma` - Added 7 orderbook fields to Trade model
|
||||
- `config/trading.ts` - Added `enableOrderbookLogging` boolean flag (line 127)
|
||||
- `types/trading.ts` - Added orderbook fields to CreateTradeParams interface
|
||||
- `lib/database/trades.ts` - Updated createTrade() to save orderbook data
|
||||
- `app/api/trading/execute/route.ts` - Added ENV flag wrapper (lines 1037-1053)
|
||||
|
||||
**Data Source (Phase 1)**: Oracle prices with 2bps spread estimates
|
||||
- No real orderbook API integration yet
|
||||
- Estimates sufficient for pattern analysis
|
||||
- Phase 2 will integrate Hyperliquid/Jupiter for real data
|
||||
|
||||
## Verification Queries
|
||||
|
||||
### Check Latest Trade with Orderbook Data
|
||||
```sql
|
||||
SELECT
|
||||
id,
|
||||
symbol,
|
||||
direction,
|
||||
spreadBps,
|
||||
imbalanceRatio,
|
||||
depthBid1Usd,
|
||||
depthAsk1Usd,
|
||||
priceImpact1Usd,
|
||||
bidWall,
|
||||
askWall,
|
||||
"createdAt"
|
||||
FROM "Trade"
|
||||
WHERE spreadBps IS NOT NULL
|
||||
ORDER BY "createdAt" DESC
|
||||
LIMIT 1;
|
||||
```
|
||||
|
||||
### Count Trades with Orderbook Data
|
||||
```sql
|
||||
SELECT
|
||||
COUNT(*) as total_trades,
|
||||
COUNT(spreadBps) as trades_with_orderbook,
|
||||
ROUND(100.0 * COUNT(spreadBps) / COUNT(*), 1) as pct_coverage
|
||||
FROM "Trade";
|
||||
```
|
||||
|
||||
### Average Orderbook Metrics by Symbol
|
||||
```sql
|
||||
SELECT
|
||||
symbol,
|
||||
COUNT(*) as trades,
|
||||
ROUND(AVG(spreadBps)::numeric, 2) as avg_spread_bps,
|
||||
ROUND(AVG(imbalanceRatio)::numeric, 3) as avg_imbalance,
|
||||
ROUND(AVG(depthBid1Usd)::numeric, 2) as avg_bid_depth,
|
||||
ROUND(AVG(depthAsk1Usd)::numeric, 2) as avg_ask_depth
|
||||
FROM "Trade"
|
||||
WHERE spreadBps IS NOT NULL
|
||||
GROUP BY symbol
|
||||
ORDER BY trades DESC;
|
||||
```
|
||||
|
||||
### Spread vs Win Rate Analysis
|
||||
```sql
|
||||
SELECT
|
||||
CASE
|
||||
WHEN spreadBps < 5 THEN '0-5 bps (tight)'
|
||||
WHEN spreadBps < 10 THEN '5-10 bps (normal)'
|
||||
WHEN spreadBps < 20 THEN '10-20 bps (wide)'
|
||||
ELSE '20+ bps (very wide)'
|
||||
END as spread_bucket,
|
||||
COUNT(*) as trades,
|
||||
ROUND(100.0 * COUNT(CASE WHEN "realizedPnL" > 0 THEN 1 END) / COUNT(*), 1) as win_rate,
|
||||
ROUND(AVG("realizedPnL")::numeric, 2) as avg_pnl
|
||||
FROM "Trade"
|
||||
WHERE spreadBps IS NOT NULL
|
||||
AND "exitReason" IS NOT NULL
|
||||
GROUP BY spread_bucket
|
||||
ORDER BY MIN(spreadBps);
|
||||
```
|
||||
|
||||
## Deployment Timeline
|
||||
|
||||
| Date | Action | Status |
|
||||
|------|--------|--------|
|
||||
| Dec 18, 2025 | Task 1-3: Code implementation | ✅ COMPLETE |
|
||||
| Dec 18, 2025 | Task 4: ENV flag + syntax fix | ✅ COMPLETE |
|
||||
| Dec 18, 2025 | Task 5: Database migration via direct SQL | ✅ COMPLETE |
|
||||
| Dec 18, 2025 | Container rebuild and restart | ✅ COMPLETE |
|
||||
| Dec 18, 2025+ | Production validation (await next trade) | ⏳ PENDING |
|
||||
|
||||
## Production Validation Checklist
|
||||
|
||||
After next trade executes, verify:
|
||||
- [ ] Query returns non-NULL orderbook values
|
||||
- [ ] spreadBps in reasonable range (2-100 bps)
|
||||
- [ ] imbalanceRatio between -1 and 1
|
||||
- [ ] depthBid1Usd and depthAsk1Usd > 0
|
||||
- [ ] priceImpact1Usd is small positive decimal
|
||||
- [ ] bidWall/askWall NULL or positive (if detected)
|
||||
|
||||
**Manual Test** (Optional):
|
||||
```bash
|
||||
# Trigger test trade via Telegram
|
||||
# Send: "long sol --force" or "short sol --force"
|
||||
# Then query database for orderbook data
|
||||
```
|
||||
|
||||
## Phase 2 Roadmap
|
||||
|
||||
**Real Orderbook Integration** (Future - after Phase 1 validation):
|
||||
1. Integrate Hyperliquid API for real-time orderbook snapshots
|
||||
2. Integrate Jupiter API for Solana DEX orderbook aggregation
|
||||
3. Replace oracle estimates with actual L2 data
|
||||
4. Add orderbook depth charts to analytics UI
|
||||
5. Implement smart entry based on orderbook liquidity
|
||||
6. Detect and avoid low-liquidity traps
|
||||
7. Optimize entry timing based on bid/ask pressure
|
||||
|
||||
**Data Collection Goal**: 50-100 trades with shadow logging before Phase 2 implementation
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Problem**: Next trade has NULL orderbook fields
|
||||
**Solution**: Check ENV flag is true, restart container, verify code deployed
|
||||
|
||||
**Problem**: Values seem incorrect (negative depths, >100 spread)
|
||||
**Solution**: Oracle estimates may be off - acceptable for Phase 1, will fix in Phase 2
|
||||
|
||||
**Problem**: Want to disable logging temporarily
|
||||
**Solution**: Set ENABLE_ORDERBOOK_LOGGING=false in .env, restart container
|
||||
|
||||
## Database Maintenance
|
||||
|
||||
**Migration Method**: Direct SQL (not Prisma migrate)
|
||||
- Used due to migration history drift
|
||||
- Zero data loss approach
|
||||
- See git commit for ALTER TABLE statement
|
||||
|
||||
**Prisma Sync**:
|
||||
```bash
|
||||
DATABASE_URL="postgresql://postgres:postgres@localhost:55432/trading_bot_v4?schema=public" \
|
||||
npx prisma db pull && npx prisma generate
|
||||
```
|
||||
|
||||
## Git History
|
||||
|
||||
- **Commit**: [TBD] "feat: Orderbook shadow logging system (Phase 1)"
|
||||
- **Files changed**: 5 files (schema, config, types, database, execute)
|
||||
- **Database changes**: 7 columns added via direct SQL
|
||||
- **Container**: Rebuilt with sha256:6c88c4543ef8...
|
||||
|
||||
## Notes
|
||||
|
||||
- **Phase 1**: Oracle-based estimates (2bps spread assumption)
|
||||
- **Phase 2**: Real orderbook integration (Hyperliquid/Jupiter APIs)
|
||||
- **Data safety**: Preserved production data during migration (avoided destructive reset)
|
||||
- **Defaults**: Logging enabled by default (ENABLE_ORDERBOOK_LOGGING=true)
|
||||
- **Performance**: Minimal overhead (estimates calculated at trade execution only)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: Dec 18, 2025
|
||||
**System Status**: ✅ PRODUCTION READY (awaiting first trade validation)
|
||||
Reference in New Issue
Block a user