User Request: Replace blind 2-hour restart timer with smart monitoring that only restarts when accountUnsubscribe errors actually occur
Changes:
. Health Monitor (NEW):
- Created lib/monitoring/drift-health-monitor.ts
- Tracks accountUnsubscribe errors in 30-second sliding window
- Triggers container restart via flag file when 50+ errors detected
- Prevents unnecessary restarts when SDK healthy
. Drift Client:
- Removed blind scheduleReconnection() and 2-hour timer
- Added interceptWebSocketErrors() to catch SDK errors
- Patches console.error to monitor for accountUnsubscribe patterns
- Starts health monitor after successful initialization
- Removed unused reconnect() method and reconnectTimer field
. Health API (NEW):
- GET /api/drift/health - Check current error count and health status
- Returns: healthy boolean, errorCount, threshold, message
- Useful for external monitoring and debugging
Impact:
- System only restarts when actual memory leak detected
- Prevents unnecessary downtime every 2 hours
- More targeted response to SDK issues
- Better operational stability
Files:
- lib/monitoring/drift-health-monitor.ts (NEW - 165 lines)
- lib/drift/client.ts (removed timer, added error interception)
- app/api/drift/health/route.ts (NEW - health check endpoint)
Testing:
- Health monitor starts on initialization: ✅
- API endpoint returns healthy status: ✅
- No blind reconnection scheduled: ✅
Changes:
- Updated roadmap status from 'planned' to 'complete'
- Added checkmarks for implemented features:
✅ Position Manager tracks MFE/MAE every 2 seconds
✅ Database stores maxFavorableExcursion and maxAdverseExcursion
✅ Analytics dashboard displays avg MFE/MAE per indicator version
✅ Version comparison shows MFE/MAE trends
✅ Optimization API analyzes MFE vs TP1 rate
- Added future enhancement note for distribution charts
Evidence:
- Position Manager: lib/trading/position-manager.ts (lines 53-55, 140, 1127+)
- Database: Trade model with MFE/MAE fields
- Analytics: app/analytics/page.tsx (lines 77-78, 566-579, 654-655)
- Optimization API: app/api/optimization/analyze/route.ts
User Request: 'i think we already have this implemented?'
Confirmed: MAE/MFE tracking is fully operational
TypeScript build error: qualityScore not in interface
Fix: Added qualityScore?: number to ExecuteTradeResponse type
Files Modified:
- app/api/trading/execute/route.ts (interface update)
User Request: Show quality score in Telegram when position opened
Changes:
- Updated execute endpoint response to include qualityScore field
- n8n workflow already checks for qualityScore in response
- When present, displays: ⭐ Quality: XX/100
Impact:
- Users now see quality score immediately on position open
- Previously only saw score on blocked signals
- Better visibility into trade quality at entry
Files Modified:
- app/api/trading/execute/route.ts (added qualityScore to response)
- Fixed tp1Hit/tp2Hit -> tp1Filled/tp2Filled in Runner Performance query
- Fixed atr -> atrAtEntry in ATR vs MFE Correlation and Data Collection queries
- Added Analytics card to homepage with link to /analytics/optimization
- Added Home button to optimization page header
- All 7 analyses now working without SQL errors
- Created /api/optimization/analyze endpoint with 7 SQL analyses
- Replaced old TP/SL page with comprehensive dashboard
- Analyses: Quality Score Distribution, Direction Performance, Blocked Signals, Runner Performance, ATR vs MFE, Indicator Versions, Data Collection Status
- Real-time refresh capability
- Actionable recommendations based on data thresholds
- Roadmap links at bottom
- Addresses user request for automated SQL analysis dashboard
- Added MIN_SIGNAL_QUALITY_SCORE_LONG and _SHORT fields to Settings interface
- Replaced single quality score field with three fields:
1. Global Fallback (91) - for BTC and other symbols
2. LONG Signals (90) - based on 71.4% WR data analysis
3. SHORT Signals (95) - based on toxic 28.6% WR data, blocks low-quality shorts
- Updated app/api/settings/route.ts GET/POST handlers to support direction-specific fields
- Fixed field naming consistency (MIN_SIGNAL_QUALITY_SCORE vs MIN_QUALITY_SCORE)
- User can now adjust direction-specific thresholds via settings UI without .env editing
- Container deployed: 2025-11-23T14:25:34 UTC
Critical Bug Fix:
- archivedVersions was used before declaration (line 147 vs line 165)
- Caused 'Cannot access before initialization' error
- Moved versionDescriptions and archivedVersions declarations to top
- Now defined BEFORE usage in resultsWithArchived.map()
Impact: Analytics page was completely broken (stuck on loading)
Resolution: API now returns data correctly, UI functional
Error: ReferenceError: Cannot access 'g' before initialization
Fix: Proper variable ordering in route.ts
**BUG:** Telegram 'short sol' blocked by multi-timeframe data collection filter
- Filter checked 'timeframe !== 5' which blocked 'manual' timeframe
- Manual trades from Telegram should execute, not be saved for analysis
**FIX:** Updated condition to 'timeframe !== 5 && timeframe !== manual'
- Allows both 5min TradingView signals AND manual Telegram trades
- Only blocks 15min/1H/4H/Daily for data collection
**FILES:** app/api/trading/execute/route.ts line 114
**DEPLOYED:** Nov 20, 2025 15:42 CET
Stats API was recalculating P&L from entry/exit prices which didn't
account for TP1+runner partial closes. This caused incorrect P&L
display (-$26.10 instead of +$46.97).
Fixed:
- Use database realizedPnL (now corrected to match Drift UI)
- Added debug logging to show trade count and total
- Stats now correctly show v8 performance: +$46.97
Note: Database P&L values were corrected in previous commit (cd6f590)
to match Drift UI's actual TP1+runner close values.
- Query Drift Protocol's cumulativeDeposits for ground truth
- Discovered user deposited ,440.61 (not hardcoded 46)
- Previous withdrawals: ~26.78 (not tracked in ENV)
- Database P&L -.01 is correct
- Reconciliation: ,440.61 deposits - .01 P&L - 26.78 withdrawn = 11.82 current
- Available profit now shows current balance (includes all trading results)
- Created /api/drift/account-summary endpoint for account reconciliation
- Statistics now mathematically consistent and trustworthy
- Fixed LAST_WITHDRAWAL_TIME type (null | string)
- Removed parseFloat on health.freeCollateral (already number)
- Fixed getDriftClient() → getClient() method name
- Build now compiles successfully
Deployed: Withdrawal system now live on dashboard
Implemented comprehensive price tracking for multi-timeframe signal analysis.
**Components Added:**
- lib/analysis/blocked-signal-tracker.ts - Background job tracking prices
- app/api/analytics/signal-tracking/route.ts - Status/metrics endpoint
**Features:**
- Automatic price tracking at 1min, 5min, 15min, 30min intervals
- TP1/TP2/SL hit detection using ATR-based targets
- Max favorable/adverse excursion tracking (MFE/MAE)
- Analysis completion after 30 minutes
- Background job runs every 5 minutes
- Entry price captured from signal time
**Database Changes:**
- Added entryPrice field to BlockedSignal (for price tracking baseline)
- Added maxFavorablePrice, maxAdversePrice fields
- Added maxFavorableExcursion, maxAdverseExcursion fields
**Integration:**
- Auto-starts on container startup
- Tracks all DATA_COLLECTION_ONLY signals
- Uses same TP/SL calculation as live trades (ATR-based)
- Calculates profit % based on direction (long vs short)
**API Endpoints:**
- GET /api/analytics/signal-tracking - View tracking status and metrics
- POST /api/analytics/signal-tracking - Manually trigger update (auth required)
**Purpose:**
Enables data-driven multi-timeframe comparison. After 50+ signals per
timeframe, can analyze which timeframe (5min vs 15min vs 1H vs 4H vs Daily)
has best win rate, profit potential, and signal quality.
**What It Tracks:**
- Price at 1min, 5min, 15min, 30min after signal
- Would TP1/TP2/SL have been hit?
- Maximum profit/loss during 30min window
- Complete analysis of signal profitability
**How It Works:**
1. Signal comes in (15min, 1H, 4H, Daily) → saved to BlockedSignal
2. Background job runs every 5min
3. Queries current price from Pyth
4. Calculates profit % from entry
5. Checks if TP/SL thresholds crossed
6. Updates MFE/MAE if new highs/lows
7. After 30min, marks analysisComplete=true
**Future Analysis:**
After 50+ signals per timeframe:
- Compare TP1 hit rates across timeframes
- Identify which timeframe has highest win rate
- Determine optimal signal frequency vs quality trade-off
- Switch production to best-performing timeframe
User requested: "i want all the bells and whistles. lets make the
powerhouse more powerfull. i cant see any reason why we shouldnt"
Settings UI was using wrong variable name (MIN_QUALITY_SCORE) while
code reads MIN_SIGNAL_QUALITY_SCORE. This caused quality score changes
in settings UI to have no effect.
Fixed:
- Settings API now reads/writes MIN_SIGNAL_QUALITY_SCORE
- Updated .env file to use correct variable name
- User's quality score increase to 81 will now work
Related: User increased min quality from 60 to 81 to filter out
small chop trades (avoiding -$99 trade with quality score 80).
- TypeScript build error: currentPrice not in interface
- Correct field name is signalPrice (already defined)
- Fixes multi-timeframe data collection compilation
- Only 5min signals execute trades (production)
- 15min/1H/4H/Daily signals saved to BlockedSignal table for analysis
- Enables cross-timeframe performance comparison
- Zero financial risk - non-5min signals just collect data
- blockReason: 'DATA_COLLECTION_ONLY' for easy filtering
- Returns HTTP 200 (not 400) since this is expected behavior
- Prepares for future timeframe optimization decisions
- Moved positionManager.addTrade() to AFTER database save succeeds
- Changed database error handling to return HTTP 500 (not silent fail)
- Test endpoint now enforces same pattern as execute endpoint
- Prevents untracked positions when database save fails
- Root cause of trade manual-1763391075992 compounding to -19.43
Before: Test endpoint added to Position Manager first, saved to DB after
After: Test endpoint saves to DB first, only adds to PM if DB succeeds
Impact: No more untracked positions from test trades with failed DB saves
- Created /api/trading/place-exit-orders endpoint
- Created restore-orders.mjs script
- Issue: Next.js creates separate Drift instances per route
- Workaround: Use /api/trading/cancel-orders to remove orphaned orders
Current situation:
- 32 orphaned orders existed and were cancelled
- Position Manager should auto-place new orders
- Manual order placement endpoint needs refactoring
**The 4 Loss Problem:**
Multiple trades today opened opposite positions before previous closed:
- 11:15 SHORT manual close
- 11:21 LONG opened + hit SL (-.84)
- 11:21 SHORT opened same minute (both positions live)
- Result: Hedge with limited capital = double risk
**Root Cause:**
- Execute endpoint had 2-second delay after close
- During rate limiting, close takes 30+ seconds
- New position opened before old one confirmed closed
- Both positions live = hedge you can't afford at 100% capital
**Fix Applied:**
1. Block flip if close fails (don't open new position)
2. Wait for Drift confirmation (up to 15s), not just tx confirmation
3. Poll Drift every 2s to verify position actually closed
4. Only proceed with new position after verified closure
5. Return HTTP 500 if position still exists after 15s
**Impact:**
- ✅ NO MORE accidental hedges
- ✅ Guaranteed old position closed before new opens
- ✅ Protects limited capital from double exposure
- ✅ Fails safe (blocks flip rather than creating hedge)
**Trade-off:**
- Flips now take 2-15s longer (verification wait)
- But eliminates hedge risk that caused -4 losses
Files modified:
- app/api/trading/execute/route.ts: Enhanced flip sequence with verification
- Removed app/api/drift/account-state/route.ts (had TypeScript errors)
- Identified 25 trades with corrupted P&L from compounding bug
- Recalculated correct P&L for all affected trades (7 days)
- Total corrections: 63.23 in fake profits removed
- Today's actual P&L: -.97 (11 trades) - was showing -9.87 before fixes
- Last 7 days actual P&L: +7.04 (70 trades) - was showing +27.90
Bug pattern:
- P&L updated multiple times during close verification wait
- Each update added to previous value instead of replacing
- Worst case: -8.17 stored vs -.77 actual (7× compounding)
Root cause: Common Pitfall #48 - closingInProgress flag fix deployed
Nov 16 18:26 UTC but damage already done to historical data
Files modified:
- Database: 25 trades updated via SQL recalculation
- Fixed trades from Nov 10-16 with ABS(stored - calculated) > $1
- Modified /api/analytics/last-trade to extract currentSize from configSnapshot.positionManagerState
- For open positions (exitReason === null), displays runner size after TP1 instead of original positionSizeUSD
- Falls back to original positionSizeUSD for closed positions or when Position Manager state unavailable
- Fixes UI showing 2.54 when actual runner is 2.59
- Provides accurate exposure visibility on analytics dashboard
Example: Position opens at 2.54, TP1 closes 70% → runner 2.59 → UI now shows 2.59
- Memory leak identified: Drift SDK accumulates WebSocket subscriptions over time
- Root cause: accountUnsubscribe errors pile up when connections close/reconnect
- Symptom: Heap grows to 4GB+ after 10+ hours, eventual OOM crash
- Solution: Automatic reconnection every 4 hours to clear subscriptions
Changes:
- lib/drift/client.ts: Add reconnectTimer and scheduleReconnection()
- lib/drift/client.ts: Implement private reconnect() method
- lib/drift/client.ts: Clear timer in disconnect()
- app/api/drift/reconnect/route.ts: Manual reconnection endpoint (POST)
- app/api/drift/reconnect/route.ts: Reconnection status endpoint (GET)
Impact:
- Prevents JavaScript heap out of memory crashes
- Telegram bot timeouts resolved (was failing due to unresponsive bot)
- System will auto-heal every 4 hours instead of requiring manual restart
- Emergency manual reconnect available via API if needed
Tested: Container restarted successfully, no more WebSocket accumulation expected
- Set signalSource='manual' for Telegram trades, 'tradingview' for TradingView
- Updated analytics queries to exclude manual trades from indicator analysis
- getTradingStats() filters manual trades (TradingView performance only)
- Version comparison endpoint filters manual trades
- Created comprehensive filtering guide: docs/MANUAL_TRADE_FILTERING.md
- Ensures clean data for indicator optimization without contamination
- Alchemy Growth (10,000 CU/s) can handle longer confirmation waits
- Increased timeout from 30s to 60s in both openPosition() and closePosition()
- Added debug logging to execute endpoint to trace hang points
- Configured dual RPC: Alchemy primary (transactions), Helius fallback (subscriptions)
- Previous 30s timeout was causing premature failures during Solana congestion
- This should resolve 'Transaction was not confirmed in 30.00 seconds' errors
Related: User reported n8n webhook returning 500 with timeout error
- Changed 'pricePosition' to 'pricePositionAtEntry' in extreme positions query
- Fixed database error: column "pricePosition" does not exist
Context:
- API was failing with Error 42703 (column not found)
- Database schema uses 'pricePositionAtEntry', not 'pricePosition'
- Version comparison section now loads correctly in analytics dashboard
- Fixed extremePositionStats type to match actual SQL query fields
- Changed .count to .trades (query returns 'trades' column, not 'count')
- Simplified extreme positions metrics (removed missing avg_adx and weak_adx_count)
- Fixed version comparison fallback from 'v1' to 'unknown'
Technical:
- SQL query only returns: version, trades, wins, total_pnl, avg_quality_score
- Code was trying to access non-existent fields causing TypeScript errors
- Build now succeeds, container deployed
- Changed SQL queries to use indicatorVersion (TradingView strategy versions)
- Updated version descriptions to only show v5/v6/unknown
- v5 = Buy/Sell Signal strategy (pre-Nov 12)
- v6 = HalfTrend + BarColor strategy (Nov 12+)
- unknown = Pre-version-tracking trades
Context:
- User clarified: 'v4 is v6. the version reflects the moneyline version'
- Dashboard should show indicator strategy versions, not scoring logic versions
- Added dynamicATRAnalysis section to /api/analytics/tp-sl-optimization
- Analyzes v6 trades with ATR data to compare fixed vs dynamic targets
- Dynamic targets: TP2=2x ATR, SL=1.5x ATR (from config)
- Shows +39.8% advantage with 14 trades (.72 improvement)
- Includes data sufficiency check (need 30+ trades)
- Recommendation logic: WAIT/IMPLEMENT/CONSIDER/NEUTRAL based on sample size and advantage
- Returns detailed metrics: sample size, avg ATR, hit rates, P&L comparison
- Integrates seamlessly with existing MAE/MFE analysis
Current status: 14/30 trades collected, insufficient for implementation
Expected: Frontend will display this data to track progress toward 30-trade threshold
CRITICAL FIX: Previous implementation showed incorrect price movements
(100% instead of 0.2%) because currentPrice wasn't available in
check-risk endpoint.
Changes:
- app/api/trading/check-risk/route.ts: Fetch current price from Pyth
price monitor before quality scoring
- lib/trading/signal-quality.ts: Added validation and detailed logging
- Check if currentPrice available, apply penalty if missing
- Log actual prices: $X → $Y = Z%
- Include prices in penalty/allowance messages
Example outputs:
Flip-flop in tight range: 4min ago, only 0.20% move ($143.86 → $143.58) (-25 pts)
Direction change after 10.2% move ($170.00 → $153.00, 12min ago) - reversal allowed
This fixes the false positive that allowed a 0.2% flip-flop earlier today.
Deployed: 09:42 CET Nov 14, 2025
Improved flip-flop penalty logic to distinguish between:
- Chop (bad): <2% price move from opposite signal → -25 penalty
- Reversal (good): ≥2% price move from opposite signal → allowed
Changes:
- lib/database/trades.ts: getRecentSignals() now returns oppositeDirectionPrice
- lib/trading/signal-quality.ts: Added currentPrice parameter, price movement check
- app/api/trading/check-risk/route.ts: Added currentPrice to RiskCheckRequest interface
- app/api/trading/execute/route.ts: Pass openResult.fillPrice as currentPrice
- app/api/analytics/reentry-check/route.ts: Pass currentPrice from metrics
Example scenarios:
- ETH $170 SHORT → $153 LONG (10% move) = reversal allowed ✅
- ETH $154.50 SHORT → $154.30 LONG (0.13% move) = chop blocked ⚠️
Deployed: 09:18 CET Nov 14, 2025
Container: trading-bot-v4
PHASE 1 IMPLEMENTATION:
Signal quality scoring now checks database for recent trading patterns
and applies penalties to prevent overtrading and flip-flop losses.
NEW PENALTIES:
1. Overtrading: 3+ signals in 30min → -20 points
- Detects consolidation zones where system generates excessive signals
- Counts both executed trades AND blocked signals
2. Flip-flop: Opposite direction in last 15min → -25 points
- Prevents rapid long→short→long whipsaws
- Example: SHORT at 10:00, LONG at 10:12 = blocked
3. Alternating pattern: Last 3 trades flip directions → -30 points
- Detects choppy market conditions
- Pattern like long→short→long = system getting chopped
DATABASE INTEGRATION:
- New function: getRecentSignals() in lib/database/trades.ts
- Queries last 30min of trades + blocked signals
- Checks last 3 executed trades for alternating pattern
- Zero performance impact (fast indexed queries)
ARCHITECTURE:
- scoreSignalQuality() now async (requires database access)
- All callers updated: check-risk, execute, reentry-check
- skipFrequencyCheck flag available for special cases
- Frequency penalties included in qualityResult breakdown
EXPECTED IMPACT:
- Eliminate overnight flip-flop losses (like SOL $141-145 chop)
- Reduce overtrading during sideways consolidation
- Better capital preservation in non-trending markets
- Should improve win rate by 5-10% by avoiding worst setups
TESTING:
- Deploy and monitor next 5 signals in choppy markets
- Check logs for frequency penalty messages
- Analyze if blocked signals would have been losers
Files changed:
- lib/database/trades.ts: Added getRecentSignals()
- lib/trading/signal-quality.ts: Made async, added frequency checks
- app/api/trading/check-risk/route.ts: await + symbol parameter
- app/api/trading/execute/route.ts: await + symbol parameter
- app/api/analytics/reentry-check/route.ts: await + skipFrequencyCheck
- Auto-close phantom positions immediately via market order
- Return HTTP 200 (not 500) to allow n8n workflow continuation
- Save phantom trades to database with full P&L tracking
- Exit reason: 'manual' category for phantom auto-closes
- Protects user during unavailable hours (sleeping, no phone)
- Add Docker build best practices to instructions (background + tail)
- Document phantom system as Critical Component #1
- Add Common Pitfall #30: Phantom notification workflow
Why auto-close:
- User can't always respond to phantom alerts
- Unmonitored position = unlimited risk exposure
- Better to exit with small loss/gain than leave exposed
- Re-entry possible if setup actually good
Files changed:
- app/api/trading/execute/route.ts: Auto-close logic
- .github/copilot-instructions.md: Documentation + build pattern
Root Cause:
- Execute endpoint saved to database AFTER adding to Position Manager
- Database save failures were silently caught and ignored
- API returned success even when DB save failed
- Container restarts lost in-memory Position Manager state
- Result: Unprotected positions with no TP/SL monitoring
Fixes Applied:
1. Database-First Pattern (app/api/trading/execute/route.ts):
- MOVED createTrade() BEFORE positionManager.addTrade()
- If database save fails, return HTTP 500 with critical error
- Error message: 'CLOSE POSITION MANUALLY IMMEDIATELY'
- Position Manager only tracks database-persisted trades
- Ensures container restarts can restore all positions
2. Transaction Timeout (lib/drift/orders.ts):
- Added 30s timeout to confirmTransaction() in closePosition()
- Prevents API from hanging during network congestion
- Uses Promise.race() pattern for timeout enforcement
3. Telegram Error Messages (telegram_command_bot.py):
- Parse JSON for ALL responses (not just 200 OK)
- Extract detailed error messages from 'message' field
- Shows critical warnings to user immediately
- Fail-open: proceeds if analytics check fails
4. Position Manager (lib/trading/position-manager.ts):
- Move lastPrice update to TOP of monitoring loop
- Ensures /status endpoint always shows current price
Verification:
- Test trade cmhxj8qxl0000od076m21l58z executed successfully
- Database save completed BEFORE Position Manager tracking
- SL triggered correctly at -$4.21 after 15 minutes
- All protection systems working as expected
Impact:
- Eliminates risk of unprotected positions
- Provides immediate critical warnings if DB fails
- Enables safe container restarts with full position recovery
- Verified with live test trade on production
See: CRITICAL_INCIDENT_UNPROTECTED_POSITION.md for full incident report
