diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 06d3f72..4999c22 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -3420,6 +3420,110 @@ This section contains the **TOP 10 MOST CRITICAL** pitfalls that every AI agent - **Git commit:** [Health monitoring deployed Dec 8, 2025 - detects missing orders] - **Status:** ⚠️ Health monitor deployed (detects issue), root cause fix pending +79. **CRITICAL: Smart Validation Queue Never Monitors - In-Memory Queue Lost on Container Restart (CRITICAL - Dec 9, 2025):** + - **Symptom:** Quality 50-89 signals blocked and saved to database, but validation queue never monitors them for price confirmation + - **User Report:** "the smart validation system should have entered the trade as it shot up shouldnt it?" + - **Financial Impact:** Missed +$18.56 manual entry (SOL-PERP LONG quality 85, price moved +1.21% in 1 minute = 4× the +0.3% confirmation threshold) + - **Real Incident (Dec 9, 2025 15:40):** + * Signal: cmiyqy6uf03tcn30722n02lnk + * Quality: 85/90 (blocked correctly per thresholds) + * Entry Price: $134.94 + * Price after 1min: $136.57 (+1.21%) + * Confirmation threshold: +0.3% + * System should have: Queued → Monitored → Entered at confirmation + * What happened: Signal saved to database, queue NEVER monitored it + * User had to manually enter → +$18.56 profit + - **Root Cause #1 - In-Memory Queue Lost on Restart:** + * File: `lib/trading/smart-validation-queue.ts` + * Queue uses `Map` in-memory storage + * BlockedSignal records saved to PostgreSQL ✅ + * But queue Map is empty after container restart ❌ + * startSmartValidation() just created empty singleton, never loaded from database + - **Root Cause #2 - Production Logger Silencing:** + * logger.log() calls silenced when NODE_ENV=production + * File: `lib/utils/logger.ts` - logger.log() only works in dev mode + * Startup messages never appeared in container logs + * Silent failure - no errors, no indication queue was empty + - **THE FIX (Dec 9, 2025 - DEPLOYED):** + ```typescript + // In lib/trading/smart-validation-queue.ts startSmartValidation() + export async function startSmartValidation(): Promise { + const queue = getSmartValidationQueue() + + // Query BlockedSignal table for signals within 30-minute entry window + const thirtyMinutesAgo = new Date(Date.now() - 30 * 60 * 1000) + + const recentBlocked = await prisma.blockedSignal.findMany({ + where: { + blockReason: 'QUALITY_SCORE_TOO_LOW', + signalQualityScore: { gte: 50, lt: 90 }, // Marginal quality range + createdAt: { gte: thirtyMinutesAgo }, + }, + }) + + console.log(`🔄 Restoring ${recentBlocked.length} pending signals from database`) + + // Re-queue each signal with original parameters + for (const signal of recentBlocked) { + await queue.addSignal({ /* signal params */ }) + } + + console.log(`✅ Smart validation restored ${recentBlocked.length} signals, monitoring started`) + } + ``` + - **Why Both Fixes Were Needed:** + 1. Database restoration: Load pending signals from PostgreSQL on startup + 2. console.log(): Replace logger.log() calls with console.log() for production visibility + 3. Without #1: Queue always empty after restart + 4. Without #2: Couldn't debug why queue was empty (no logs) + - **Expected Behavior After Fix:** + * Container restart: Queries database for signals within 30-minute window + * Signals found: Re-queued with original entry price, quality score, metrics + * Monitoring starts: 30-second price checks begin immediately + * Logs show: "🔄 Restoring N pending signals from database" + * Confirmation: "👁️ Smart validation monitoring started (checks every 30s)" + - **Verification Commands:** + ```bash + # Check startup logs + docker logs trading-bot-v4 2>&1 | grep -E "(Smart|validation|Restor)" + + # Expected output: + # 🧠 Starting smart entry validation system... + # 🔄 Restoring N pending signals from database + # ✅ Smart validation restored N signals, monitoring started + # 👁️ Smart validation monitoring started (checks every 30s) + + # If N=0, check database for recent signals + docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c \ + "SELECT COUNT(*) FROM \"BlockedSignal\" WHERE \"blockReason\" = 'QUALITY_SCORE_TOO_LOW' + AND \"signalQualityScore\" BETWEEN 50 AND 89 + AND \"createdAt\" > NOW() - INTERVAL '30 minutes';" + ``` + - **Why This Matters:** + * **This is a REAL MONEY system** - validation queue is designed to catch marginal signals that confirm + * Quality 50-89 signals = borderline setups that need price confirmation + * Without validation: Miss profitable confirmed moves (like +$18.56 opportunity) + * System appeared to work (signals blocked correctly, saved to database) + * But critical validation step never executed (queue empty, monitoring off) + - **Prevention Rules:** + 1. NEVER use in-memory-only data structures for critical financial logic + 2. ALWAYS restore state from database on startup for trading systems + 3. ALWAYS use console.log() for critical startup messages (not logger.log()) + 4. ALWAYS verify monitoring actually started (check logs for confirmation) + 5. Add database restoration for ANY system that queues/monitors signals + 6. Test container restart scenarios to catch in-memory state loss + - **Red Flags Indicating This Bug:** + * BlockedSignal records exist in database but no queue monitoring logs + * Price moves meet confirmation threshold but no execution + * User manually enters trades that validation queue should have handled + * No "👁️ Smart validation check: N pending signals" logs every 30 seconds + * Telegram shows "⏰ SIGNAL QUEUED FOR VALIDATION" but nothing after + - **Files Changed:** + * lib/trading/smart-validation-queue.ts (Lines 456-500, 137-175, 117-127) + - **Git commit:** 2a1badf "critical: Fix Smart Validation Queue - restore signals from database on startup" + - **Deploy Status:** ✅ DEPLOYED Dec 9, 2025 17:07 CET + - **Status:** ✅ Fixed - Queue now restores pending signals on startup, production logging enabled + 72. **CRITICAL: MFE Data Unit Mismatch - ALWAYS Filter by Date (CRITICAL - Dec 5, 2025):** - **Symptom:** SQL analysis shows "20%+ average MFE" but TP1 (0.6% target) never hits - **Root Cause:** Old Trade records stored MFE/MAE in DOLLARS, new records store PERCENTAGES