From 4239c99057f53925978d4851d0a33c7084285337 Mon Sep 17 00:00:00 2001 From: mindesbunister Date: Mon, 1 Dec 2025 23:51:40 +0100 Subject: [PATCH] docs: Add Common Pitfall #66 - Smart Entry Validation Queue symbol normalization bug MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Symptom: Abandonment notifications showing impossible prices (26 → 8.18 in 30s) - Root cause: Symbol format mismatch (TradingView 'SOLUSDT' vs cache 'SOL-PERP') - Fix: Added normalizeTradingViewSymbol() in check-risk endpoint before validation queue - Impact: Cache lookup now succeeds, Telegram shows correct abandonment prices - Files: check-risk/route.ts line 9 (import), lines 432-444 (normalization) - Commit: 6cec2e8 deployed Dec 1, 2025 - Lesson: Always normalize symbols at integration boundaries, cache key mismatches fail silently --- .github/copilot-instructions.md | 82 +++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 607b43f..8dcd2e1 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -4839,6 +4839,88 @@ trade.realizedPnL += actualRealizedPnL // NOT: result.realizedPnL from SDK - **Impact:** Blocked 45 minutes of distributed work, wasted cluster compute, caught before Stage 1 analysis - **Files changed:** `cluster/distributed_worker.py` lines 67-77 +66. **Smart Entry Validation Queue wrong price display (CRITICAL - Fixed Dec 1, 2025):** + - **Symptom:** Abandonment notifications in Telegram showing incorrect/impossible prices (e.g., $126.00 → $98.18 = -22.08% drop in 30 seconds) + - **User Report:** "the abandonend signal shows the wrong price. it never went that low" + - **Root Cause:** Symbol format mismatch between validation queue storage and market data cache lookup + * TradingView webhook sends: `"SOLUSDT"`, `"ETHUSDT"`, etc. + * check-risk endpoint passed: `body.symbol` directly to validation queue without normalization + * Validation queue stored: `signal.symbol = "SOLUSDT"` + * Market data cache uses: Normalized Drift format `"SOL-PERP"`, `"ETH-PERP"` + * Cache lookup: `marketDataCache.get("SOLUSDT")` returns `null` (key not found) + * Result: No cached data, wrong/stale price used in Telegram abandonment notification + - **Real Incident (Dec 1, 2025):** + * Signal queued at $126.00 + * Abandonment notification showed $98.18 + * Reported drop: -22.08% in 30 seconds (physically impossible for SOL) + * Actual: Cache lookup failed due to symbol format, displayed stale/wrong price + - **Impact:** Every Smart Entry Validation Queue abandonment notification shows incorrect price, misleading user about system behavior and market conditions + - **Data Flow Before Fix:** + 1. TradingView → check-risk: `{"symbol": "SOLUSDT", ...}` + 2. check-risk → validation queue: `symbol: "SOLUSDT"` (no normalization) + 3. Validation queue stores: `signal.symbol = "SOLUSDT"` + 4. Validation loop: `marketDataCache.get("SOLUSDT")` → **null** (key mismatch) + 5. Telegram notification: Shows wrong/stale price + - **Data Flow After Fix:** + 1. TradingView → check-risk: `{"symbol": "SOLUSDT", ...}` + 2. check-risk normalizes: `const normalizedSymbol = normalizeTradingViewSymbol("SOLUSDT")` → `"SOL-PERP"` + 3. check-risk → validation queue: `symbol: "SOL-PERP"` (normalized) + 4. Validation loop: `marketDataCache.get("SOL-PERP")` → **success** (key matches) + 5. Telegram notification: Shows correct current price from cache + - **Fix (app/api/trading/check-risk/route.ts):** + ```typescript + // Line 9: Added import + import { normalizeTradingViewSymbol } from '@/config/trading' + + // Lines 432-444: Normalize symbol before validation queue + // CRITICAL FIX (Dec 1, 2025): Normalize TradingView symbol format to Drift format + // Bug: Market data cache uses "SOL-PERP" but TradingView sends "SOLUSDT" + // Without normalization, validation queue can't find matching price data + // Result: Wrong/stale price shown in Telegram abandonment notifications + const normalizedSymbol = normalizeTradingViewSymbol(body.symbol) + + const queued = await validationQueue.addSignal({ + blockReason: 'QUALITY_SCORE_TOO_LOW', + symbol: normalizedSymbol, // Use normalized format for cache lookup + direction: body.direction, + originalPrice: currentPrice, + qualityScore: qualityScore.score, + // ... other parameters + }) + + // Line 458: Updated console.log + console.log(`🧠 Signal queued for smart validation: ${normalizedSymbol} ${body.direction}...`) + ``` + - **Files Changed:** + * `app/api/trading/check-risk/route.ts` (line 9 import, lines 432-444 normalization, line 458 log) + * Total: 6 insertions including comment block + - **Related Components (No Changes Needed):** + * `lib/trading/smart-validation-queue.ts` - Receives normalized symbol, validation logic correct + * `lib/trading/market-data-cache.ts` - Uses normalized format "SOL-PERP" (line 11 comment) + * `lib/notifications/telegram.ts` - Displays whatever price passed, formatter correct + * `app/api/trading/market-data/route.ts` - Already normalizes correctly (reference implementation) + * `config/trading.ts` - Contains `normalizeTradingViewSymbol()` function at line 238 + - **Git Commit:** 6cec2e8 "critical: Fix Smart Entry Validation Queue wrong price display" + - **Deployed:** Dec 1, 2025, 23:45:21 +0100 (container trading-bot-v4 restarted with fix) + - **Verification Required:** + * Wait for next marginal quality signal (quality 50-89) to be queued + * Monitor validation queue logs: Should show normalized symbol format ("SOL-PERP" not "SOLUSDT") + * Check Telegram abandonment: Should show correct price matching market reality + * Verify cache lookup succeeds: Console logs "✅ Using fresh TradingView data for SOL-PERP" + - **Lessons Learned:** + 1. **Symbol format consistency is CRITICAL across all system components** - One endpoint without normalization breaks entire data flow + 2. **Cache key mismatches fail silently** - `marketDataCache.get("WRONG-KEY")` returns `null`, no error thrown + 3. **Always normalize at integration boundaries** - External webhooks (TradingView) → Internal systems (validation queue, cache) require format conversion + 4. **Market data cache uses Drift format as standard** - "SOL-PERP", "ETH-PERP", "BTC-PERP" (not TradingView "SOLUSDT", "ETHUSDT") + 5. **ALL endpoints feeding validation queue must normalize** - check-risk, execute, market-data all need consistent symbol handling + 6. **Verify data flow end-to-end** - Trace from webhook → storage → lookup → display to catch format mismatches + 7. **Silent failures cause data corruption downstream** - Wrong price displayed misleads user about system behavior + - **Why This Matters:** + * Smart Entry Validation Queue is profit recovery system for marginal quality signals (expected +$1,823/month) + * Wrong abandonment prices mislead user about system decisions (appears system abandoned profitable setups) + * In real money trading, user relies on accurate notifications for manual intervention decisions + * Cache lookups are critical path for all validation queue price checks (every 30 seconds) + ## File Conventions - **API routes:** `app/api/[feature]/[action]/route.ts` (Next.js 15 App Router)