diff --git a/docs/QUALITY_THRESHOLD_VALIDATION.md b/docs/QUALITY_THRESHOLD_VALIDATION.md new file mode 100644 index 0000000..aa75063 --- /dev/null +++ b/docs/QUALITY_THRESHOLD_VALIDATION.md @@ -0,0 +1,187 @@ +# Quality Threshold Validation Plan + +## Overview +System currently blocks signals with quality score <91. First validation shows potential false negative (quality 80 signal would have profited +0.52%). Need data to determine if threshold is optimal or too restrictive. + +## Discovery (Nov 22, 2025) + +**User Observation:** "Green dots shot up" - TradingView Money Line signals blocked by system moved favorably + +**Signal 1 Analysis (Nov 21, 16:50):** +- **Blocked:** Quality 80 (threshold 91) +- **Primary reason:** ADX 16.6 (weak trend, needs 18+) +- **Entry:** $126.20 +- **Peak:** $126.86 within 1 minute +- **Profit:** +0.52% (+$43 on $8,300 position) +- **TP1 Target:** +1.51% (would NOT have hit) +- **Result:** FALSE NEGATIVE - blocked a profitable signal + +**Signal 2 (Nov 21, 23:00):** +- **Blocked:** Quality 70 (threshold 91) +- **Primary reason:** RSI 25.6 oversold on SHORT +- **Status:** Data collection in progress +- **Entry price:** Bug (0 value, needs investigation) + +## Data Collection Plan + +### Phase 1: Collect 20-30 Quality-Blocked Signals (2-4 weeks) + +**System Enhancement (Nov 22, 2025):** +- BlockedSignalTracker now tracks `QUALITY_SCORE_TOO_LOW` signals +- Price movement captured at 1min, 5min, 15min, 30min intervals +- TP1/TP2/SL hit detection using ATR-based targets +- Max favorable/adverse excursion tracked + +**SQL Query for Progress:** +```sql +-- Check data collection progress +SELECT + COUNT(*) as total_blocked, + SUM(CASE WHEN "analysisComplete" THEN 1 ELSE 0 END) as complete, + ROUND(AVG("signalQualityScore")::numeric, 1) as avg_quality, + MIN("signalQualityScore") as min_quality, + MAX("signalQualityScore") as max_quality +FROM "BlockedSignal" +WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW' + AND "createdAt" >= NOW() - INTERVAL '30 days'; +``` + +### Phase 2: Analysis (After 20+ Complete Signals) + +**Win Rate Comparison:** +```sql +-- Compare blocked signals vs executed trades +WITH blocked_stats AS ( + SELECT + COUNT(*) as total, + SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) as winners, + SUM(CASE WHEN "wouldHitSL" THEN 1 ELSE 0 END) as losers, + ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe, + ROUND(AVG("maxAdverseExcursion")::numeric, 2) as avg_mae + FROM "BlockedSignal" + WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW' + AND "analysisComplete" = true +), +executed_stats AS ( + SELECT + COUNT(*) as total, + SUM(CASE WHEN "realizedPnL" > 0 THEN 1 ELSE 0 END) as winners, + SUM(CASE WHEN "realizedPnL" < 0 THEN 1 ELSE 0 END) as losers, + ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe, + ROUND(AVG("maxAdverseExcursion")::numeric, 2) as avg_mae + FROM "Trade" + WHERE "indicatorVersion" = 'v8' + AND "exitReason" IS NOT NULL +) +SELECT + 'Blocked Signals' as category, + b.total, + b.winners, + b.losers, + ROUND(100.0 * b.winners / NULLIF(b.total, 0), 1) as win_rate, + b.avg_mfe, + b.avg_mae +FROM blocked_stats b +UNION ALL +SELECT + 'Executed Trades', + e.total, + e.winners, + e.losers, + ROUND(100.0 * e.winners / NULLIF(e.total, 0), 1) as win_rate, + e.avg_mfe, + e.avg_mae +FROM executed_stats e; +``` + +**Quality Score Distribution:** +```sql +-- Analyze blocked signal performance by quality tier +SELECT + CASE + WHEN "signalQualityScore" >= 85 THEN '85-90 (Near Threshold)' + WHEN "signalQualityScore" >= 80 THEN '80-84 (Marginal)' + WHEN "signalQualityScore" >= 75 THEN '75-79 (Weak)' + ELSE '70-74 (Very Weak)' + END as quality_tier, + COUNT(*) as count, + SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) as tp1_hits, + SUM(CASE WHEN "wouldHitSL" THEN 1 ELSE 0 END) as sl_hits, + ROUND(100.0 * SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) / COUNT(*), 1) as win_rate, + ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe +FROM "BlockedSignal" +WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW' + AND "analysisComplete" = true +GROUP BY quality_tier +ORDER BY MIN("signalQualityScore") DESC; +``` + +**ADX/RSI Patterns:** +```sql +-- Check if specific metrics predict blocked signal outcomes +SELECT + CASE + WHEN adx >= 20 THEN 'Strong Trend (ADX 20+)' + WHEN adx >= 18 THEN 'Moderate Trend (ADX 18-20)' + ELSE 'Weak Trend (ADX <18)' + END as trend_strength, + COUNT(*) as count, + SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) as winners, + ROUND(100.0 * SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) / COUNT(*), 1) as win_rate +FROM "BlockedSignal" +WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW' + AND "analysisComplete" = true + AND adx IS NOT NULL +GROUP BY trend_strength +ORDER BY MIN(adx) DESC; +``` + +## Decision Framework + +### Scenario 1: Blocked Signals Are Losers (<40% Win Rate) +**Action:** Keep threshold at 91 +**Reasoning:** System correctly filtering low-quality setups +**Conclusion:** Current false negative (quality 80 +0.52%) was statistical outlier + +### Scenario 2: Blocked Signals Are Winners (50%+ Win Rate) +**Action:** Lower threshold to 85 +**Reasoning:** Too restrictive, missing profitable opportunities +**Expected Impact:** +3-5 trades/week, potential +10-15% monthly P&L improvement + +### Scenario 3: Quality 80-84 Wins, 85-90 Loses +**Action:** Adjust threshold to 85, focus ADX/RSI weight optimization +**Reasoning:** Sharp cutoff exists, but threshold slightly misplaced +**Next Step:** Refine scoring weights to push marginal setups above 91 + +### Scenario 4: Pattern-Specific (e.g., Low ADX Wins in Strong Trends) +**Action:** Add context filters instead of lowering threshold +**Example:** Allow ADX 16-18 IF volume >1.2x AND price position mid-range +**Reasoning:** Preserve quality 91 standard but add intelligent exceptions + +## Timeline + +- **Week 1-2:** Collect 10-15 blocked signals +- **Week 3-4:** Collect 10-15 more blocked signals (total 20-30) +- **End of Week 4:** Run analysis queries, determine action +- **Week 5:** Implement changes if needed, document results + +## Success Metrics + +**Before Optimization:** +- Quality 91 threshold +- v8: 8 trades, 57.1% WR, +$262.70 total +- 1 confirmed false negative (quality 80, +0.52% missed) + +**After Optimization Target:** +- Maintain/improve win rate (≥55%) +- Increase trade frequency if lowering threshold (+3-5/week) +- Reduce false negatives (<5% of blocked signals profitable) +- Data-validated threshold (not guesswork) + +## Notes + +- **User requirement:** "data is king" - no changes without validation +- **Conservative approach:** Collect full dataset before any threshold adjustments +- **Financial impact:** Quality 80 signal = $43 missed profit on $8,300 position +- **Risk:** Lowering threshold too much could increase losses more than gains +- **Key insight:** Perfect quality separation at ≥95 still holds (5/5 trades won)