trading_bot_v4/docs/QUALITY_THRESHOLD_VALIDATION.md

Quality Threshold Validation Plan

Overview

System currently blocks signals with quality score <91. First validation shows potential false negative (quality 80 signal would have profited +0.52%). Need data to determine if threshold is optimal or too restrictive.

Discovery (Nov 22, 2025)

User Observation: "Green dots shot up" - TradingView Money Line signals blocked by system moved favorably

Signal 1 Analysis (Nov 21, 16:50):

• Blocked: Quality 80 (threshold 91)
• Primary reason: ADX 16.6 (weak trend, needs 18+)
• Entry: $126.20
• Peak: $126.86 within 1 minute
• Profit: +0.52% (+$43 on $8,300 position)
• TP1 Target: +1.51% (would NOT have hit)
• Result: FALSE NEGATIVE - blocked a profitable signal

Signal 2 (Nov 21, 23:00):

• Blocked: Quality 70 (threshold 91)
• Primary reason: RSI 25.6 oversold on SHORT
• Status: Data collection in progress
• Entry price: Bug (0 value, needs investigation)

Data Collection Plan

Phase 1: Collect 20-30 Quality-Blocked Signals (2-4 weeks)

System Enhancement (Nov 22, 2025):

• BlockedSignalTracker now tracks QUALITY_SCORE_TOO_LOW signals
• Price movement captured at 1min, 5min, 15min, 30min intervals
• TP1/TP2/SL hit detection using ATR-based targets
• Max favorable/adverse excursion tracked

SQL Query for Progress:

-- Check data collection progress
SELECT 
  COUNT(*) as total_blocked,
  SUM(CASE WHEN "analysisComplete" THEN 1 ELSE 0 END) as complete,
  ROUND(AVG("signalQualityScore")::numeric, 1) as avg_quality,
  MIN("signalQualityScore") as min_quality,
  MAX("signalQualityScore") as max_quality
FROM "BlockedSignal"
WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW'
  AND "createdAt" >= NOW() - INTERVAL '30 days';

Phase 2: Analysis (After 20+ Complete Signals)

Win Rate Comparison:

-- Compare blocked signals vs executed trades
WITH blocked_stats AS (
  SELECT 
    COUNT(*) as total,
    SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) as winners,
    SUM(CASE WHEN "wouldHitSL" THEN 1 ELSE 0 END) as losers,
    ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe,
    ROUND(AVG("maxAdverseExcursion")::numeric, 2) as avg_mae
  FROM "BlockedSignal"
  WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW'
    AND "analysisComplete" = true
),
executed_stats AS (
  SELECT 
    COUNT(*) as total,
    SUM(CASE WHEN "realizedPnL" > 0 THEN 1 ELSE 0 END) as winners,
    SUM(CASE WHEN "realizedPnL" < 0 THEN 1 ELSE 0 END) as losers,
    ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe,
    ROUND(AVG("maxAdverseExcursion")::numeric, 2) as avg_mae
  FROM "Trade"
  WHERE "indicatorVersion" = 'v8'
    AND "exitReason" IS NOT NULL
)
SELECT 
  'Blocked Signals' as category,
  b.total,
  b.winners,
  b.losers,
  ROUND(100.0 * b.winners / NULLIF(b.total, 0), 1) as win_rate,
  b.avg_mfe,
  b.avg_mae
FROM blocked_stats b
UNION ALL
SELECT 
  'Executed Trades',
  e.total,
  e.winners,
  e.losers,
  ROUND(100.0 * e.winners / NULLIF(e.total, 0), 1) as win_rate,
  e.avg_mfe,
  e.avg_mae
FROM executed_stats e;

Quality Score Distribution:

-- Analyze blocked signal performance by quality tier
SELECT 
  CASE 
    WHEN "signalQualityScore" >= 85 THEN '85-90 (Near Threshold)'
    WHEN "signalQualityScore" >= 80 THEN '80-84 (Marginal)'
    WHEN "signalQualityScore" >= 75 THEN '75-79 (Weak)'
    ELSE '70-74 (Very Weak)'
  END as quality_tier,
  COUNT(*) as count,
  SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) as tp1_hits,
  SUM(CASE WHEN "wouldHitSL" THEN 1 ELSE 0 END) as sl_hits,
  ROUND(100.0 * SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) / COUNT(*), 1) as win_rate,
  ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe
FROM "BlockedSignal"
WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW'
  AND "analysisComplete" = true
GROUP BY quality_tier
ORDER BY MIN("signalQualityScore") DESC;

ADX/RSI Patterns:

-- Check if specific metrics predict blocked signal outcomes
SELECT 
  CASE 
    WHEN adx >= 20 THEN 'Strong Trend (ADX 20+)'
    WHEN adx >= 18 THEN 'Moderate Trend (ADX 18-20)'
    ELSE 'Weak Trend (ADX <18)'
  END as trend_strength,
  COUNT(*) as count,
  SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) as winners,
  ROUND(100.0 * SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) / COUNT(*), 1) as win_rate
FROM "BlockedSignal"
WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW'
  AND "analysisComplete" = true
  AND adx IS NOT NULL
GROUP BY trend_strength
ORDER BY MIN(adx) DESC;

Decision Framework

Scenario 1: Blocked Signals Are Losers (<40% Win Rate)

Action: Keep threshold at 91 Reasoning: System correctly filtering low-quality setups Conclusion: Current false negative (quality 80 +0.52%) was statistical outlier

Scenario 2: Blocked Signals Are Winners (50%+ Win Rate)

Action: Lower threshold to 85 Reasoning: Too restrictive, missing profitable opportunities Expected Impact: +3-5 trades/week, potential +10-15% monthly P&L improvement

Scenario 3: Quality 80-84 Wins, 85-90 Loses

Action: Adjust threshold to 85, focus ADX/RSI weight optimization Reasoning: Sharp cutoff exists, but threshold slightly misplaced Next Step: Refine scoring weights to push marginal setups above 91

Scenario 4: Pattern-Specific (e.g., Low ADX Wins in Strong Trends)

Action: Add context filters instead of lowering threshold Example: Allow ADX 16-18 IF volume >1.2x AND price position mid-range Reasoning: Preserve quality 91 standard but add intelligent exceptions

Timeline

• Week 1-2: Collect 10-15 blocked signals
• Week 3-4: Collect 10-15 more blocked signals (total 20-30)
• End of Week 4: Run analysis queries, determine action
• Week 5: Implement changes if needed, document results

Success Metrics

Before Optimization:

• Quality 91 threshold
• v8: 8 trades, 57.1% WR, +$262.70 total
• 1 confirmed false negative (quality 80, +0.52% missed)

After Optimization Target:

• Maintain/improve win rate (≥55%)
• Increase trade frequency if lowering threshold (+3-5/week)
• Reduce false negatives (<5% of blocked signals profitable)
• Data-validated threshold (not guesswork)

Notes

• User requirement: "data is king" - no changes without validation
• Conservative approach: Collect full dataset before any threshold adjustments
• Financial impact: Quality 80 signal = $43 missed profit on $8,300 position
• Risk: Lowering threshold too much could increase losses more than gains
• Key insight: Perfect quality separation at ≥95 still holds (5/5 trades won)