docs: Add quality threshold validation plan
Created QUALITY_THRESHOLD_VALIDATION.md: - Discovery section: Documented Nov 22 false negative finding - Data collection plan: 20-30 blocked signals over 2-4 weeks - Analysis queries: Win rate comparison, quality tiers, ADX/RSI patterns - Decision framework: 4 scenarios with clear action items - Timeline: 4-week data collection + analysis - Success metrics: Maintain 55%+ WR, reduce false negatives Purpose: Complete reference for threshold optimization process. All decisions data-driven, no changes without validation.
This commit is contained in:
mindesbunister
187
docs/QUALITY_THRESHOLD_VALIDATION.md
Normal file
187
docs/QUALITY_THRESHOLD_VALIDATION.md
Normal file
@@ -0,0 +1,187 @@
|
||||
# Quality Threshold Validation Plan
|
||||
|
||||
## Overview
|
||||
System currently blocks signals with quality score <91. First validation shows potential false negative (quality 80 signal would have profited +0.52%). Need data to determine if threshold is optimal or too restrictive.
|
||||
|
||||
## Discovery (Nov 22, 2025)
|
||||
|
||||
**User Observation:** "Green dots shot up" - TradingView Money Line signals blocked by system moved favorably
|
||||
|
||||
**Signal 1 Analysis (Nov 21, 16:50):**
|
||||
- **Blocked:** Quality 80 (threshold 91)
|
||||
- **Primary reason:** ADX 16.6 (weak trend, needs 18+)
|
||||
- **Entry:** $126.20
|
||||
- **Peak:** $126.86 within 1 minute
|
||||
- **Profit:** +0.52% (+$43 on $8,300 position)
|
||||
- **TP1 Target:** +1.51% (would NOT have hit)
|
||||
- **Result:** FALSE NEGATIVE - blocked a profitable signal
|
||||
|
||||
**Signal 2 (Nov 21, 23:00):**
|
||||
- **Blocked:** Quality 70 (threshold 91)
|
||||
- **Primary reason:** RSI 25.6 oversold on SHORT
|
||||
- **Status:** Data collection in progress
|
||||
- **Entry price:** Bug (0 value, needs investigation)
|
||||
|
||||
## Data Collection Plan
|
||||
|
||||
### Phase 1: Collect 20-30 Quality-Blocked Signals (2-4 weeks)
|
||||
|
||||
**System Enhancement (Nov 22, 2025):**
|
||||
- BlockedSignalTracker now tracks `QUALITY_SCORE_TOO_LOW` signals
|
||||
- Price movement captured at 1min, 5min, 15min, 30min intervals
|
||||
- TP1/TP2/SL hit detection using ATR-based targets
|
||||
- Max favorable/adverse excursion tracked
|
||||
|
||||
**SQL Query for Progress:**
|
||||
```sql
|
||||
-- Check data collection progress
|
||||
SELECT
|
||||
COUNT(*) as total_blocked,
|
||||
SUM(CASE WHEN "analysisComplete" THEN 1 ELSE 0 END) as complete,
|
||||
ROUND(AVG("signalQualityScore")::numeric, 1) as avg_quality,
|
||||
MIN("signalQualityScore") as min_quality,
|
||||
MAX("signalQualityScore") as max_quality
|
||||
FROM "BlockedSignal"
|
||||
WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW'
|
||||
AND "createdAt" >= NOW() - INTERVAL '30 days';
|
||||
```
|
||||
|
||||
### Phase 2: Analysis (After 20+ Complete Signals)
|
||||
|
||||
**Win Rate Comparison:**
|
||||
```sql
|
||||
-- Compare blocked signals vs executed trades
|
||||
WITH blocked_stats AS (
|
||||
SELECT
|
||||
COUNT(*) as total,
|
||||
SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) as winners,
|
||||
SUM(CASE WHEN "wouldHitSL" THEN 1 ELSE 0 END) as losers,
|
||||
ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe,
|
||||
ROUND(AVG("maxAdverseExcursion")::numeric, 2) as avg_mae
|
||||
FROM "BlockedSignal"
|
||||
WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW'
|
||||
AND "analysisComplete" = true
|
||||
),
|
||||
executed_stats AS (
|
||||
SELECT
|
||||
COUNT(*) as total,
|
||||
SUM(CASE WHEN "realizedPnL" > 0 THEN 1 ELSE 0 END) as winners,
|
||||
SUM(CASE WHEN "realizedPnL" < 0 THEN 1 ELSE 0 END) as losers,
|
||||
ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe,
|
||||
ROUND(AVG("maxAdverseExcursion")::numeric, 2) as avg_mae
|
||||
FROM "Trade"
|
||||
WHERE "indicatorVersion" = 'v8'
|
||||
AND "exitReason" IS NOT NULL
|
||||
)
|
||||
SELECT
|
||||
'Blocked Signals' as category,
|
||||
b.total,
|
||||
b.winners,
|
||||
b.losers,
|
||||
ROUND(100.0 * b.winners / NULLIF(b.total, 0), 1) as win_rate,
|
||||
b.avg_mfe,
|
||||
b.avg_mae
|
||||
FROM blocked_stats b
|
||||
UNION ALL
|
||||
SELECT
|
||||
'Executed Trades',
|
||||
e.total,
|
||||
e.winners,
|
||||
e.losers,
|
||||
ROUND(100.0 * e.winners / NULLIF(e.total, 0), 1) as win_rate,
|
||||
e.avg_mfe,
|
||||
e.avg_mae
|
||||
FROM executed_stats e;
|
||||
```
|
||||
|
||||
**Quality Score Distribution:**
|
||||
```sql
|
||||
-- Analyze blocked signal performance by quality tier
|
||||
SELECT
|
||||
CASE
|
||||
WHEN "signalQualityScore" >= 85 THEN '85-90 (Near Threshold)'
|
||||
WHEN "signalQualityScore" >= 80 THEN '80-84 (Marginal)'
|
||||
WHEN "signalQualityScore" >= 75 THEN '75-79 (Weak)'
|
||||
ELSE '70-74 (Very Weak)'
|
||||
END as quality_tier,
|
||||
COUNT(*) as count,
|
||||
SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) as tp1_hits,
|
||||
SUM(CASE WHEN "wouldHitSL" THEN 1 ELSE 0 END) as sl_hits,
|
||||
ROUND(100.0 * SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) / COUNT(*), 1) as win_rate,
|
||||
ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe
|
||||
FROM "BlockedSignal"
|
||||
WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW'
|
||||
AND "analysisComplete" = true
|
||||
GROUP BY quality_tier
|
||||
ORDER BY MIN("signalQualityScore") DESC;
|
||||
```
|
||||
|
||||
**ADX/RSI Patterns:**
|
||||
```sql
|
||||
-- Check if specific metrics predict blocked signal outcomes
|
||||
SELECT
|
||||
CASE
|
||||
WHEN adx >= 20 THEN 'Strong Trend (ADX 20+)'
|
||||
WHEN adx >= 18 THEN 'Moderate Trend (ADX 18-20)'
|
||||
ELSE 'Weak Trend (ADX <18)'
|
||||
END as trend_strength,
|
||||
COUNT(*) as count,
|
||||
SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) as winners,
|
||||
ROUND(100.0 * SUM(CASE WHEN "wouldHitTP1" THEN 1 ELSE 0 END) / COUNT(*), 1) as win_rate
|
||||
FROM "BlockedSignal"
|
||||
WHERE "blockReason" = 'QUALITY_SCORE_TOO_LOW'
|
||||
AND "analysisComplete" = true
|
||||
AND adx IS NOT NULL
|
||||
GROUP BY trend_strength
|
||||
ORDER BY MIN(adx) DESC;
|
||||
```
|
||||
|
||||
## Decision Framework
|
||||
|
||||
### Scenario 1: Blocked Signals Are Losers (<40% Win Rate)
|
||||
**Action:** Keep threshold at 91
|
||||
**Reasoning:** System correctly filtering low-quality setups
|
||||
**Conclusion:** Current false negative (quality 80 +0.52%) was statistical outlier
|
||||
|
||||
### Scenario 2: Blocked Signals Are Winners (50%+ Win Rate)
|
||||
**Action:** Lower threshold to 85
|
||||
**Reasoning:** Too restrictive, missing profitable opportunities
|
||||
**Expected Impact:** +3-5 trades/week, potential +10-15% monthly P&L improvement
|
||||
|
||||
### Scenario 3: Quality 80-84 Wins, 85-90 Loses
|
||||
**Action:** Adjust threshold to 85, focus ADX/RSI weight optimization
|
||||
**Reasoning:** Sharp cutoff exists, but threshold slightly misplaced
|
||||
**Next Step:** Refine scoring weights to push marginal setups above 91
|
||||
|
||||
### Scenario 4: Pattern-Specific (e.g., Low ADX Wins in Strong Trends)
|
||||
**Action:** Add context filters instead of lowering threshold
|
||||
**Example:** Allow ADX 16-18 IF volume >1.2x AND price position mid-range
|
||||
**Reasoning:** Preserve quality 91 standard but add intelligent exceptions
|
||||
|
||||
## Timeline
|
||||
|
||||
- **Week 1-2:** Collect 10-15 blocked signals
|
||||
- **Week 3-4:** Collect 10-15 more blocked signals (total 20-30)
|
||||
- **End of Week 4:** Run analysis queries, determine action
|
||||
- **Week 5:** Implement changes if needed, document results
|
||||
|
||||
## Success Metrics
|
||||
|
||||
**Before Optimization:**
|
||||
- Quality 91 threshold
|
||||
- v8: 8 trades, 57.1% WR, +$262.70 total
|
||||
- 1 confirmed false negative (quality 80, +0.52% missed)
|
||||
|
||||
**After Optimization Target:**
|
||||
- Maintain/improve win rate (≥55%)
|
||||
- Increase trade frequency if lowering threshold (+3-5/week)
|
||||
- Reduce false negatives (<5% of blocked signals profitable)
|
||||
- Data-validated threshold (not guesswork)
|
||||
|
||||
## Notes
|
||||
|
||||
- **User requirement:** "data is king" - no changes without validation
|
||||
- **Conservative approach:** Collect full dataset before any threshold adjustments
|
||||
- **Financial impact:** Quality 80 signal = $43 missed profit on $8,300 position
|
||||
- **Risk:** Lowering threshold too much could increase losses more than gains
|
||||
- **Key insight:** Perfect quality separation at ≥95 still holds (5/5 trades won)
|
||||
Reference in New Issue
Block a user