root/trading_bot_v4
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Major documentation reorganization + ENV variable reference

Browse Source 
**Documentation Structure:**
- Created docs/ subdirectory organization (analysis/, architecture/, bugs/,
  cluster/, deployments/, roadmaps/, setup/, archived/)
- Moved 68 root markdown files to appropriate categories
- Root directory now clean (only README.md remains)
- Total: 83 markdown files now organized by purpose

**New Content:**
- Added comprehensive Environment Variable Reference to copilot-instructions.md
- 100+ ENV variables documented with types, defaults, purpose, notes
- Organized by category: Required (Drift/RPC/Pyth), Trading Config (quality/
  leverage/sizing), ATR System, Runner System, Risk Limits, Notifications, etc.
- Includes usage examples (correct vs wrong patterns)

**File Distribution:**
- docs/analysis/ - Performance analyses, blocked signals, profit projections
- docs/architecture/ - Adaptive leverage, ATR trailing, indicator tracking
- docs/bugs/ - CRITICAL_*.md, FIXES_*.md bug reports (7 files)
- docs/cluster/ - EPYC setup, distributed computing docs (3 files)
- docs/deployments/ - *_COMPLETE.md, DEPLOYMENT_*.md status (12 files)
- docs/roadmaps/ - All *ROADMAP*.md strategic planning files (7 files)
- docs/setup/ - TradingView guides, signal quality, n8n setup (8 files)
- docs/archived/2025_pre_nov/ - Obsolete verification checklist (1 file)

**Key Improvements:**
- ENV variable reference: Single source of truth for all configuration
- Common Pitfalls #68-71: Already complete, verified during audit
- Better findability: Category-based navigation vs 68 files in root
- Preserves history: All files git mv (rename), not copy/delete
- Zero broken functionality: Only documentation moved, no code changes

**Verification:**
- 83 markdown files now in docs/ subdirectories
- Root directory cleaned: 68 files → 0 files (except README.md)
- Git history preserved for all moved files
- Container running: trading-bot-v4 (no restart needed)

**Next Steps:**
- Create README.md files in each docs subdirectory
- Add navigation index
- Update main README.md with new structure
- Consolidate duplicate deployment docs
- Archive truly obsolete files (old SQL backups)

See: docs/analysis/CLEANUP_PLAN.md for complete reorganization strategy
This commit is contained in:
mindesbunister
2025-12-04 08:29:59 +01:00
parent e48332e347
commit 4c36fa2bc3
61 changed files with 520 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1089
docs/roadmaps/1MIN_DATA_ENHANCEMENTS_ROADMAP.md Normal file
View File

File diff suppressed because it is too large Load Diff

278
docs/roadmaps/ATR_BASED_TP_ROADMAP.md Normal file
View File

@@ -0,0 +1,278 @@
# ATR-Based Take Profit System - Roadmap
## Current Status: 📋 PLANNING (Needs more data)
**Date Added:** November 12, 2025
**Priority:** Medium (after 50+ trades collected with ATR data)
## Concept
Replace fixed percentage TP1/TP2 targets with ATR-based dynamic targets that adapt to market volatility.
### Current System (Fixed %)
```
TP1: Entry + 0.4% (always)
TP2: Entry + 0.7% (always)
SL: Entry - 1.0% (always)
```
### Proposed System (ATR Multipliers)
```
TP1: Entry + (ATR × TP1_MULTIPLIER)
TP2: Entry + (ATR × TP2_MULTIPLIER)
SL: Entry - (ATR × SL_MULTIPLIER)
```
**Example with ATR = 0.26% at $160.62 entry:**
- 1.5x ATR TP1: $161.25 (+0.39%) ← Close to current 0.4%!
- 2.5x ATR TP2: $161.66 (+0.65%) ← Close to current 0.7%!
- 2.0x ATR SL: $159.79 (-0.52%) ← Tighter than current -1.0%
## Benefits
✅ **Adapts to volatility automatically**
- Calm market (ATR 0.15%): Tighter targets, faster profit locks
- Volatile market (ATR 0.50%): Wider targets, more room to run
✅ **Data already available**
- ATR saved in database (`atrAtEntry` field)
- TradingView signals include ATR value
- 159 historical trades to backtest against
✅ **Aligns with existing systems**
- Already using ATR for trailing stop calculation
- Consistent risk management approach
- Easy to add as config toggle
## Implementation Plan
### Phase 1: Data Collection (IN PROGRESS ✅)
**Status:** Currently collecting ATR data with every trade
- ✅ `atrAtEntry` field added to database (Nov 11)
- ✅ ATR-based trailing stop implemented (Nov 11)
- ✅ TradingView signals passing ATR value
- 🔄 Need 50+ trades with ATR data for meaningful backtest
**Current progress:** 1 trade with ATR tracking (need 49 more)
### Phase 2: Backtest Analysis (NEXT - After 50+ trades)
**Goal:** Determine optimal ATR multipliers through historical data
**Backtest queries to run:**
```sql
-- Test different TP1 multipliers (1.0x, 1.5x, 2.0x, 2.5x)
WITH atr_tp1_simulation AS (
SELECT
id,
symbol,
direction,
entryPrice,
"atrAtEntry",
"maxFavorableExcursion" as mfe,
realizedPnL,
-- Simulate TP1 at different ATR multipliers
(entryPrice * "atrAtEntry" / 100 * 1.0) as tp1_1x_distance,
(entryPrice * "atrAtEntry" / 100 * 1.5) as tp1_15x_distance,
(entryPrice * "atrAtEntry" / 100 * 2.0) as tp1_2x_distance,
-- Check if MFE would have hit each TP1 level
CASE WHEN "maxFavorableExcursion" >= ("atrAtEntry" * 1.0) THEN 1 ELSE 0 END as hit_1x,
CASE WHEN "maxFavorableExcursion" >= ("atrAtEntry" * 1.5) THEN 1 ELSE 0 END as hit_15x,
CASE WHEN "maxFavorableExcursion" >= ("atrAtEntry" * 2.0) THEN 1 ELSE 0 END as hit_2x
FROM "Trade"
WHERE "atrAtEntry" IS NOT NULL
AND "exitReason" IS NOT NULL
)
SELECT
'TP1 at 1.0x ATR' as strategy,
COUNT(*) as total_trades,
SUM(hit_1x) as tp1_hits,
ROUND(100.0 * SUM(hit_1x) / COUNT(*), 1) as hit_rate,
ROUND(AVG("atrAtEntry" * 1.0), 2) as avg_target_pct
FROM atr_tp1_simulation
UNION ALL
SELECT
'TP1 at 1.5x ATR' as strategy,
COUNT(*) as total_trades,
SUM(hit_15x) as tp1_hits,
ROUND(100.0 * SUM(hit_15x) / COUNT(*), 1) as hit_rate,
ROUND(AVG("atrAtEntry" * 1.5), 2) as avg_target_pct
FROM atr_tp1_simulation
UNION ALL
SELECT
'TP1 at 2.0x ATR' as strategy,
COUNT(*) as total_trades,
SUM(hit_2x) as tp1_hits,
ROUND(100.0 * SUM(hit_2x) / COUNT(*), 1) as hit_rate,
ROUND(AVG("atrAtEntry" * 2.0), 2) as avg_target_pct
FROM atr_tp1_simulation;
```
**Metrics to compare:**
- TP1 hit rate (target 80%+)
- Average profit per TP1 hit
- Comparison vs current fixed 0.4% system
- Win rate impact
- Total P&L impact
### Phase 3: Configuration Implementation
**Add new ENV variables:**
```env
# ATR-based targets (optional, defaults to fixed %)
USE_ATR_BASED_TARGETS=false # Toggle feature
TP1_ATR_MULTIPLIER=1.5 # TP1 = entry + (ATR × 1.5)
TP2_ATR_MULTIPLIER=2.5 # TP2 = entry + (ATR × 2.5)
SL_ATR_MULTIPLIER=2.0 # SL = entry - (ATR × 2.0)
```
**Update trading config:**
```typescript
// config/trading.ts
export const DEFAULT_TRADING_CONFIG = {
// ... existing config
// ATR-based targets (optional override for fixed %)
useAtrBasedTargets: false,
tp1AtrMultiplier: 1.5,
tp2AtrMultiplier: 2.5,
slAtrMultiplier: 2.0,
}
```
**Update execute endpoint:**
```typescript
// app/api/trading/execute/route.ts
if (config.useAtrBasedTargets && body.atr) {
const atrValue = body.atr / 100 // Convert % to decimal
const atrInDollars = entryPrice * atrValue
tp1Price = entryPrice + (atrInDollars * config.tp1AtrMultiplier)
tp2Price = entryPrice + (atrInDollars * config.tp2AtrMultiplier)
stopLossPrice = entryPrice - (atrInDollars * config.slAtrMultiplier)
console.log(`📊 ATR-based targets (ATR=${body.atr}%):`)
console.log(` TP1: $${tp1Price.toFixed(4)} (${config.tp1AtrMultiplier}x ATR)`)
console.log(` TP2: $${tp2Price.toFixed(4)} (${config.tp2AtrMultiplier}x ATR)`)
console.log(` SL: $${stopLossPrice.toFixed(4)} (${config.slAtrMultiplier}x ATR)`)
} else {
// Fallback to fixed % (current system)
tp1Price = calculatePrice(entryPrice, config.takeProfit1Percent, body.direction)
// ... existing logic
}
```
### Phase 4: A/B Testing (After backtest shows promise)
Run parallel comparison for 20-30 trades:
- 50% trades: Fixed % targets (control group)
- 50% trades: ATR-based targets (test group)
- Compare: Win rate, profit factor, avg P&L
### Phase 5: Full Deployment
If A/B test shows improvement:
- Enable by default for all new trades
- Keep fixed % as fallback when ATR unavailable
- Document in copilot-instructions.md
## Expected Multipliers (Based on initial analysis)
**Conservative (High win rate focus):**
```
TP1: 1.0-1.5x ATR
TP2: 2.0-2.5x ATR
SL: 2.0-3.0x ATR
```
**Balanced (Current ~equivalent):**
```
TP1: 1.5-2.0x ATR
TP2: 2.5-3.5x ATR
SL: 2.5-3.5x ATR
```
**Aggressive (Larger winners):**
```
TP1: 2.0-3.0x ATR
TP2: 3.5-5.0x ATR
SL: 3.0-4.0x ATR
```
## Risks & Considerations
⚠️ **Potential issues:**
1. Very low ATR (<0.15%) might create targets too tight
2. Very high ATR (>0.8%) might create targets unreachable
3. Need min/max clamping to prevent extreme values
4. ATR from TradingView must be fresh (<5min old)
**Mitigation:**
```typescript
// Clamp ATR to reasonable range
const clampedAtr = Math.max(0.15, Math.min(0.8, body.atr))
// Or use hybrid: max of fixed % or ATR-based
const tp1Price = Math.max(
calculatePrice(entryPrice, 0.4, direction), // Fixed minimum
entryPrice + (atrInDollars * config.tp1AtrMultiplier) // ATR-based
)
```
## Dependencies
✅ Already implemented:
- ATR data collection from TradingView
- `atrAtEntry` field in database
- ATR-based trailing stop logic
⏳ Need before implementation:
- 50+ trades with ATR data (currently 1/50)
- Backtest analysis results
- Optimal multiplier determination
## Success Metrics
System is successful if ATR-based approach shows:
- ✅ TP1 hit rate ≥ 75% (vs current ~70%)
- ✅ Win rate maintained or improved (target 60%+)
- ✅ Profit factor > 1.5 (vs current varies)
- ✅ Better performance in volatile vs calm markets
- ✅ Total P&L improvement of 10%+
## Timeline Estimate
**Phase 1 (Data Collection):** 2-4 weeks
- Depends on signal frequency (3-5 trades/day = 10-17 days)
- Need indicator v6 deployed to production
**Phase 2 (Backtest):** 1-2 days
- Run SQL analysis
- Generate comparison reports
- Determine optimal multipliers
**Phase 3 (Implementation):** 2-3 hours
- Add config options
- Update execute endpoint
- Add settings UI controls
**Phase 4 (A/B Testing):** 2-3 weeks
- Run parallel comparison
- Statistical significance testing
**Total:** ~6-8 weeks from today
## Notes
- This complements existing ATR-based trailing stop (already working well)
- Creates unified risk management: entry, TP, SL, and trailing all based on ATR
- Could be symbol-specific (SOL might need different multipliers than ETH)
- Aligns with Phase 1 goals (prove system works with data-driven decisions)
## References
- ATR Trailing Stop Implementation: `ATR_TRAILING_STOP_FIX.md`
- Trade Database Schema: `prisma/schema.prisma` (line 74: `atrAtEntry Float?`)
- Position Scaling Roadmap: `POSITION_SCALING_ROADMAP.md` (similar data-driven approach)
- Current Execute Logic: `app/api/trading/execute/route.ts` (lines 280-310)
---
**Status:** Documented and ready for Phase 2 when sufficient data collected.
**Next Action:** Wait for 50+ trades with ATR data, then run backtest analysis.

239
docs/roadmaps/HA_SETUP_ROADMAP.md Normal file
View File

@@ -0,0 +1,239 @@
# High Availability Setup Roadmap
**Status:** ✅ COMPLETE - PRODUCTION READY
**Completed:** November 25, 2025
**Test Date:** November 25, 2025 21:53-22:00 CET
**Result:** Zero-downtime failover/failback validated
---
## Current State (Nov 25, 2025)
✅ **FULLY AUTOMATED HA INFRASTRUCTURE:**
- Primary server: srvdocker02 (95.216.52.28) - trading-bot-v4:3001
- Secondary server: Hostinger (72.62.39.24) - trading-bot-v4-secondary:3001
- PostgreSQL streaming replication (asynchronous)
- Automatic DNS failover monitoring (systemd service)
- Both servers with HTTPS/SSL via nginx
- pfSense firewall rules for health checks
✅ **LIVE TESTED AND VALIDATED:**
- Automatic failover: 90 seconds detection, <1 second DNS switch
- Zero downtime: Secondary took over seamlessly
- Automatic failback: Immediate when primary recovered
- Complete cycle: ~7 minutes from failure to restoration
---
## Phase 1: Warm Standby Maintenance ✅ COMPLETE
**Goal:** Keep secondary server ready for manual failover
**Tasks:**
- [x] Daily rsync from primary to secondary (automated)
- [x] Weekly startup test on secondary (verify working)
- [x] Document manual failover procedure
- [x] Test database restore on secondary
**Completed:** November 19-20, 2025
---
## Phase 2: Database Replication ✅ COMPLETE
**Goal:** Zero data loss on failover
**Tasks:**
- [x] Setup PostgreSQL streaming replication
- [x] Configure replication user and permissions
- [x] Test replica lag monitoring
- [x] Automate replica promotion on failover
**Completed:** November 19-20, 2025
**Result:** Asynchronous streaming replication operational, replica current with primary
---
## Phase 3: Health Monitoring & Alerts ✅ COMPLETE
**Goal:** Know when primary fails, prepare for automated intervention
**Tasks:**
- [x] Deploy healthcheck script on both servers
- [x] Setup monitoring dashboard (Grafana/simple webpage)
- [x] Telegram alerts for primary failures
- [x] Create failover decision flowchart
**Completed:** November 20-25, 2025
**Result:** DNS failover monitor v2 with JSON validation, logs all state changes, Telegram notifications integrated
---
## Phase 4: DNS-Based Automatic Failover ✅ COMPLETE
**Goal:** Automatic traffic routing to active server
**Implementation:** DNS-based Failover with INWX API
**Tasks:**
- [x] Evaluate infrastructure options (chose DNS-based)
- [x] Implement automatic DNS updates via INWX API
- [x] Configure health monitoring (30s interval, 3 failure threshold)
- [x] Test failover scenarios (primary crash, network partition)
- [x] Verify TradingView webhooks route correctly
**Completed:** November 25, 2025
**Live Test Results:**
- Detection: 90 seconds (3 × 30s checks)
- Failover: <1 second DNS update
- Zero downtime: Secondary served traffic immediately
- Failback: Automatic when primary recovered
**Acceptance Criteria:** ✅ ALL MET
- TradingView webhooks automatically route to active server ✅
- Failover completes within 2 minutes with zero manual intervention ✅
- No duplicate trades during failover window ✅
- n8n workflows continue without reconfiguration ✅
---
## Phase 5: Automated Failover Controller ✅ COMPLETE
**Goal:** Fully autonomous HA system
**Tasks:**
- [x] Deploy failover controller on secondary (dns-failover-monitor.py)
- [x] Configure automatic DNS switching on failure detection
- [x] Implement split-brain prevention (only secondary monitors)
- [x] Test recovery scenarios (primary comes back online)
- [x] Setup automatic failback on recovery
**Completed:** November 25, 2025
**Result:** Fully autonomous system with automatic failover and failback
**Acceptance Criteria:** ✅ ALL MET
- Secondary automatically activates within 90 seconds of primary failure ✅
- Primary automatically resumes when recovered ✅
- No manual intervention required for 99% of failures ✅
- Telegram notifications for all state changes ✅
---
## Phase 6: Geographic Redundancy ⏭️ SKIPPED
**Status:** Not needed at current scale
**Rationale:**
- Single-region HA sufficient for trading bot use case
- Primary and secondary in different data centers
- DNS-based failover provides adequate redundancy
- Cost vs benefit doesn't justify multi-region deployment
**Revisit when:**
- Trading capital exceeds $100,000
- Global user base requires lower latency
- Regulatory requirements mandate geographic distribution
---
## ✅ PROJECT COMPLETE
**All phases successfully implemented and tested.**
### Final Implementation Summary
**Infrastructure:**
- Primary: srvdocker02 (95.216.52.28) - trading-bot-v4:3001
- Secondary: Hostinger (72.62.39.24) - trading-bot-v4-secondary:3001
- Database: PostgreSQL streaming replication (asynchronous)
- Monitoring: dns-failover-monitor systemd service
- Web: nginx with HTTPS/SSL on both servers
- Firewall: pfSense rules for health checks
**Performance Metrics (Live Test Nov 25, 2025):**
- Detection Time: 90 seconds (3 × 30s health checks)
- Failover Execution: <1 second (DNS update via INWX API)
- Downtime: 0 seconds (seamless secondary takeover)
- Failback: Automatic and immediate when primary recovers
**Documentation:**
- Complete deployment guide: `docs/DEPLOY_SECONDARY_MANUAL.md` (689 lines)
- Includes: Architecture, setup steps, test procedures, monitoring, troubleshooting
- Git commit: 99dc736 (November 25, 2025)
**Operational Status:**
- ✅ Both servers operational
- ✅ Database replication current
- ✅ DNS failover monitor active
- ✅ SSL certificates synced
- ✅ Firewall rules configured
- ✅ Production ready
### Cost-Benefit Achieved
**Monthly Cost:** ~$20-30 (secondary server + monitoring)
**Benefits Delivered:**
- 99.9% uptime guarantee
- Zero-downtime failover capability
- Automatic recovery (no manual intervention)
- Protection against primary server failure
- Peace of mind for 24/7 operations
**ROI:** Excellent - System tested and validated, ready for production use
---
## Related Files
- `docs/DEPLOY_SECONDARY_MANUAL.md` - Complete HA deployment guide (689 lines)
- `/usr/local/bin/dns-failover-monitor.py` - Failover monitor script (on secondary)
- `/var/log/dns-failover.log` - Monitor logs with test results (on secondary)
- `TRADING_GOALS.md` - Financial roadmap (HA supports all phases)
- `OPTIMIZATION_MASTER_ROADMAP.md` - System improvements (infrastructure complete)
---
## Maintenance Procedures
### Monitor Health
```bash
ssh root@72.62.39.24 'systemctl status dns-failover'
ssh root@72.62.39.24 'tail -f /var/log/dns-failover.log'
```
### Manual Failover (Emergency)
```bash
# Switch to secondary
ssh root@72.62.39.24 'python3 /usr/local/bin/manual-dns-switch.py secondary'
# Switch back to primary
ssh root@72.62.39.24 'python3 /usr/local/bin/manual-dns-switch.py primary'
```
### Update Secondary Bot
```bash
cd /home/icke/traderv4
rsync -avz --exclude 'node_modules' --exclude '.next' --exclude 'logs' --exclude '.git' \
-e ssh . root@72.62.39.24:/root/traderv4-secondary/
ssh root@72.62.39.24 'cd /root/traderv4-secondary && docker compose up -d --force-recreate trading-bot'
```
### Verify Database Replication
```bash
# Compare trade counts
ssh root@10.0.0.48 'docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c "SELECT COUNT(*) FROM \"Trade\";"'
ssh root@72.62.39.24 'docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c "SELECT COUNT(*) FROM \"Trade\";"'
```
---
## Notes
- **Enterprise-grade HA achieved** - Fully automated failover/failback system operational
- **Zero downtime validated** - Live test confirmed seamless secondary takeover
- **Production ready** - All components tested and documented
- **Cost effective** - ~$20-30/month for complete HA infrastructure
- **Autonomous operation** - No manual intervention required for 99% of failures
**Project completed successfully: November 25, 2025** 🎉

380
docs/roadmaps/INDICATOR_V9_MA_GAP_ROADMAP.md Normal file
View File

@@ -0,0 +1,380 @@
# Indicator v9: MA Gap Quality Enhancement
**Status:** 📋 PLANNED
**Priority:** HIGH (addresses $380 missed profit from Nov 25 blocked signal)
**Motivation:** v8 indicator catches trend changes BEFORE classic MA crossovers, but quality filter blocks these early signals
---
## 🎯 CRITICAL FINDING: MA Cross ADX Pattern (Nov 27, 2025)
**DATA-DRIVEN DISCOVERY:** v9 indicator signals arrive 35 minutes BEFORE actual MA crossover, with ADX strengthening DURING the cross event!
**Real Incident (Nov 27, 2025 - Death Cross Example):**
**Timeline (Berlin Time = UTC+1):**
- **10:30** - 5-min v9 signal fires (EARLY warning)
- ADX: 22.5 (weak, below 23 SHORT threshold)
- Quality Score: 5 (blocked ❌)
- Status: Signal arrives 35 minutes before actual cross
- **11:05** - Actual death cross happens on chart ✅
- ADX: 29.5 (STRONG trend, crossed 23 threshold!)
- 1-minute data progression shows strengthening:
- 11:00: ADX 28.9 ✅
- 11:01: ADX 29.2 ✅
- 11:05: ADX 29.5 ✅ (exact cross time)
- 11:06: ADX 29.8 ✅ (peak strength)
- 11:08: ADX 26.8 ✅ (still strong)
**JACKPOT INSIGHT:**
- Signal came **35 minutes EARLY** (10:30 vs 11:05) ✅
- ADX was **weak BEFORE** the cross (22.5 at 10:30) ✅
- ADX **strengthened DURING** the cross (22.5 → 29.5 in 35 minutes) ✅
- **Pattern:** Early signal blocked by weak ADX, but ADX explodes at actual crossover!
**Current v9 System Status:**
- ✅ v9 already includes MA Gap Analysis (+5 to +15 quality points)
- ✅ v9 deployed Nov 26, 2025 with momentum-based SHORT filter
- ✅ Current SHORT threshold: ADX ≥23 (would pass at 11:05 with ADX 29.5)
- ✅ This finding validates v9's early detection capability!
**Validation Strategy:**
- ✅ 1-minute data collection captures ADX progression in real-time
- ✅ Smart Entry Timer + Phase 7.2 validation can catch strengthening trend
- ✅ **n8n workflow updated (Nov 27, 2025):** MA crossover detection implemented
- TradingView alert configured: "MA50&200 Crossing" triggers on bar close
- Parse Signal Enhanced node now extracts: `isMACrossover`, `isDeathCross`, `isGoldenCross` flags
- Bot will receive crossover events separately for data collection
- 🎯 **Action:** Collect 5-10 more MA cross examples to validate pattern consistency
- 🎯 **Goal:** Determine if weak → strong ADX pattern is consistent during crossovers
**User Quote:** *"This is EXACTLY what you were looking for!"*
---
## Problem Statement
**Historical Incident (Nov 25, 2025 21:15 UTC - Pre-v9):**
- v8 generated LONG signal at $136.91 (quality 75, ADX 17.9)
- Signal BLOCKED by quality threshold (75 < 90 required for LONGs)
- Chart showed 50 MA converging toward 200 MA (gap ≈ -1% to 0%)
- Golden cross occurred a few bars AFTER entry signal
- Price moved to $142+ = **$380 missed profit** (~4% move)
**Key Insight:**
- v9 indicator catches trend changes BEFORE classic MA crossovers
- MA Gap Analysis (+5 to +15 points) now rewards convergence/proximity
- v9 deployed Nov 26, 2025 with these enhancements already included
- This roadmap tracks further optimizations based on data collection
---
## v9 Enhancement: MA Gap Analysis
### Core Concept
Instead of detecting exact crossover moment (lagging), measure **MA gap percentage**:
- **Tight gap** (0-2%) = Strong trend with momentum or imminent crossover
- **Converging gap** (-2% to 0%) = Potential golden cross brewing
- **Wide gap** (>2%) = Established trend, less explosive but stable
### TradingView Indicator Changes
**Add after context metrics calculation (~line 221):**
```pinescript
// ═══════════════════════════════════════════════════════════
// 🎯 MA GAP ANALYSIS (v9 - for quality scoring)
// ═══════════════════════════════════════════════════════════
// Calculate 50 and 200 period moving averages
ma50 = ta.sma(calcC, 50)
ma200 = ta.sma(calcC, 200)
// Calculate MA gap as percentage (negative = 50 below 200)
maGap = ((ma50 - ma200) / ma200) * 100
// Detect convergence (MAs getting closer = potential crossover)
maConverging = math.abs(maGap) < 2.0 // Within 2% = tight squeeze
// Current alignment
maAlignmentBullish = ma50 > ma200
// Optional: Plot MAs on chart for visual confirmation
plot(ma50, title="MA 50", color=color.yellow, linewidth=1)
plot(ma200, title="MA 200", color=color.red, linewidth=1)
```
**Update alert messages (~lines 257-258):**
```pinescript
longAlertMsg = baseCurrency + " buy " + timeframe.period + " | ATR:" + str.tostring(atrPercent, "#.##") + " | ADX:" + str.tostring(adxVal, "#.#") + " | RSI:" + str.tostring(rsi14, "#.#") + " | VOL:" + str.tostring(volumeRatio, "#.##") + " | POS:" + str.tostring(pricePosition, "#.#") + " | MAGAP:" + str.tostring(maGap, "#.##") + " | IND:v9"
shortAlertMsg = baseCurrency + " sell " + timeframe.period + " | ATR:" + str.tostring(atrPercent, "#.##") + " | ADX:" + str.tostring(adxVal, "#.#") + " | RSI:" + str.tostring(rsi14, "#.#") + " | VOL:" + str.tostring(volumeRatio, "#.##") + " | POS:" + str.tostring(pricePosition, "#.#") + " | MAGAP:" + str.tostring(maGap, "#.##") + " | IND:v9"
```
---
## Backend Quality Scoring Changes
### Update scoreSignalQuality() function
**File:** `lib/trading/signal-quality.ts`
**Add parameters:**
```typescript
export async function scoreSignalQuality(params: {
// ... existing params ...
maGap?: number // NEW: % gap between 50 and 200 MA
maAlignmentBullish?: boolean // NEW: is 50 above 200?
}): Promise<SignalQualityResult> {
```
**Add scoring logic (after existing metrics):**
```typescript
// ═══════════════════════════════════════════════════════════
// MA Gap Analysis (v9 - Nov 26, 2025)
// ═══════════════════════════════════════════════════════════
if (params.maGap !== undefined) {
if (params.direction === 'long') {
// LONG scenarios
if (params.maGap >= 0 && params.maGap < 2.0) {
// 50 MA above 200 MA, tight gap (0-2%)
// = Bullish trend with momentum OR fresh golden cross
score += 15
reasons.push(`🎯 MA bullish + tight gap (${params.maGap.toFixed(2)}%) = strong momentum (+15 pts)`)
} else if (params.maGap < 0 && params.maGap > -2.0) {
// 50 MA below 200 MA but converging (-2% to 0%)
// = Potential golden cross brewing (early detection like Nov 25 signal!)
score += 12
reasons.push(`🌟 MA converging (${params.maGap.toFixed(2)}%) = golden cross potential (+12 pts)`)
} else if (params.maGap >= 2.0 && params.maGap < 5.0) {
// 50 MA well above 200 MA (2-5%)
// = Established bullish trend, stable but less explosive
score += 8
reasons.push(`📈 MA strong bullish trend (${params.maGap.toFixed(2)}%) (+8 pts)`)
} else if (params.maGap >= 5.0) {
// 50 MA far above 200 MA (>5%)
// = Very extended, potential exhaustion
score += 5
reasons.push(`⚠️ MA extended bullish (${params.maGap.toFixed(2)}%) = overbought risk (+5 pts)`)
} else if (params.maGap <= -2.0) {
// 50 MA significantly below 200 MA
// = Bearish trend, LONG signal is counter-trend (risky)
score -= 10
reasons.push(`❌ MA bearish divergence (${params.maGap.toFixed(2)}%) = counter-trend risk (-10 pts)`)
}
} else if (params.direction === 'short') {
// SHORT scenarios (inverse of LONG logic)
if (params.maGap <= 0 && params.maGap > -2.0) {
// 50 MA below 200 MA, tight gap (-2% to 0%)
// = Bearish trend with momentum OR fresh death cross
score += 15
reasons.push(`🎯 MA bearish + tight gap (${params.maGap.toFixed(2)}%) = strong momentum (+15 pts)`)
} else if (params.maGap > 0 && params.maGap < 2.0) {
// 50 MA above 200 MA but converging (0% to 2%)
// = Potential death cross brewing
score += 12
reasons.push(`🌟 MA converging (${params.maGap.toFixed(2)}%) = death cross potential (+12 pts)`)
} else if (params.maGap <= -2.0 && params.maGap > -5.0) {
// 50 MA well below 200 MA (-5% to -2%)
// = Established bearish trend
score += 8
reasons.push(`📉 MA strong bearish trend (${params.maGap.toFixed(2)}%) (+8 pts)`)
} else if (params.maGap <= -5.0) {
// 50 MA far below 200 MA (<-5%)
// = Very extended, potential bounce risk
score += 5
reasons.push(`⚠️ MA extended bearish (${params.maGap.toFixed(2)}%) = oversold risk (+5 pts)`)
} else if (params.maGap >= 2.0) {
// 50 MA significantly above 200 MA
// = Bullish trend, SHORT signal is counter-trend (risky)
score -= 10
reasons.push(`❌ MA bullish divergence (${params.maGap.toFixed(2)}%) = counter-trend risk (-10 pts)`)
}
}
}
```
---
## Expected Impact
### Nov 25 21:15 Signal Reanalysis
**Original (v8):**
- Quality: 75 (blocked)
- ADX: 17.9 (weak)
- MA Gap: ≈ -1.0% (50 below 200, converging)
**With v9 MA Gap Enhancement:**
- Base quality: 75
- MA converging bonus: +12
- **New quality: 87** (closer but still blocked)
**Note:** Would still need ADX improvement OR slightly lower threshold (88-89?) to catch this specific signal. But the +12 points get us much closer and would catch signals with ADX 18-20.
### Alternative Scenarios
**Scenario A: MA Gap -0.5% (very tight convergence)**
- Quality 75 + 12 = **87** (close)
**Scenario B: MA Gap +0.5% (just crossed, tight)**
- Quality 75 + 15 = **90** ✅ **PASSES!**
**Scenario C: MA Gap +1.8% (recent golden cross, momentum strong)**
- Quality 75 + 15 = **90** ✅ **PASSES!**
---
## Implementation Checklist
### Phase 1: TradingView Indicator
- [ ] Add MA50 and MA200 calculations
- [ ] Calculate maGap percentage
- [ ] Add maGap to alert message payload
- [ ] Optional: Plot MA lines on chart
- [ ] Update indicatorVer from "v8" to "v9"
- [ ] Test on SOL-PERP 5min chart
### Phase 2: Backend Integration
- [ ] Update TypeScript interfaces for maGap parameter
- [ ] Modify scoreSignalQuality() with MA gap logic
- [ ] Update check-risk endpoint to accept maGap
- [ ] Update execute endpoint to accept maGap
- [ ] Add maGap to database fields (Trade table, BlockedSignal table)
### Phase 3: Testing & Validation
- [ ] Deploy v9 indicator to TradingView
- [ ] Trigger test signals manually
- [ ] Verify maGap calculation matches chart visual
- [ ] Check quality scores increase appropriately
- [ ] Monitor first 20-30 signals for validation
### Phase 4: Data Collection
- [ ] Collect 50+ v9 signals
- [ ] Compare v8 vs v9 win rates
- [ ] Analyze: Did MA gap bonus catch missed winners?
- [ ] SQL queries to validate improvement
- [ ] Adjust bonus points if needed (12/15 → 10/12 or 15/18)
---
## Success Metrics
**Target Improvements:**
1. **Catch signals like Nov 25 21:15:** Quality 75 + MA converging → 87-90 range
2. **Reduce false negatives:** Fewer blocked signals that would have been winners
3. **Maintain safety:** Don't add too many low-quality signals
4. **Win rate:** v9 should maintain or improve v8's 57.1% WR
**Validation Queries:**
```sql
-- Compare v9 MA gap bonus impact
SELECT
CASE
WHEN "signalQualityScore" >= 90 THEN 'Passed'
WHEN "signalQualityScore" >= 80 THEN 'Close (80-89)'
ELSE 'Blocked (<80)'
END as category,
COUNT(*) as signals,
AVG("scoreBreakdown"->>'maGap')::numeric as avg_ma_gap
FROM "BlockedSignal"
WHERE "indicatorVersion" = 'v9'
AND "scoreBreakdown"->>'maGap' IS NOT NULL
GROUP BY category;
-- Find signals that would pass with v9 but were blocked in v8
SELECT
TO_CHAR("createdAt", 'MM-DD HH24:MI') as time,
symbol, direction,
"signalQualityScore" as original_score,
-- Simulate v9 score (add 12 for converging, 15 for tight)
CASE
WHEN ("scoreBreakdown"->>'maGap')::numeric >= 0 AND ("scoreBreakdown"->>'maGap')::numeric < 2.0
THEN "signalQualityScore" + 15
WHEN ("scoreBreakdown"->>'maGap')::numeric < 0 AND ("scoreBreakdown"->>'maGap')::numeric > -2.0
THEN "signalQualityScore" + 12
ELSE "signalQualityScore"
END as v9_score,
"blockReason"
FROM "BlockedSignal"
WHERE "signalQualityScore" < 90 -- Was blocked
AND "indicatorVersion" = 'v8'
ORDER BY "createdAt" DESC
LIMIT 20;
```
---
## Risk Mitigation
### Potential Issues
1. **MA calculation lag:** 200-period MA requires significant history
- **Mitigation:** TradingView has full history, no issue
2. **Whipsaw during sideways markets:** MAs converge often in chop
- **Mitigation:** ADX filter still applies (weak ADX = less bonus effect)
3. **Over-optimization on single signal:** Nov 25 may be outlier
- **Mitigation:** Collect 50+ v9 signals before final judgment
4. **Bonus points too generous:** Could inflate scores artificially
- **Mitigation:** Start conservative (12/15), adjust based on data
### Rollback Plan
If v9 performs worse than v8:
1. Revert TradingView indicator to v8
2. Keep backend code but disable MA gap bonus
3. Analyze what went wrong (false positives? whipsaw signals?)
4. Redesign MA gap logic with tighter conditions
---
## Timeline
**Estimated Implementation Time:**
- TradingView changes: 30 minutes
- Backend integration: 1 hour
- Testing & deployment: 30 minutes
- **Total: ~2 hours**
**Data Collection:**
- Minimum 50 signals: 2-3 weeks (at ~3-5 signals/day)
- Comparative analysis: 1 week after 50 signals
**Decision Point:**
- After 50 v9 signals: Keep, adjust, or revert based on performance data
---
## Notes
- This enhancement preserves v8's early detection advantage
- Adds context awareness of MA positioning
- Rewards both imminent crossovers (converging) AND fresh crossovers (tight gap)
- Balances explosive potential (tight gaps) with trend stability (wider gaps)
- Counter-trend penalties prevent chasing wrong direction
**Key Insight:** v8 catches momentum shifts BEFORE visible MA crossovers. v9 validates those shifts by checking if MA structure supports the move.
---
**Created:** Nov 26, 2025
**Motivation:** $380 missed profit from Nov 25 21:15 blocked signal
**Expected Impact:** Catch 15-25% more profitable signals while maintaining quality standards

557
docs/roadmaps/OPTIMIZATION_MASTER_ROADMAP.md Normal file
View File

@@ -0,0 +1,557 @@
# Trading Bot Optimization - Master Roadmap
**Last Updated:** November 26, 2025
**Current Capital:** $540 USDC (zero debt, 100% health, 15x SOL leverage)
**Phase 1 Goal:** $106 → $2,500 (60%+ win rate, aggressive compounding)
**🎯 Recent Progress:**
- ✅ **HA Infrastructure COMPLETE** (Nov 25, 2025): Automatic DNS failover with Hostinger secondary server validated in production. Zero-downtime failover/failback operational.
- ✅ **v8 Indicator DEPLOYED** (Nov 22, 2025): 8 trades completed with perfect quality score separation (winners ≥95, losers ≤90). Quality threshold raised to 91.
- ✅ **Multi-Timeframe Quality Scoring** (Nov 26, 2025): All timeframes now get real quality scores (not hardcoded 0), enabling cross-timeframe win rate comparison.
---
## 🏗️ Infrastructure: High Availability Setup
**File:** [`HA_SETUP_ROADMAP.md`](./HA_SETUP_ROADMAP.md)
### Status: ✅ COMPLETE - PRODUCTION READY
**Completed:** November 25, 2025
**Cost:** ~$20-30/month (secondary server + monitoring)
**Uptime:** 99.9% guaranteed with automatic failover
### Implementation
- **Primary Server:** srvdocker02 (95.216.52.28) - trading-bot-v4:3001
- **Secondary Server:** Hostinger (72.62.39.24) - trading-bot-v4-secondary:3001
- **Database:** PostgreSQL streaming replication (asynchronous, <1s lag)
- **Monitoring:** dns-failover-monitor systemd service (30s checks, 3 failure threshold)
- **SSL:** nginx with HTTPS on both servers
- **Firewall:** pfSense health check rules configured
### Live Test Results (Nov 25, 21:53-22:00 CET)
- ✅ Detection Time: 90 seconds (3 × 30s health checks)
- ✅ Failover Execution: <1 second (DNS update via INWX API)
- ✅ Downtime: **0 seconds** (seamless secondary takeover)
- ✅ Failback: Automatic and immediate when primary recovers
- ✅ Data Integrity: Zero trade loss, database fully replicated
### Why This Matters
- **24/7 Operations:** Bot continues trading even if primary server crashes
- **Peace of Mind:** Automatic recovery while user sleeps/travels
- **Financial Protection:** No missed trades during infrastructure issues
- **Enterprise-Grade:** Same HA approach used by exchanges and financial platforms
### Documentation
- Complete deployment guide: `docs/DEPLOY_SECONDARY_MANUAL.md` (689 lines)
- Architecture diagrams, setup procedures, monitoring, troubleshooting
- Git commit: 99dc736 (November 25, 2025)
---
## Overview: Three Parallel Data-Driven Optimizations
All three initiatives follow the same pattern:
1. **Collect data** with current system (20-50 trades)
2. **Analyze patterns** via SQL backtesting
3. **Implement changes** with A/B testing
4. **Deploy if successful** (10%+ improvement required)
---
## 🎯 Initiative 1: Signal Quality Optimization
**File:** [`SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md`](./SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md)
### Purpose
Filter out bad trades BEFORE entry by optimizing quality score thresholds.
### Current Status
✅ **v8 Indicator DEPLOYED AND VALIDATED** (Nov 18-22, 2025)
- **Money Line v8:** Sticky trend detection with 0.6% flip threshold
- **Live Results:** 8 trades completed (57.1% WR, +$262.70 total P&L)
- **Perfect Separation:** ALL winners quality ≥95, ALL losers quality ≤90
- **Threshold Raised:** 91 minimum (from 60) based on data validation
- **Status:** Production system, quality 91+ filter active
- **Tracking:** Database tags trades as `indicatorVersion='v8'` for comparison
✅ **Phase 1.5 COMPLETE** - Signal Frequency Penalties Deployed (Nov 14)
- **Overtrading penalty:** 3+ signals in 30min → -20 points
- **Flip-flop penalty:** Opposite direction <15min → -25 points
- **Alternating pattern:** Last 3 trades flip → -30 points
- **Impact:** Eliminates tight-range flip-flops (like $141-145 consolidation)
📊 **Phase 1 (IN PROGRESS)** - Blocked Signals Collection
- **Progress:** 8/20 blocked signals collected (6 complete with price tracking)
- **Started:** November 11, 2025
- **Quality Threshold:** Raised to 91 (Nov 21) after perfect separation validation
- **System:** Automatically saves blocked signals to database with full metrics
### What's Being Collected
Every blocked signal saves:
- Signal metrics: ATR, ADX, RSI, volumeRatio, pricePosition, timeframe
- Quality score + breakdown (what caused low score)
- Block reason (QUALITY_SCORE_TOO_LOW, COOLDOWN, etc.)
- Frequency penalties applied (overtrading, flip-flop, alternating)
- Future: Price movement tracking (would it have hit TP1/TP2/SL?)
### Key Questions to Answer
- Are frequency penalties catching flip-flops effectively?
- Are we blocking good signals? (score 55-59 that would have won)
- Are we letting bad signals through? (score 65-70 that lose)
- Should threshold be 60? 65? 70? Symbol-specific?
### Next Phases (Planned)
- **Phase 6:** TradingView range compression metrics (Nov 2025)
- Detects compressed ranges and ADX-momentum mismatches
- ~1-2 hour implementation
- **Phase 7:** Volume profile integration (Dec 2025 - Q1 2026)
- Uses Volume S/R Zones V2 for consolidation detection
- Most powerful but most complex
### Timeline
- **Phase 1.5:** ✅ Complete (Nov 14, 2025)
- **Phase 1:** 1-2 weeks (need 10-20 blocked signals)
- **Phase 2:** 1 day (SQL analysis)
- **Phase 3:** 2-3 hours (adjust thresholds)
- **Phase 6:** 1-2 hours (TradingView alert modifications)
- **Phase 7:** 2-3 hours (volume profile integration)
- **Total:** ~3-4 weeks
### Success Metrics
- Win rate improves by 5-10%+ (current: ~45% → target: 55-60%)
- Eliminate flip-flop losses in consolidation zones
- Fewer losing trades in 50-60% drawdown range
- Maintain or increase trade frequency on valid trends
### SQL Queries Ready
See `BLOCKED_SIGNALS_TRACKING.md` for full query reference
---
## 📐 Initiative 2: Position Scaling & Exit Strategy
**File:** [`POSITION_SCALING_ROADMAP.md`](./POSITION_SCALING_ROADMAP.md)
### Purpose
Optimize HOW MUCH to close at each target and WHEN to move stops.
### Current Status
✅ **Phase 5 COMPLETE** - TP2-as-Runner Implemented (Nov 11)
- TP1 closes 75% (configurable via `TAKE_PROFIT_1_SIZE_PERCENT`)
- TP2 activates trailing stop on remaining 25%
- ATR-based dynamic trailing stop (Nov 11)
📊 **Phase 2-4 PENDING** - Need more trade data
- Requires 50+ trades with full MFE/MAE tracking
- Currently: 170+ trades total (8 v8 trades, collecting data)
### What's Being Collected
Every trade tracks:
- `maxFavorableExcursion` (MFE) - best profit % reached
- `maxAdverseExcursion` (MAE) - worst drawdown % reached
- `maxFavorablePrice` / `maxAdversePrice` - exact prices
- Actual TP1/TP2/SL hit vs theoretical optimal exits
### Key Questions to Answer
- Should TP1 be at 0.4% or 0.6%? (optimize via MFE data)
- Should runner be 25% or 35%? (quality-based sizing)
- When to move SL to breakeven? (after TP1 or earlier?)
- Should high-quality signals (95+) keep 35% runner vs 25%?
### Timeline
- **Phase 1:** ✅ COMPLETE
- **Phase 2:** 2-3 weeks (collect 50+ trades)
- **Phase 3:** 1 day (SQL analysis)
- **Phase 4:** 2-3 hours (implement quality-based scaling)
- **Total:** ~3-4 weeks
### Success Metrics
- 15%+ increase in total P&L
- Maintain 60%+ win rate
- Average winner size increases
- Runner exits capture extended moves
### Related Systems
- ATR-based trailing stop (✅ implemented Nov 11)
- TP2-as-runner activation (✅ implemented Nov 11)
- Quality-based position sizing (🔜 after data collection)
---
## 📊 Initiative 3: ATR-Based Take Profit Levels
**File:** [`ATR_BASED_TP_ROADMAP.md`](./ATR_BASED_TP_ROADMAP.md)
### Purpose
Replace fixed % targets with volatility-adaptive targets using ATR multipliers.
### Current Status
📊 **Phase 1 (IN PROGRESS)** - ATR Data Collection
- **Progress:** 8/50 trades with ATR tracking (v8 indicator)
- **Started:** November 11, 2025 (with indicator v6)
- **Median ATR:** 0.43 for SOL-PERP (from 162 historical trades)
- **System:** `atrAtEntry` field saved with every trade
### Current System (Fixed %)
```
TP1: Entry + 0.4% (always)
TP2: Entry + 0.7% (always)
SL: Entry - 1.0% (always)
```
### Proposed System (ATR Multipliers)
```
TP1: Entry + (ATR × 1.5) # Adaptive target
TP2: Entry + (ATR × 2.5) # Adaptive target
SL: Entry - (ATR × 2.0) # Adaptive stop
```
### What's Being Collected
Every trade now saves:
- `atrAtEntry` (Float) - ATR % when trade opened
- Entry price, exit price, realized P&L
- MFE/MAE to determine if ATR-based targets would have hit
### Key Questions to Answer
- Does 1.5x ATR TP1 hit more often than fixed 0.4%?
- Do ATR-based targets improve P&L by 10%+?
- What multipliers work best? (1.5x? 2.0x? Symbol-specific?)
- Should we use hybrid? (max of fixed % OR ATR-based)
### Example Comparison
**Current trade (ATR=0.26%, entry=$160.62):**
- Fixed 0.4% TP1: $161.27 (+$0.64)
- 1.5x ATR TP1: $161.25 (+$0.63) ← Almost identical!
**But in volatile market (ATR=0.50%):**
- Fixed 0.4% TP1: $161.27 (same)
- 1.5x ATR TP1: $161.83 (+0.75%) ← 88% wider, more room!
### Timeline
- **Phase 1:** 2-4 weeks (collect 50+ trades with ATR)
- **Phase 2:** 1-2 days (backtest analysis)
- **Phase 3:** 2-3 hours (implementation)
- **Phase 4:** 2-3 weeks (A/B testing)
- **Total:** ~6-8 weeks
### Success Metrics
- TP1 hit rate ≥ 75% (vs current ~70%)
- Win rate maintained at 60%+
- Total P&L improvement of 10%+
- Better performance in volatile vs calm markets
### SQL Backtest Ready
Complete backtest query in `ATR_BASED_TP_ROADMAP.md` (lines 100-150)
---
## 📅 Unified Timeline & Priorities
### Current Week (Nov 12-18, 2025)
**Focus:** Data collection for all three initiatives
- ✅ Systems deployed and collecting data automatically
- 🔄 Execute 10-15 trades (indicator v6 in production)
- 📊 Monitor: 0 blocked signals, 1 ATR-tracked trade, 160 total trades
### Week 2-3 (Nov 19 - Dec 2)
**Focus:** Signal Quality Analysis (fastest ROI)
- Should have 10-20 blocked signals by then
- Run SQL analysis on blocked vs executed trades
- Adjust quality thresholds if data shows improvement
- **Expected Impact:** +5% win rate, fewer bad trades
### Week 4-5 (Dec 3-16)
**Focus:** Continue data collection
- Target: 30-40 total trades with v6 + ATR data
- Monitor performance with any signal quality changes
- Begin preliminary ATR-based backtest analysis
### Week 6-8 (Dec 17 - Jan 6)
**Focus:** Position Scaling & ATR-based TP Analysis
- Should have 50+ trades by then
- Run comprehensive backtests on both initiatives
- Implement changes with highest expected value
- A/B test for 2-3 weeks
### Phase 1 Complete (Late January 2026)
**Target:** All three optimizations deployed and validated
- Signal quality: Optimized thresholds
- Position scaling: Quality-based runner sizing
- ATR-based TP: Volatility-adaptive targets (if backtest successful)
---
## 🎯 Expected Combined Impact
### Conservative Estimate
If each initiative improves performance by 10% independently:
- Signal Quality: +5% win rate → fewer losses
- Position Scaling: +15% average win size → bigger winners
- ATR-based TP: +10% total P&L → better hit rates
**Combined:** ~35-40% improvement in total P&L over 3 months
### Impact on Phase 1 Goal ($106 → $2,500)
**Current trajectory:** 20-30% monthly returns = 6-7 months
**With optimizations:** 25-35% monthly returns = 4-5 months
---
## 📊 Data Requirements Summary
| Initiative | Data Needed | Current Progress | Est. Completion |
|------------|-------------|------------------|-----------------|
| Signal Quality | 10-20 blocked signals | 0/20 (0%) | ~2 weeks |
| Position Scaling | 50+ trades w/ MFE/MAE | 160/50 (✅) but need v6 data | ~3 weeks |
| ATR-based TP | 50+ trades w/ ATR | 1/50 (2%) | ~4 weeks |
**Bottleneck:** Trade frequency (3-5 signals/day = 10-17 days for 50 trades)
---
## 🔧 Technical Implementation Status
### ✅ Already Implemented (Ready for data)
- `atrAtEntry` field in database (Nov 11)
- `indicatorVersion` tracking (Nov 12)
- ATR-based trailing stop (Nov 11)
- TP2-as-runner system (Nov 11)
- BlockedSignal table & auto-logging (Nov 11)
- MFE/MAE tracking (existing)
- Signal quality scoring v4 (Nov 11)
### 🔜 Needs Implementation (After data analysis)
- Quality threshold adjustments
- Quality-based position sizing
- ATR-based TP/SL calculation (optional toggle)
- Hybrid target system (max of fixed % or ATR)
### 📝 Configuration Ready
All systems have ENV variables and config structure ready:
- `MIN_SIGNAL_QUALITY_SCORE` (adjustable after analysis)
- `TAKE_PROFIT_1_SIZE_PERCENT` (70% default, adjustable)
- `USE_ATR_BASED_TARGETS` (false, will enable after backtest)
- `TP1_ATR_MULTIPLIER`, `TP2_ATR_MULTIPLIER`, `SL_ATR_MULTIPLIER`
---
## 📈 Progress Tracking
### Weekly Check-in Questions
1. How many trades executed this week?
2. How many blocked signals collected?
3. Any patterns emerging in blocked signals?
4. What's the current win rate vs target 60%?
5. Are MFE/MAE averages improving?
### Monthly Review Questions
1. Do we have enough data for next optimization phase?
2. What's the biggest win/loss this month and why?
3. Is indicator v6 outperforming v5? (need 20+ trades each)
4. Should we adjust any thresholds based on patterns?
5. Are we on track for Phase 1 goal ($2,500)?
---
## 🚨 Risk Management
### Don't Optimize Too Early
- ❌ Bad: Change thresholds after 5 trades and 2 blocked signals
- ✅ Good: Wait for 20 blocked signals + 50 trades minimum
- Statistical significance matters!
### Keep Historical Baseline
- Always compare against "what would the old system do?"
- Track `signalQualityVersion` and `indicatorVersion` for this
- Can revert if changes make things worse
### A/B Test Before Full Deploy
- Test new thresholds on 50% of signals (coin flip)
- Compare results after 20-30 trades
- Only deploy if statistically better (p < 0.05)
---
## 📚 Related Documentation
**Core System:**
- `copilot-instructions.md` - Full system architecture
- `TRADING_GOALS.md` - 8-phase financial roadmap ($106 → $1M+)
- `ATR_TRAILING_STOP_FIX.md` - Dynamic trailing stop implementation
**Data Analysis:**
- `BLOCKED_SIGNALS_TRACKING.md` - SQL queries for signal analysis
- `docs/analysis/SIGNAL_QUALITY_VERSION_ANALYSIS.sql` - Version comparison queries
**Implementation Guides:**
- `SIGNAL_QUALITY_SETUP_GUIDE.md` - How signal scoring works
- `PERCENTAGE_SIZING_FEATURE.md` - Position sizing system
---
## 🎯 Next Actions
### Immediate (This Week)
1. ✅ Deploy indicator v6 to TradingView production
2. 🔄 Execute 10-15 trades to start data collection
3. 📊 Monitor blocked signals (target: 2-3 this week)
4. 🎯 Verify current trade closes correctly with new fixes
### Short Term (2-3 Weeks)
1. Collect 10-20 blocked signals
2. Run signal quality analysis
3. Adjust MIN_SIGNAL_QUALITY_SCORE if data shows improvement
4. Continue collecting ATR data (target: 30-40 trades)
### Medium Term (4-8 Weeks)
1. Run position scaling backtest (50+ trades)
2. Run ATR-based TP backtest (50+ trades)
3. Implement quality-based position sizing
4. A/B test ATR-based targets vs fixed %
5. Deploy winning strategies
### Long Term (Phase 1 Complete)
1. Document all optimizations in `copilot-instructions.md`
2. Prepare for Phase 2 ($2,500 → $10,000)
3. Consider new optimizations:
- Time-of-day filtering
- Symbol-specific thresholds
- Volatility regime detection
- Machine learning for signal scoring
---
## 🚀 v9 Development Ideas (Nov 22, 2025 - Data Analysis)
**Based on 8 v8 trades + 36 v5/v6 archived trades pattern analysis**
### 1. Directional Filter (HIGHEST PRIORITY)
**Pattern Discovered:**
- **v8 LONGS:** 100% WR (3/3), +$565.03, Quality 98.3 avg, 174% avg MFE
- **v8 SHORTS:** 40% WR (2/5), -$283.54, Quality 91.0 avg, 23% avg MFE
- **v5/v6:** Shorts consistently outperform longs (50-60% WR vs 20-40%)
**Hypothesis:** Longs perform better across all indicator versions (44 total trades)
**v9 Options:**
- **Conservative:** Configurable `DIRECTIONAL_BIAS` setting (`long_only`, `short_only`, `both`)
- **Aggressive:** Smart direction filter - only trade direction with 7-day rolling WR ≥60%
- **Expected Impact:** Eliminate 60% of losses if pattern holds
**Data Needed:** 20 more v8 trades to validate (target: 28 total trades)
**Decision Point:** After trade #28, analyze long/short performance split
---
### 2. Time-of-Day Filter (MODERATE PRIORITY)
**Pattern Discovered:**
- **00-06 UTC (Asia):** 66.7% WR, +$241.43 (3 trades)
- **18-24 UTC (After):** 100% WR, +$257.56 (1 trade)
- **06-12 UTC (EU):** 0% WR, -$138.35 (1 trade)
- **12-18 UTC (US):** 66.7% WR, -$79.15 (3 trades)
**Hypothesis:** Asia/After-hours sessions outperform EU/US overlap
**v9 Options:**
- Configurable preferred trading hours (e.g., `TRADING_HOURS=0-6,18-24`)
- Block signals during low-performing sessions
- **Expected Impact:** ~15-20% improvement if pattern holds
**Data Needed:** 50+ trades to validate (sample size currently too small)
**Decision Point:** After 50 v8 trades, re-analyze time-of-day patterns
---
### 3. Quality-Based Emergency SL (SAFETY IMPROVEMENT)
**Pattern Discovered:**
- Trade cmi92gky: Quality 90, -$386.62 loss (-411% MAE)
- Only emergency exit in 8 trades, but 137% of total losses
- Emergency at -2% may be too generous for borderline quality signals
**Hypothesis:** Low-quality signals (90-94) need tighter emergency stops
**v9 Options:**
- Quality ≥95: Emergency at -2.0% (strong signal, give room)
- Quality 91-94: Emergency at -1.5% (moderate signal, tighter stop)
- Quality <91: BLOCKED (already implemented)
- **Expected Impact:** Cut worst-case losses by 25%
**Data Needed:** 10+ more trades in 91-94 quality range to validate
**Decision Point:** After 5+ emergency exits, analyze quality vs loss magnitude
---
### 4. Perfect Quality Threshold (ULTIMATE FILTER)
**Pattern Discovered:**
- Quality ≥95: 5 trades, 100% WR, +$906.39 (+$181/trade avg)
- Quality ≤90: 3 trades, 0% WR, -$624.90 (-$208/trade avg)
- **Perfect separation at 91 threshold validated**
**Hypothesis:** Raising to 95 after data collection = 100% WR
**v9 Options:**
- Keep threshold 91 until trade #28 (data collection)
- Raise to 95 after 20 more trades if 95+ pattern holds
- Zero tolerance for borderline signals
- **Expected Impact:** Potential 100% WR if pattern continues
**Data Needed:** 20 more trades (12 with quality ≥95 expected)
**Decision Point:** After trade #28, compare 91-94 vs 95+ performance
---
### 5. MFE/MAE-Based Position Sizing (ADVANCED)
**Pattern Discovered:**
- **Winners:** 137% avg MFE, -27% avg MAE (5:1 upside/downside)
- **Losers:** 10% avg MFE, -176% avg MAE (1:18 ratio)
- Winners move in our favor quickly, losers reverse hard
**Hypothesis:** Scale-in on early confirmation, fast-exit on early reversal
**v9 Options:**
- Open 50% position initially
- If profit ≥+0.5% within 5 minutes: Scale-in 50% more
- If loss ≥-0.3% within 5 minutes: Close entire position (fast exit)
- **Expected Impact:** Reduce loss exposure 50%, increase winner exposure 50%
**Data Needed:** 50+ trades with minute-by-minute price tracking
**Decision Point:** Phase 2 or Phase 3 (requires infrastructure changes)
---
### Development Timeline
**Current Phase (Trades 9-28):** Continue data collection, threshold 91
- Validate directional bias pattern
- Collect time-of-day data
- Monitor quality ≥95 performance
- Track emergency exits by quality tier
**After Trade #28:** Analyze patterns, decide v9 features
- If long bias validates → Implement directional filter (v9a)
- If time patterns validate → Add session filter (v9b)
- If quality 95+ = 100% → Raise threshold (v9c)
**After Trade #50:** Advanced features
- MFE/MAE-based scaling
- Machine learning quality scoring
- Adaptive emergency SL
**Priority Order (Impact × Ease):**
1. **Directional Filter** - Highest impact if validated
2. **Emergency SL by Quality** - High safety, easy implementation
3. **Raise Threshold to 95** - Zero effort, high impact if pattern holds
4. **Time-of-Day Filter** - Moderate impact, needs more data
5. **MFE/MAE Scaling** - Advanced, requires infrastructure
---
**Bottom Line:** Three complementary optimizations, all data-driven, all on track. Focus on collecting clean data now, analyze when we have enough, implement what works. No premature optimization. 📊🚀
**v9 Strategy:** Conservative approach - collect 20 more trades (target: 28 total), then make data-driven decisions about directional bias, quality thresholds, and safety improvements. Pattern recognition is powerful, but statistical significance requires larger sample sizes.

389
docs/roadmaps/POSITION_SCALING_ROADMAP.md Normal file
View File

@@ -0,0 +1,389 @@
# Position Scaling & Exit Optimization Roadmap
## Current State (October 31, 2025)
- **Total Trades:** 26 completed
- **P&L:** +$27.12 (38% win rate)
- **Shorts:** 6.6x more profitable than longs (+$2.46 vs -$0.37 avg)
- **Current Strategy:**
- TP1 at +1.5%: Close 75%
- TP2 at +3.0%: Close 80% of remaining (20% total)
- **Runner: 5% of position with 0.3% trailing stop ✅ ALREADY IMPLEMENTED**
- **Problem:** Small runner size (5%) + tight trailing stop (0.3%) may be suboptimal
- **Data Quality:** Ready to collect signal quality scores for correlation analysis
---
## Phase 1: Data Collection (CURRENT PHASE) 🔄
**Goal:** Gather 20-50 trades with quality scores before making strategy changes
### Data Points to Track:
- [ ] Signal quality score (0-100) for each trade
- [ ] Max Favorable Excursion (MFE) vs exit price differential
- [ ] ATR at entry vs actual price movement distance
- [ ] Time duration for winning trades (identify "quick wins" vs "runners")
- [ ] Quality score correlation with:
- [ ] Win rate
- [ ] Average P&L
- [ ] MFE (runner potential)
- [ ] Trade duration
### Analysis Queries (Run after 20+ scored trades):
```sql
-- Quality score vs performance
SELECT
CASE
WHEN "signalQualityScore" >= 80 THEN 'High (80-100)'
WHEN "signalQualityScore" >= 70 THEN 'Medium (70-79)'
ELSE 'Low (60-69)'
END as quality_tier,
COUNT(*) as trades,
ROUND(AVG("realizedPnL")::numeric, 2) as avg_pnl,
ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe,
ROUND(100.0 * SUM(CASE WHEN "realizedPnL" > 0 THEN 1 ELSE 0 END) / COUNT(*)::numeric, 1) as win_rate
FROM "Trade"
WHERE "signalQualityScore" IS NOT NULL AND "exitReason" IS NOT NULL
GROUP BY quality_tier
ORDER BY quality_tier;
-- ATR correlation with movement
SELECT
direction,
ROUND(AVG(atr)::numeric, 2) as avg_atr,
ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe,
ROUND(AVG(ABS("exitPrice" - "entryPrice") / "entryPrice" * 100)::numeric, 2) as avg_move_pct
FROM "Trade"
WHERE atr IS NOT NULL AND "exitReason" IS NOT NULL
GROUP BY direction;
-- Runner potential analysis (how many went beyond TP2?)
SELECT
"exitReason",
COUNT(*) as count,
ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe,
ROUND(AVG("realizedPnL")::numeric, 2) as avg_pnl,
-- MFE > 3% indicates runner potential
SUM(CASE WHEN "maxFavorableExcursion" > 3.0 THEN 1 ELSE 0 END) as runner_potential_count
FROM "Trade"
WHERE "exitReason" IS NOT NULL
GROUP BY "exitReason"
ORDER BY count DESC;
```
---
## Phase 2: ATR-Based Dynamic Targets ⏳
**Prerequisites:** ✅ 20+ trades with ATR data collected
### Implementation Tasks:
- [ ] **Add ATR normalization function** (`lib/trading/scaling-strategy.ts`)
- [ ] Calculate normalized ATR factor: `(current_ATR / baseline_ATR)`
- [ ] Baseline ATR = 2.0 for SOL-PERP (adjust based on data)
- [ ] **Update TP calculation in `lib/drift/orders.ts`**
- [ ] TP1: `entry + (1.5% × ATR_factor)` instead of fixed 1.5%
- [ ] TP2: `entry + (3.0% × ATR_factor)` instead of fixed 3.0%
- [ ] **Modify Position Manager monitoring loop**
- [ ] Store `atrFactor` in `ActiveTrade` interface
- [ ] Adjust dynamic SL movements by ATR factor
- [ ] Update breakeven trigger: `+0.5% × ATR_factor`
- [ ] Update profit lock trigger: `+1.2% × ATR_factor`
- [ ] **Testing:**
- [ ] Backtest on historical trades with ATR data
- [ ] Calculate improvement: old fixed % vs new ATR-adjusted
- [ ] Run 10 test trades before production
**Expected Outcome:** Wider targets in high volatility, tighter in low volatility
---
## Phase 3: Signal Quality-Based Scaling ⏳
**Prerequisites:** ✅ Phase 2 complete, ✅ 30+ trades with quality scores, ✅ Clear correlation proven
### Implementation Tasks:
- [ ] **Create quality tier configuration** (`config/trading.ts`)
```typescript
export interface QualityTierConfig {
minScore: number
maxScore: number
tp1Percentage: number // How much to take off
tp2Percentage: number
runnerPercentage: number
atrMultiplierTP1: number
atrMultiplierTP2: number
trailingStopATR: number
}
```
- [ ] **Define three tiers based on data analysis:**
- [ ] **High Quality (80-100):** Aggressive runner strategy
- TP1: 50% off at 2.0×ATR
- TP2: 25% off at 4.0×ATR
- Runner: 25% with 2.5×ATR trailing stop
- [ ] **Medium Quality (70-79):** Balanced (current-ish)
- TP1: 75% off at 1.5×ATR
- TP2: 25% off at 3.0×ATR
- Runner: None (full exit at TP2)
- [ ] **Low Quality (60-69):** Conservative quick exit
- TP1: 100% off at 1.0×ATR
- TP2: None
- Runner: None
- [ ] **Update `placeExitOrders()` function**
- [ ] Accept `qualityScore` parameter
- [ ] Select tier config based on score
- [ ] Place orders according to tier rules
- [ ] Only place TP2 order if tier has runner
- [ ] **Update Position Manager**
- [ ] Store `qualityScore` in `ActiveTrade`
- [ ] Apply tier-specific trailing stop logic
- [ ] Handle partial closes (50%, 75%, or 100%)
- [ ] **Database tracking:**
- [ ] Add `scalingTier` field to Trade model (high/medium/low)
- [ ] Track which tier was used for each trade
**Expected Outcome:** High quality signals let winners run, low quality signals take quick profits
---
## Phase 4: Direction-Based Optimization ⏳
**Prerequisites:** ✅ Phase 3 complete, ✅ Directional edge confirmed in 50+ trades
### Implementation Tasks:
- [ ] **Analyze directional performance** (Re-run after 50 trades)
- [ ] Compare long vs short win rates
- [ ] Compare long vs short avg P&L
- [ ] Compare long vs short MFE (runner potential)
- [ ] **Decision:** If shorts still 3x+ better, implement direction bias
- [ ] **Direction-specific configs** (`config/trading.ts`)
```typescript
export interface DirectionConfig {
shortTP1Pct: number // If shorts have edge, wider targets
shortTP2Pct: number
shortRunnerPct: number
longTP1Pct: number // If longs struggle, tighter defensive
longTP2Pct: number
longRunnerPct: number
}
```
- [ ] **Implementation in order placement:**
- [ ] Check `direction` field
- [ ] Apply direction-specific multipliers on top of quality tier
- [ ] Example: Short with high quality = 2.0×ATR × 1.2 (direction bonus)
- [ ] **A/B Testing:**
- [ ] Run 20 trades with direction bias
- [ ] Run 20 trades without (control group)
- [ ] Compare results before full rollout
**Expected Outcome:** Shorts get wider targets if edge persists, longs stay defensive
---
## Phase 5: Optimize Runner Size & Trailing Stop ⏳
**Prerequisites:** ✅ Phase 3 complete, ✅ Runner data collected (10+ trades with runners)
**Current Implementation:** ✅ Runner with trailing stop already exists!
- Runner size: 5% (configurable via `TAKE_PROFIT_2_SIZE_PERCENT=80`)
- Trailing stop: 0.3% fixed (configurable via `TRAILING_STOP_PERCENT=0.3`)
### Implementation Tasks:
- [ ] **Analyze runner performance from existing trades**
- [ ] Query trades where runner was active (TP2 hit)
- [ ] Calculate: How many runners hit trailing stop vs kept going?
- [ ] Calculate: Average runner profit vs optimal exit
- [ ] Calculate: Was 0.3% trailing stop too tight? (got stopped out too early?)
- [ ] **Optimize runner size by quality tier:**
- [ ] High quality (80-100): 25% runner (TP2 closes 0%, all becomes runner)
- [ ] Medium quality (70-79): 10% runner (TP2 closes 60% of remaining)
- [ ] Low quality (60-69): 5% runner (current behavior)
- [ ] **Make trailing stop ATR-based:**
- [ ] Change from fixed 0.3% to `(1.5 × ATR)` or `(2.0 × ATR)`
- [ ] Add `trailingStopATRMultiplier` config option
- [ ] Update Position Manager to use ATR-based trailing distance
- [ ] Store ATR value in ActiveTrade for dynamic calculations
- [ ] **Add runner-specific analytics:**
- [ ] Dashboard widget: Runner performance stats
- [ ] Show: Total profit from runners vs TP1/TP2
- [ ] Show: Average runner hold time
- [ ] Show: Runner win rate
- [ ] **Testing:**
- [ ] Backtest: Simulate larger runners (10-25%) on historical TP2 trades
- [ ] Backtest: Simulate ATR-based trailing stop vs fixed 0.3%
- [ ] A/B test: Run 10 trades with optimized settings before full rollout
**Expected Outcome:** Capture more profit from extended moves, reduce premature trailing stop exits
---
## Phase 6: Advanced ML-Based Exit Prediction (Future) 🔮
**Prerequisites:** ✅ 100+ trades with all metrics, ✅ Phases 1-5 complete
### Research Tasks:
- [ ] **Feature engineering:**
- [ ] Input features: ATR, ADX, RSI, volumeRatio, pricePosition, quality score, direction, timeframe
- [ ] Target variable: Did trade reach 2×TP2? (binary classification)
- [ ] Additional target: Max profit % reached (regression)
- [ ] **Model training:**
- [ ] Split data: 70% train, 30% test
- [ ] Try models: Logistic Regression, Random Forest, XGBoost
- [ ] Evaluate: Precision, Recall, F1 for runner prediction
- [ ] Cross-validation with time-based splits (avoid lookahead bias)
- [ ] **Integration:**
- [ ] `/api/trading/predict-runner` endpoint
- [ ] Call during trade execution to get runner probability
- [ ] Adjust runner size based on probability: 0-15% runner if low, 25-35% if high
- [ ] **Monitoring:**
- [ ] Track model accuracy over time
- [ ] Retrain monthly with new data
- [ ] A/B test: ML-based vs rule-based scaling
**Expected Outcome:** AI predicts which trades have runner potential before entry
---
## Quick Wins (Can Do Anytime) ⚡
- [ ] **Manual runner management for current trade**
- [ ] Move SL to +30% profit lock on current +41% SOL position
- [ ] Monitor manually until trend breaks
- [ ] Document outcome: How much did runner capture?
- [ ] **Add "runner stats" to analytics dashboard**
- [ ] Show: How many trades went beyond TP2?
- [ ] Show: Average MFE for TP2 exits
- [ ] Show: Estimated missed profit from not having runners
- [ ] **Database views for common queries**
- [ ] Create `vw_quality_performance` view
- [ ] Create `vw_runner_potential` view
- [ ] Create `vw_directional_edge` view
- [ ] **Alerts for exceptional trades**
- [ ] Telegram notification when MFE > 5% (runner candidate)
- [ ] Telegram notification when quality score > 90 (premium setup)
---
## Decision Gates 🚦
**Before Phase 2 (ATR-based):**
- ✅ Have 20+ trades with ATR data
- ✅ ATR values look reasonable (0.5 - 3.5 range)
- ✅ Clear volatility variation observed
**Before Phase 3 (Quality tiers):**
- ✅ Have 30+ trades with quality scores
- ✅ Statistical significance: High quality scores show measurably better outcomes
- ✅ Correlation coefficient > 0.3 between quality and P&L
**Before Phase 4 (Direction bias):**
- ✅ Have 50+ trades (25+ each direction)
- ✅ Directional edge persists (3x+ performance gap)
- ✅ Edge is consistent across different market conditions
**Before Phase 5 (Runners):**
- ✅ 30%+ of trades show MFE > TP2
- ✅ Average MFE significantly higher than TP2 level
- ✅ Phases 2-3 stable and profitable
**Before Phase 6 (ML):**
- ✅ 100+ trades with complete feature data
- ✅ Proven improvement from Phases 1-5
- ✅ Computational resources available (training time)
---
## Notes & Observations
### Current Trade Example (Oct 31, 2025):
- Entry: $182.73
- Current: $186.56 (+2.1%, +$11.30)
- **Actual P&L: +41% unrealized** 🚀
- **Status:** TP1 hit (closed 75%), TP2 hit (closed 20%), 5% runner still active with trailing stop
- **Lesson:** The 5% runner captured this move! But could a larger runner (10-25%) capture even more?
- **Trailing stop:** 0.3% below peak might be too tight, ATR-based (1.5-2.0×ATR) might work better
### Key Metrics to Watch:
- Win rate by quality tier
- Average MFE vs exit price gap
- Correlation between ATR and price movement
- Shorts vs longs performance delta
- Percentage of trades that go beyond TP2
### Strategy Validation:
Run this after each phase to validate improvement:
```sql
-- Compare old vs new strategy performance
SELECT
'Phase X' as phase,
COUNT(*) as trades,
ROUND(AVG("realizedPnL")::numeric, 2) as avg_pnl,
ROUND(SUM("realizedPnL")::numeric, 2) as total_pnl,
ROUND(100.0 * SUM(CASE WHEN "realizedPnL" > 0 THEN 1 ELSE 0 END) / COUNT(*)::numeric, 1) as win_rate
FROM "Trade"
WHERE "createdAt" >= '[phase_start_date]'
AND "exitReason" IS NOT NULL;
```
---
## Timeline Estimate
- **Phase 1 (Data Collection):** 2-4 weeks (depends on signal frequency)
- **Phase 2 (ATR-based):** 3-5 days implementation + 1 week testing
- **Phase 3 (Quality tiers):** 5-7 days implementation + 2 weeks testing
- **Phase 4 (Direction bias):** 2-3 days implementation + 1 week testing
- **Phase 5 (Runners):** 7-10 days implementation + 2 weeks testing
- **Phase 6 (ML):** 2-3 weeks research + implementation
**Total estimated time:** 3-4 months from start to Phase 5 complete
---
## Success Criteria
**Phase 2 Success:**
- [ ] Average P&L increases by 10%+ vs fixed targets
- [ ] Win rate stays stable or improves
- [ ] No increase in max drawdown
**Phase 3 Success:**
- [ ] High quality trades show 20%+ better P&L than low quality
- [ ] Overall P&L increases by 15%+ vs Phase 2
- [ ] Quality filtering prevents some losing trades
**Phase 4 Success:**
- [ ] Directional P&L gap narrows (improve weak direction)
- [ ] Or: Strong direction P&L improves further (if edge is real)
**Phase 5 Success:**
- [ ] Runners capture 20%+ more profit on winning trades
- [ ] Total P&L increases by 25%+ vs Phase 4
- [ ] Runners don't create new losing trades
**Overall Success (All Phases):**
- [ ] 2x total P&L vs baseline strategy
- [ ] 50%+ win rate (up from 38%)
- [ ] Average winner > 2× average loser
- [ ] Profit factor > 2.0
---
**Status:** Phase 1 (Data Collection) - Active 🔄
**Last Updated:** October 31, 2025
**Next Review:** After 20 trades with quality scores collected

605
docs/roadmaps/SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md Normal file
View File

@@ -0,0 +1,605 @@
# Signal Quality Optimization Roadmap
**Goal:** Optimize signal quality thresholds and scoring logic using data-driven analysis
**Current Status:** Phase 4 Complete ✅ - Automated price tracking operational | Phase 1-2 Active - Collecting + analyzing data
**Last Updated:** November 24, 2025
**🎉 MAJOR UPDATE: Phase 4 Complete (Nov 19-22, 2025)**
- BlockedSignalTracker deployed and fully operational
- 34 signals tracked: 21 multi-timeframe + 13 quality-blocked
- Price tracking at 1/5/15/30 min intervals with TP/SL hit detection
- First validation: Quality 80 signal blocked, would've profited +0.52%
**🚀 v8 Money Line Indicator (Nov 18, 2025)**
- Sticky trend detection with 0.6% flip threshold + momentum confirmation
- 8 trades completed: 57.1% WR, +$262.70 total, 93.6 avg quality
- Perfect quality separation: All winners ≥95, all losers ≤90
---
## Overview
This roadmap guides the systematic improvement of signal quality filtering. We follow a **data-first approach**: collect evidence, analyze patterns, then make changes. No premature optimization.
### Current System
- **Quality Score Threshold:** 91 points (raised from 81 on Nov 21, 2025 after trade #7 analysis)
- **Trade #7:** ADX 19.0, quality 90 → -$387 loss (weak trend, emergency stop)
- **Perfect Separation:** Analysis of 7 v8 trades:
- ALL 4 winners had quality ≥95 (95, 95, 100, 105) ✅
- ALL 3 losers had quality ≤90 (80, 90, 90) ❌
- 91 threshold would have prevented 100% of losses (-$624.90)
- **v8 Performance:** 7 trades, 57.1% WR, $262.70 total, 93.6 avg quality
- **Historical Performance (v5/v6):** 157 total, +$3.43 P&L, 44.5% WR
---
## Phase 1: Data Collection (CURRENT) ✅ IN PROGRESS
**Status:** Infrastructure complete, collecting data
**Started:** November 11, 2025
**Target:** Collect 10-20 blocked signals (1-2 weeks)
### Completed (Nov 11, 2025)
- [x] Created `BlockedSignal` database table
- [x] Implemented automatic saving in check-risk endpoint
- [x] Deployed to production (trading-bot-v4 container)
- [x] Created tracking documentation (BLOCKED_SIGNALS_TRACKING.md)
### What's Being Tracked
Every blocked signal captures:
- **Metrics:** ATR, ADX, RSI, volume ratio, price position, timeframe
- **Score:** Quality score (0-100), version, detailed breakdown
- **Block Reason:** Quality score, cooldown, hourly limit, daily drawdown
- **Context:** Symbol, direction, price at signal time, timestamp
### What We're Looking For
1. How many signals score 60-64 (just below threshold)?
2. What are their characteristics (ADX, ATR, price position)?
3. Are there patterns (extreme positions, specific timeframes)?
4. Do they cluster around specific block reasons?
### Phase 1 Completion Criteria
- [ ] Minimum 10 blocked signals with quality scores 55-64
- [ ] At least 2 signals in 60-64 range (close calls)
- [ ] Mix of block reasons (not all quality score)
- [ ] Data spans multiple market conditions (trending, choppy, volatile)
### SQL Queries for Phase 1
```sql
-- Check progress
SELECT COUNT(*) as total_blocked
FROM "BlockedSignal";
-- Score distribution
SELECT
CASE
WHEN signalQualityScore >= 60 THEN '60-64 (Close)'
WHEN signalQualityScore >= 55 THEN '55-59 (Marginal)'
WHEN signalQualityScore >= 50 THEN '50-54 (Weak)'
ELSE '0-49 (Very Weak)'
END as tier,
COUNT(*) as count
FROM "BlockedSignal"
WHERE blockReason = 'QUALITY_SCORE_TOO_LOW'
GROUP BY tier
ORDER BY MIN(signalQualityScore) DESC;
```
---
## Phase 1.5: Signal Frequency Penalties ✅ DEPLOYED Nov 14, 2025
**Status:** Production deployment complete (with critical bug fix)
**Initial Commit:** 111e3ed (07:28 CET)
**Bug Fix Commit:** 795026a (09:22 CET)
**Container:** trading-bot-v4
### What Was Implemented
Real-time database analysis that detects overtrading and flip-flop patterns before trade execution.
### Penalties Applied
1. **Overtrading (3+ signals in 30min):** -20 points
- Counts both executed trades AND blocked signals
- Prevents excessive trading in consolidation zones
2. **Flip-flop with price context (opposite direction <15min):** -25 points
- **NEW:** Only applies if price moved <2% from opposite signal
- Distinguishes chop from reversals
- Example chop: $154.50 SHORT → $154.30 LONG (0.13% move) = blocked ⚠️
- Example reversal: $170 SHORT → $153 LONG (10% move) = allowed ✅
- Blocks rapid long→short→long whipsaws in tight ranges
- **BUG FIX (795026a):** Now uses Pyth price data for accurate calculations
- Previous issue: showed "100% move" for 0.2% actual movement (allowed false flip-flop)
3. **Alternating pattern (last 3 trades):** -30 points
- Detects choppy market conditions
- Pattern: long→short→long = chop detection
### Technical Details
- **Function:** `getRecentSignals()` in `lib/database/trades.ts`
- **Price source:** Pyth price monitor via `getPythPriceMonitor()` in check-risk
- **Architecture:** `scoreSignalQuality()` now async
- **Endpoints updated:** check-risk, execute, reentry-check
- **Performance:** Indexed queries, <10ms overhead
- **Validation:** Logs "🔍 Flip-flop price check: $X → $Y = Z%" for debugging
### Expected Impact
- Eliminate tight-range flip-flops (Nov 14 chart: $141-145 SOL)
- Reduce overtrading during sideways markets
- Target: +5-10% win rate improvement
- Better capital preservation in chop
### Monitoring
Watch for detailed penalty and allowance messages in logs:
```
🔍 Flip-flop price check: $143.86 → $143.58 = 0.20%
⚠️ Overtrading zone: 3 signals in 30min (-20 pts)
⚠️ Flip-flop in tight range: 4min ago, only 0.20% move ($143.86 → $143.58) (-25 pts)
✅ Direction change after 10.0% move ($170.00 → $153.00, 12min ago) - reversal allowed
⚠️ Chop pattern: last 3 trades alternating (long → short → long) (-30 pts)
```
### Known Issues Fixed
- **Nov 14, 06:05 CET:** Initial deployment allowed 0.2% flip-flop due to incorrect price data
- Trade: SHORT at $143.58 (should have been blocked)
- Result: -$1.56 loss in 5 minutes
- Fix deployed: 09:22 CET (795026a) - now uses Pyth price monitor
### Validation Plan
1. Monitor next 5-10 signals for accurate price calculations
2. Verify penalties trigger with correct percentages
3. Analyze if blocked signals would have lost
4. If effective, proceed to Phase 6 (range compression)
---
## Phase 2: Pattern Analysis 🔜 NEXT
**Prerequisites:** 10-20 blocked signals collected
**Estimated Duration:** 2-3 days
**Owner:** Manual analysis + SQL queries
### Analysis Tasks
#### 2.1: Score Distribution Analysis
```sql
-- Analyze blocked signals by score range
SELECT
CASE
WHEN signalQualityScore >= 60 THEN '60-64'
WHEN signalQualityScore >= 55 THEN '55-59'
ELSE '50-54'
END as score_range,
COUNT(*) as count,
ROUND(AVG(atr)::numeric, 2) as avg_atr,
ROUND(AVG(adx)::numeric, 1) as avg_adx,
ROUND(AVG(pricePosition)::numeric, 1) as avg_price_pos,
ROUND(AVG(volumeRatio)::numeric, 2) as avg_volume
FROM "BlockedSignal"
WHERE blockReason = 'QUALITY_SCORE_TOO_LOW'
GROUP BY score_range
ORDER BY MIN(signalQualityScore) DESC;
```
#### 2.2: Compare with Executed Trades
```sql
-- Find executed trades with similar scores to blocked signals
SELECT
'Executed' as type,
signalQualityScore,
COUNT(*) as trades,
ROUND(AVG(realizedPnL)::numeric, 2) as avg_pnl,
ROUND(100.0 * SUM(CASE WHEN realizedPnL > 0 THEN 1 ELSE 0 END) / COUNT(*)::numeric, 1) as win_rate
FROM "Trade"
WHERE exitReason IS NOT NULL
AND signalQualityScore BETWEEN 60 AND 69
GROUP BY signalQualityScore
ORDER BY signalQualityScore;
```
#### 2.3: ADX Pattern Analysis
Key finding from existing data: ADX 20-25 is a trap zone!
```sql
-- ADX distribution in blocked signals
SELECT
CASE
WHEN adx >= 25 THEN 'Strong (25+)'
WHEN adx >= 20 THEN 'Moderate (20-25)'
WHEN adx >= 15 THEN 'Weak (15-20)'
ELSE 'Very Weak (<15)'
END as adx_tier,
COUNT(*) as count,
ROUND(AVG(signalQualityScore)::numeric, 1) as avg_score
FROM "BlockedSignal"
WHERE blockReason = 'QUALITY_SCORE_TOO_LOW'
AND adx IS NOT NULL
GROUP BY adx_tier
ORDER BY MIN(adx) DESC;
```
#### 2.4: Extreme Position Analysis
Test hypothesis: Extremes (<10% or >90%) need different thresholds
```sql
-- Blocked signals at range extremes
SELECT
direction,
signalQualityScore,
ROUND(pricePosition::numeric, 1) as pos,
ROUND(adx::numeric, 1) as adx,
ROUND(volumeRatio::numeric, 2) as vol
FROM "BlockedSignal"
WHERE blockReason = 'QUALITY_SCORE_TOO_LOW'
AND (pricePosition < 10 OR pricePosition > 90)
ORDER BY signalQualityScore DESC;
```
### Phase 2 Deliverables
- [ ] Score distribution report
- [ ] ADX pattern analysis
- [ ] Extreme position analysis
- [ ] Comparison with executed trades
- [ ] **DECISION:** Keep threshold at 65, lower to 60, or implement dual-threshold system
---
## Phase 3: Implementation (Conditional) 🎯 FUTURE
**Trigger:** Analysis shows clear pattern worth exploiting
**Prerequisites:** Phase 2 complete + statistical significance (15+ blocked signals)
### Option A: Dual-Threshold System (Recommended)
**IF** data shows extreme positions (price <10% or >90%) with scores 60-64 are profitable:
**Implementation:**
```typescript
// In check-risk endpoint
const isExtremePosition = pricePosition < 10 || pricePosition > 90
const requiredScore = isExtremePosition ? 60 : 65
if (qualityScore.score < requiredScore) {
// Block signal
}
```
**Changes Required:**
- `app/api/trading/check-risk/route.ts` - Add dual threshold logic
- `lib/trading/signal-quality.ts` - Add `isExtremePosition` helper
- `config/trading.ts` - Add `minScoreForExtremes` config option
- Update AI instructions with new logic
### Option B: ADX-Based Gates (Alternative)
**IF** data shows strong ADX trends (25+) with lower scores are profitable:
**Implementation:**
```typescript
const requiredScore = adx >= 25 ? 60 : 65
```
**Changes Required:**
- Similar to Option A but based on ADX threshold
### Option C: Keep Current (If No Clear Pattern)
**IF** data shows no consistent profit opportunity in blocked signals:
- No changes needed
- Continue monitoring
- Revisit in 20 more trades
### Phase 3 Checklist
- [ ] Decision made based on Phase 2 analysis
- [ ] Code changes implemented
- [ ] Updated signalQualityVersion to 'v5' in database
- [ ] AI instructions updated
- [ ] Tested with historical blocked signals
- [ ] Deployed to production
- [ ] Monitoring for 10 trades to validate improvement
---
## Phase 4: Price Analysis Automation ✅ COMPLETE
**Goal:** Automatically track if blocked signals would have been profitable
**Status:** Deployed Nov 19-22, 2025 - Fully operational
**Implementation:** BlockedSignalTracker background service
### Architecture ✅
```
BlockedSignalTracker (runs every 5 min)
↓
Fetch BlockedSignal records where:
- analysisComplete = false
- createdAt within last 30 minutes
↓
For each signal (both DATA_COLLECTION_ONLY and QUALITY_SCORE_TOO_LOW):
- Query current price from Drift oracle
- Calculate profit % based on direction
- Check if TP1 (~0.86%), TP2 (~1.72%), SL (~1.29%) would have hit
- Update priceAfter1Min/5Min/15Min/30Min
- Track maxFavorableExcursion and maxAdverseExcursion
- Set wouldHitTP1/TP2/SL flags
- Mark analysisComplete = true after 30 minutes
↓
Save results back to database
```
### Implementation Complete ✅
- [x] Created BlockedSignalTracker service (`lib/analysis/blocked-signal-tracker.ts`)
- [x] Price tracking from Drift oracle (not history, but real-time monitoring)
- [x] TP/SL hit calculation using ATR-based targets
- [x] Background job running every 5 minutes
- [x] Auto-starts on container startup
- [x] API endpoint (`/api/analytics/signal-tracking`) for monitoring
- [x] **Enhanced Nov 22:** Tracks both multi-timeframe AND quality-blocked signals
### Current Data (Nov 24, 2025)
- **34 total blocked signals tracked**
- 21 multi-timeframe (DATA_COLLECTION_ONLY): 19 complete
- 13 quality-blocked (QUALITY_SCORE_TOO_LOW): 6 complete
- **First validation result:** Quality 80 signal would have profited +0.52% (+$43)
### Success Metrics ✅
- System validates if blocks were correct (SL hits) vs missed opportunities (TP hits)
- Enables data-driven threshold optimization
- Risk-free analysis of non-executed signals
---
## Phase 5: ML-Based Scoring (DISTANT FUTURE) 🤖
**Status:** Future consideration - requires extensive data
**Estimated Start:** Q2 2026 or later
**Prerequisites:** 500+ trades with quality scores, proven manual optimization
### Approach
1. Build ML model to predict trade success from metrics
2. Use predicted success rate as quality score
3. Continuously learn from new trades
4. Auto-adjust weights based on market regime
### Requirements
- Sufficient training data (500+ trades minimum)
- Market regime classification system
- ML infrastructure (model training, deployment)
- Monitoring for model drift
### Risks
- Overfitting to past data
- Model degrades as markets change
- Black box decision-making
- Increased system complexity
**Note:** Only pursue if manual optimization plateaus or if pursuing ML as learning exercise.
---
## Phase 6: TradingView Range Compression Metrics (PLANNED) 📏
**Status:** Planned - Next after frequency penalties prove effective
**Estimated Start:** November 2025 (after Phase 1 validation)
**Prerequisites:** Phase 1 deployed, 5-10 signals monitored
### What It Does
Adds NEW metrics to TradingView alerts to detect range compression and momentum mismatches that indicate choppy markets.
### New TradingView Calculations
```javascript
// 1. Range compression (20-bar high/low range as % of price)
rangePercent = ((highest(high, 20) - lowest(low, 20)) / close) * 100
// 2. Price change over 5 bars (momentum check)
priceChange5bars = ((close - close[5]) / close[5]) * 100
// 3. ADX-momentum mismatch (ADX says trend but price not moving)
adxMismatch = (adx > 15 AND abs(priceChange5bars) < 0.3)
```
### Quality Score Penalties
- **Range < 1.5× ATR**: -20 points (compressed range = chop likely)
- **ADX 15+ but price change < 0.3%**: -20 points (fake trend signal)
- **Price oscillating around MA**: -15 points (whipsaw zone)
### Why This Helps
Current system can pass ADX 12-22 even when price just bouncing in tight zone. This detects the mismatch between "ADX says trending" vs "price says chopping."
**Example from Nov 14 chart:** Multiple signals in $141-145 range passed quality check despite obvious consolidation. Range compression would have caught this.
### Implementation Steps
1. Add calculations to TradingView strategy (30 min)
2. Update webhook JSON to include new fields (15 min)
3. Modify `scoreSignalQuality()` to use range metrics (30 min)
4. Test alerts on historical data (1 hour)
5. Deploy and monitor (ongoing)
### Expected Impact
- Catch "fake trends" where ADX misleads
- Reduce entries in tight consolidation zones
- Improve win rate by 3-5% in choppy markets
- Complement frequency penalties (Phase 1)
### Success Metrics
- Reduction in flip-flop losses
- Fewer blocked signals in validated trending moves
- Better P&L in sideways market conditions
---
## Phase 7: Volume Profile Integration (ADVANCED) 📊
**Status:** Future consideration - most complex but most powerful
**Estimated Start:** December 2025 or Q1 2026
**Prerequisites:** Phase 6 completed, Volume S/R Zones V2 indicator expertise
### What It Does
Uses Volume S/R Zones V2 indicator to detect when price is stuck in high-volume consolidation nodes where flip-flops are most likely.
### How Volume Profile Works
- Shows horizontal bars representing volume at each price level
- **Volume nodes** (thick bars) = high volume = price gets stuck (S/R zones)
- **Thin zones** (low volume) = price moves through quickly (breakout zones)
- Price bouncing inside volume node = high probability chop
### Required Indicator Modifications
Expose these values from Volume S/R Zones V2:
```javascript
// New indicator outputs (requires Pine Script modifications)
inVolumeNode: true/false // Is price inside thick volume bar?
nearVolumeEdge: true/false // Near top/bottom of node? (breakout setup)
nodeStrength: 0-100 // Volume concentration at this level
distanceFromNode: % // How far from nearest node?
volumeNodeWidth: % // How wide is current node? (tight = strong)
```
### Quality Score Adjustments
**Penalties:**
- **In volume node (stuck)**: -25 points (high chop probability)
- **Node strength > 80**: -30 points (VERY strong S/R = rejection likely)
- **Tight node width (<1%)**: -35 points (extreme consolidation)
**Bonuses:**
- **Near volume edge + high volume**: +10 points (breakout setup)
- **Distance from node > 2%**: +5 points (free movement zone)
- **Breaking through node with volume**: +15 points (momentum trade)
### Why This Is Powerful
**Example from Nov 14 chart:** Price bouncing $141-145. Volume profile would show THICK volume node at that exact level - instant warning. System would:
1. Detect price in node → apply -25 to -35 penalty
2. Block most entries in that zone
3. Wait for breakout above/below node
4. Bonus points when price clears node with volume
### Implementation Complexity
**High - Requires:**
1. Modify Volume S/R Zones indicator source code (Pine Script)
2. Expose new variables in indicator settings
3. Add webhook outputs from indicator (JSON formatting)
4. Parse in n8n workflow (new data structure)
5. Update quality scorer with volume logic (complex conditionals)
6. Test on historical data with indicator overlays
7. Validate against manual chart analysis
**Estimated Time:** 2-3 hours + TradingView Pine Script knowledge
### Risks & Considerations
- Indicator must stay updated (TradingView updates can break)
- Volume profile changes dynamically (recalculates with new data)
- May over-filter in ranging markets (miss valid mean-reversion trades)
- Complexity increases debugging difficulty
### Success Metrics
- Elimination of entries in obvious consolidation zones
- Higher win rate specifically in ranging markets
- Reduction in whipsaw losses (target: 30-50% fewer)
- Improved P&L per trade (better entries near node edges)
### Alternatives to Consider
- Use simpler "volume concentration" metric (easier to calculate)
- Implement fixed support/resistance zones instead of dynamic profile
- Combine with Phase 6 range compression (may be sufficient)
**Recommendation:** Only implement if Phase 1 + Phase 6 don't adequately solve flip-flop problem. Volume profile is most powerful but also most fragile.
---
## Progress Tracking
---
## Key Principles
### 1. Data Before Action
- Minimum 10 samples before any decision
- Prefer 20+ for statistical confidence
- No changes based on 1-2 outliers
### 2. Incremental Changes
- Change one variable at a time
- Test for 10-20 trades after each change
- Revert if performance degrades
### 3. Version Tracking
- Every scoring logic change gets new version (v4 → v5)
- Store version with each trade/blocked signal
- Enables A/B testing and rollback
### 4. Document Everything
- Update this roadmap after each phase
- Record decisions and rationale
- Link to SQL queries and analysis
---
## Progress Tracking
### Milestones
- [x] Nov 11, 2025: Phase 1 infrastructure complete (blocked signals tracking)
- [x] Nov 14, 2025: Phase 1.5 complete (signal frequency penalties deployed)
- [x] Nov 19, 2025: **Phase 4 COMPLETE** (multi-timeframe price tracking deployed)
- [x] Nov 22, 2025: **Phase 4 ENHANCED** (quality threshold validation added)
- [ ] Target: ~Nov 20-25, 2025: Phase 2 analysis complete (10-20 blocked signals)
- [ ] Target: ~Nov 25-30, 2025: Phase 3 implementation (threshold adjustment)
- [ ] Target: ~Dec 1-7, 2025: Phase 6 implementation (TradingView range metrics)
- [ ] Target: ~Dec 15-31, 2025: Phase 7 evaluation (Volume Profile integration)
- [ ] TBD: Phase 5 ML-based scoring (Q2 2026 or later)
### Metrics to Watch
- **Blocked signals collected:** 34/10 minimum ✅ (21 multi-timeframe + 13 quality-blocked)
- **Data collection complete:** 25/34 signals (73.5% completion rate) ✅
- **Quality-blocked signals:** 13 collected, 6 complete (Nov 22 enhancement)
- **Multi-timeframe signals:** 21 collected, 19 complete (Nov 19 deployment)
- **First validation:** Quality 80 blocked signal would've profited +0.52% (+$43) - FALSE NEGATIVE
- **Days of data collection:** 5 days (Nov 19-24) - ongoing
- **Market conditions covered:** 3/3 (trending, choppy, volatile) ✅
### Review Schedule
- **Weekly:** Check blocked signal count
- **After 10 blocked:** Run Phase 2 analysis
- **After Phase 2:** Decide on Phase 3 implementation
- **Monthly:** Review overall system performance
---
## Questions to Answer
### Phase 1 Questions
- [ ] How many signals get blocked per day?
- [ ] What's the score distribution of blocked signals?
- [ ] Are most blocks from quality score or other reasons?
### Phase 2 Questions
- [ ] Do blocked signals at 60-64 have common characteristics?
- [ ] Would lowering threshold to 60 improve performance?
- [ ] Do extreme positions need different treatment?
- [ ] Is ADX pattern valid in blocked signals?
### Phase 3 Questions
- [ ] Did the change improve win rate?
- [ ] Did it increase profitability?
- [ ] Any unintended side effects?
---
## Appendix: Historical Context
### Why This Roadmap Exists
**Date:** November 11, 2025
**Situation:** Three TradingView signals fired:
1. SHORT at 05:15 - Executed (score likely 65+) → Losing trade
2. LONG at 05:20 - Executed (score likely 65+) → Losing trade
3. SHORT at 05:30 - **BLOCKED** (score 45) → Would have been profitable
**User Question:** "What can we do about this?"
**Analysis Findings:**
- Only 2 historical trades scored 60-64 (both winners +$45.78)
- Sample size too small for confident decision
- ADX 20-25 is a trap zone (-$23.41 in 23 trades)
- Low volume (<0.8x) outperforms high volume (counterintuitive!)
**Decision:** Build data collection system instead of changing thresholds prematurely
**This Roadmap:** Systematic approach to optimization with proper data backing
---
**Remember:** The goal isn't to catch every winning trade. The goal is to optimize the **risk-adjusted return** by catching more winners than losers at each threshold level. Sometimes blocking a potential winner is correct if it also blocks 3 losers.