From 71c856c720000be331d901585c61fd584eca3aff Mon Sep 17 00:00:00 2001 From: mindesbunister Date: Wed, 12 Nov 2025 12:35:44 +0100 Subject: [PATCH] docs: add master optimization roadmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Created unified roadmap consolidating all three optimization initiatives to reduce fragmentation and provide single source of truth. Three parallel data-driven optimizations: 1. Signal Quality (0/20 blocked signals) - 2-3 weeks 2. Position Scaling (160 trades, need v6 data) - 3-4 weeks 3. ATR-based TP (1/50 trades) - 6-8 weeks All follow same pattern: collect data → analyze → implement → A/B test Expected combined impact: 35-40% P&L improvement over 3 months 4-5 months to reach $2,500 Includes: - Unified timeline & priorities - Progress tracking framework - Weekly/monthly check-in questions - Risk management (no premature optimization) - Cross-references to all three roadmaps Single dashboard for all optimization efforts. --- OPTIMIZATION_MASTER_ROADMAP.md | 352 +++++++++++++++++++++++++++++++++ 1 file changed, 352 insertions(+) create mode 100644 OPTIMIZATION_MASTER_ROADMAP.md diff --git a/OPTIMIZATION_MASTER_ROADMAP.md b/OPTIMIZATION_MASTER_ROADMAP.md new file mode 100644 index 0000000..b27754c --- /dev/null +++ b/OPTIMIZATION_MASTER_ROADMAP.md @@ -0,0 +1,352 @@ +# Trading Bot Optimization - Master Roadmap + +**Last Updated:** November 12, 2025 +**Current Capital:** $97.55 USDC +**Phase 1 Goal:** $106 → $2,500 (60%+ win rate, aggressive compounding) + +--- + +## Overview: Three Parallel Data-Driven Optimizations + +All three initiatives follow the same pattern: +1. **Collect data** with current system (20-50 trades) +2. **Analyze patterns** via SQL backtesting +3. **Implement changes** with A/B testing +4. **Deploy if successful** (10%+ improvement required) + +--- + +## 🎯 Initiative 1: Signal Quality Optimization + +**File:** [`SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md`](./SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md) + +### Purpose +Filter out bad trades BEFORE entry by optimizing quality score thresholds. + +### Current Status +📊 **Phase 1 (IN PROGRESS)** - Blocked Signals Collection +- **Progress:** 0/10-20 blocked signals needed +- **Started:** November 11, 2025 +- **System:** Automatically saves blocked signals to database with full metrics + +### What's Being Collected +Every blocked signal saves: +- Signal metrics: ATR, ADX, RSI, volumeRatio, pricePosition, timeframe +- Quality score + breakdown (what caused low score) +- Block reason (QUALITY_SCORE_TOO_LOW, COOLDOWN, etc.) +- Future: Price movement tracking (would it have hit TP1/TP2/SL?) + +### Key Questions to Answer +- Are we blocking good signals? (score 55-59 that would have won) +- Are we letting bad signals through? (score 65-70 that lose) +- Should threshold be 60? 65? 70? Symbol-specific? + +### Timeline +- **Phase 1:** 1-2 weeks (need 10-20 blocked signals) +- **Phase 2:** 1 day (SQL analysis) +- **Phase 3:** 2-3 hours (adjust thresholds) +- **Total:** ~2-3 weeks + +### Success Metrics +- Win rate improves by 5%+ (60% → 65%) +- Fewer losing trades in 50-60% drawdown range +- Maintain or increase trade frequency + +### SQL Queries Ready +See `BLOCKED_SIGNALS_TRACKING.md` for full query reference + +--- + +## 📐 Initiative 2: Position Scaling & Exit Strategy + +**File:** [`POSITION_SCALING_ROADMAP.md`](./POSITION_SCALING_ROADMAP.md) + +### Purpose +Optimize HOW MUCH to close at each target and WHEN to move stops. + +### Current Status +✅ **Phase 5 COMPLETE** - TP2-as-Runner Implemented (Nov 11) +- TP1 closes 75% (configurable via `TAKE_PROFIT_1_SIZE_PERCENT`) +- TP2 activates trailing stop on remaining 25% +- ATR-based dynamic trailing stop (Nov 11) + +📊 **Phase 2-4 PENDING** - Need more trade data +- Requires 50+ trades with full MFE/MAE tracking +- Currently: 160 trades total, need more with v6 indicator data + +### What's Being Collected +Every trade tracks: +- `maxFavorableExcursion` (MFE) - best profit % reached +- `maxAdverseExcursion` (MAE) - worst drawdown % reached +- `maxFavorablePrice` / `maxAdversePrice` - exact prices +- Actual TP1/TP2/SL hit vs theoretical optimal exits + +### Key Questions to Answer +- Should TP1 be at 0.4% or 0.6%? (optimize via MFE data) +- Should runner be 25% or 35%? (quality-based sizing) +- When to move SL to breakeven? (after TP1 or earlier?) +- Should high-quality signals (95+) keep 35% runner vs 25%? + +### Timeline +- **Phase 1:** ✅ COMPLETE +- **Phase 2:** 2-3 weeks (collect 50+ trades) +- **Phase 3:** 1 day (SQL analysis) +- **Phase 4:** 2-3 hours (implement quality-based scaling) +- **Total:** ~3-4 weeks + +### Success Metrics +- 15%+ increase in total P&L +- Maintain 60%+ win rate +- Average winner size increases +- Runner exits capture extended moves + +### Related Systems +- ATR-based trailing stop (✅ implemented Nov 11) +- TP2-as-runner activation (✅ implemented Nov 11) +- Quality-based position sizing (🔜 after data collection) + +--- + +## 📊 Initiative 3: ATR-Based Take Profit Levels + +**File:** [`ATR_BASED_TP_ROADMAP.md`](./ATR_BASED_TP_ROADMAP.md) + +### Purpose +Replace fixed % targets with volatility-adaptive targets using ATR multipliers. + +### Current Status +📊 **Phase 1 (IN PROGRESS)** - ATR Data Collection +- **Progress:** 1/50 trades with ATR tracking +- **Started:** November 11, 2025 (with indicator v6) +- **System:** `atrAtEntry` field saved with every trade + +### Current System (Fixed %) +``` +TP1: Entry + 0.4% (always) +TP2: Entry + 0.7% (always) +SL: Entry - 1.0% (always) +``` + +### Proposed System (ATR Multipliers) +``` +TP1: Entry + (ATR × 1.5) # Adaptive target +TP2: Entry + (ATR × 2.5) # Adaptive target +SL: Entry - (ATR × 2.0) # Adaptive stop +``` + +### What's Being Collected +Every trade now saves: +- `atrAtEntry` (Float) - ATR % when trade opened +- Entry price, exit price, realized P&L +- MFE/MAE to determine if ATR-based targets would have hit + +### Key Questions to Answer +- Does 1.5x ATR TP1 hit more often than fixed 0.4%? +- Do ATR-based targets improve P&L by 10%+? +- What multipliers work best? (1.5x? 2.0x? Symbol-specific?) +- Should we use hybrid? (max of fixed % OR ATR-based) + +### Example Comparison +**Current trade (ATR=0.26%, entry=$160.62):** +- Fixed 0.4% TP1: $161.27 (+$0.64) +- 1.5x ATR TP1: $161.25 (+$0.63) ← Almost identical! + +**But in volatile market (ATR=0.50%):** +- Fixed 0.4% TP1: $161.27 (same) +- 1.5x ATR TP1: $161.83 (+0.75%) ← 88% wider, more room! + +### Timeline +- **Phase 1:** 2-4 weeks (collect 50+ trades with ATR) +- **Phase 2:** 1-2 days (backtest analysis) +- **Phase 3:** 2-3 hours (implementation) +- **Phase 4:** 2-3 weeks (A/B testing) +- **Total:** ~6-8 weeks + +### Success Metrics +- TP1 hit rate ≥ 75% (vs current ~70%) +- Win rate maintained at 60%+ +- Total P&L improvement of 10%+ +- Better performance in volatile vs calm markets + +### SQL Backtest Ready +Complete backtest query in `ATR_BASED_TP_ROADMAP.md` (lines 100-150) + +--- + +## 📅 Unified Timeline & Priorities + +### Current Week (Nov 12-18, 2025) +**Focus:** Data collection for all three initiatives +- ✅ Systems deployed and collecting data automatically +- 🔄 Execute 10-15 trades (indicator v6 in production) +- 📊 Monitor: 0 blocked signals, 1 ATR-tracked trade, 160 total trades + +### Week 2-3 (Nov 19 - Dec 2) +**Focus:** Signal Quality Analysis (fastest ROI) +- Should have 10-20 blocked signals by then +- Run SQL analysis on blocked vs executed trades +- Adjust quality thresholds if data shows improvement +- **Expected Impact:** +5% win rate, fewer bad trades + +### Week 4-5 (Dec 3-16) +**Focus:** Continue data collection +- Target: 30-40 total trades with v6 + ATR data +- Monitor performance with any signal quality changes +- Begin preliminary ATR-based backtest analysis + +### Week 6-8 (Dec 17 - Jan 6) +**Focus:** Position Scaling & ATR-based TP Analysis +- Should have 50+ trades by then +- Run comprehensive backtests on both initiatives +- Implement changes with highest expected value +- A/B test for 2-3 weeks + +### Phase 1 Complete (Late January 2026) +**Target:** All three optimizations deployed and validated +- Signal quality: Optimized thresholds +- Position scaling: Quality-based runner sizing +- ATR-based TP: Volatility-adaptive targets (if backtest successful) + +--- + +## 🎯 Expected Combined Impact + +### Conservative Estimate +If each initiative improves performance by 10% independently: +- Signal Quality: +5% win rate → fewer losses +- Position Scaling: +15% average win size → bigger winners +- ATR-based TP: +10% total P&L → better hit rates + +**Combined:** ~35-40% improvement in total P&L over 3 months + +### Impact on Phase 1 Goal ($106 → $2,500) +**Current trajectory:** 20-30% monthly returns = 6-7 months +**With optimizations:** 25-35% monthly returns = 4-5 months + +--- + +## 📊 Data Requirements Summary + +| Initiative | Data Needed | Current Progress | Est. Completion | +|------------|-------------|------------------|-----------------| +| Signal Quality | 10-20 blocked signals | 0/20 (0%) | ~2 weeks | +| Position Scaling | 50+ trades w/ MFE/MAE | 160/50 (✅) but need v6 data | ~3 weeks | +| ATR-based TP | 50+ trades w/ ATR | 1/50 (2%) | ~4 weeks | + +**Bottleneck:** Trade frequency (3-5 signals/day = 10-17 days for 50 trades) + +--- + +## 🔧 Technical Implementation Status + +### ✅ Already Implemented (Ready for data) +- `atrAtEntry` field in database (Nov 11) +- `indicatorVersion` tracking (Nov 12) +- ATR-based trailing stop (Nov 11) +- TP2-as-runner system (Nov 11) +- BlockedSignal table & auto-logging (Nov 11) +- MFE/MAE tracking (existing) +- Signal quality scoring v4 (Nov 11) + +### 🔜 Needs Implementation (After data analysis) +- Quality threshold adjustments +- Quality-based position sizing +- ATR-based TP/SL calculation (optional toggle) +- Hybrid target system (max of fixed % or ATR) + +### 📝 Configuration Ready +All systems have ENV variables and config structure ready: +- `MIN_SIGNAL_QUALITY_SCORE` (adjustable after analysis) +- `TAKE_PROFIT_1_SIZE_PERCENT` (70% default, adjustable) +- `USE_ATR_BASED_TARGETS` (false, will enable after backtest) +- `TP1_ATR_MULTIPLIER`, `TP2_ATR_MULTIPLIER`, `SL_ATR_MULTIPLIER` + +--- + +## 📈 Progress Tracking + +### Weekly Check-in Questions +1. How many trades executed this week? +2. How many blocked signals collected? +3. Any patterns emerging in blocked signals? +4. What's the current win rate vs target 60%? +5. Are MFE/MAE averages improving? + +### Monthly Review Questions +1. Do we have enough data for next optimization phase? +2. What's the biggest win/loss this month and why? +3. Is indicator v6 outperforming v5? (need 20+ trades each) +4. Should we adjust any thresholds based on patterns? +5. Are we on track for Phase 1 goal ($2,500)? + +--- + +## 🚨 Risk Management + +### Don't Optimize Too Early +- ❌ Bad: Change thresholds after 5 trades and 2 blocked signals +- ✅ Good: Wait for 20 blocked signals + 50 trades minimum +- Statistical significance matters! + +### Keep Historical Baseline +- Always compare against "what would the old system do?" +- Track `signalQualityVersion` and `indicatorVersion` for this +- Can revert if changes make things worse + +### A/B Test Before Full Deploy +- Test new thresholds on 50% of signals (coin flip) +- Compare results after 20-30 trades +- Only deploy if statistically better (p < 0.05) + +--- + +## 📚 Related Documentation + +**Core System:** +- `copilot-instructions.md` - Full system architecture +- `TRADING_GOALS.md` - 8-phase financial roadmap ($106 → $1M+) +- `ATR_TRAILING_STOP_FIX.md` - Dynamic trailing stop implementation + +**Data Analysis:** +- `BLOCKED_SIGNALS_TRACKING.md` - SQL queries for signal analysis +- `docs/analysis/SIGNAL_QUALITY_VERSION_ANALYSIS.sql` - Version comparison queries + +**Implementation Guides:** +- `SIGNAL_QUALITY_SETUP_GUIDE.md` - How signal scoring works +- `PERCENTAGE_SIZING_FEATURE.md` - Position sizing system + +--- + +## 🎯 Next Actions + +### Immediate (This Week) +1. ✅ Deploy indicator v6 to TradingView production +2. 🔄 Execute 10-15 trades to start data collection +3. 📊 Monitor blocked signals (target: 2-3 this week) +4. 🎯 Verify current trade closes correctly with new fixes + +### Short Term (2-3 Weeks) +1. Collect 10-20 blocked signals +2. Run signal quality analysis +3. Adjust MIN_SIGNAL_QUALITY_SCORE if data shows improvement +4. Continue collecting ATR data (target: 30-40 trades) + +### Medium Term (4-8 Weeks) +1. Run position scaling backtest (50+ trades) +2. Run ATR-based TP backtest (50+ trades) +3. Implement quality-based position sizing +4. A/B test ATR-based targets vs fixed % +5. Deploy winning strategies + +### Long Term (Phase 1 Complete) +1. Document all optimizations in `copilot-instructions.md` +2. Prepare for Phase 2 ($2,500 → $10,000) +3. Consider new optimizations: + - Time-of-day filtering + - Symbol-specific thresholds + - Volatility regime detection + - Machine learning for signal scoring + +--- + +**Bottom Line:** Three complementary optimizations, all data-driven, all on track. Focus on collecting clean data now, analyze when we have enough, implement what works. No premature optimization. 📊🚀