docs: add master optimization roadmap
Created unified roadmap consolidating all three optimization initiatives to reduce fragmentation and provide single source of truth. Three parallel data-driven optimizations: 1. Signal Quality (0/20 blocked signals) - 2-3 weeks 2. Position Scaling (160 trades, need v6 data) - 3-4 weeks 3. ATR-based TP (1/50 trades) - 6-8 weeks All follow same pattern: collect data → analyze → implement → A/B test Expected combined impact: 35-40% P&L improvement over 3 months 4-5 months to reach $2,500 Includes: - Unified timeline & priorities - Progress tracking framework - Weekly/monthly check-in questions - Risk management (no premature optimization) - Cross-references to all three roadmaps Single dashboard for all optimization efforts.
This commit is contained in:
mindesbunister
352
OPTIMIZATION_MASTER_ROADMAP.md
Normal file
352
OPTIMIZATION_MASTER_ROADMAP.md
Normal file
@@ -0,0 +1,352 @@
|
||||
# Trading Bot Optimization - Master Roadmap
|
||||
|
||||
**Last Updated:** November 12, 2025
|
||||
**Current Capital:** $97.55 USDC
|
||||
**Phase 1 Goal:** $106 → $2,500 (60%+ win rate, aggressive compounding)
|
||||
|
||||
---
|
||||
|
||||
## Overview: Three Parallel Data-Driven Optimizations
|
||||
|
||||
All three initiatives follow the same pattern:
|
||||
1. **Collect data** with current system (20-50 trades)
|
||||
2. **Analyze patterns** via SQL backtesting
|
||||
3. **Implement changes** with A/B testing
|
||||
4. **Deploy if successful** (10%+ improvement required)
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Initiative 1: Signal Quality Optimization
|
||||
|
||||
**File:** [`SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md`](./SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md)
|
||||
|
||||
### Purpose
|
||||
Filter out bad trades BEFORE entry by optimizing quality score thresholds.
|
||||
|
||||
### Current Status
|
||||
📊 **Phase 1 (IN PROGRESS)** - Blocked Signals Collection
|
||||
- **Progress:** 0/10-20 blocked signals needed
|
||||
- **Started:** November 11, 2025
|
||||
- **System:** Automatically saves blocked signals to database with full metrics
|
||||
|
||||
### What's Being Collected
|
||||
Every blocked signal saves:
|
||||
- Signal metrics: ATR, ADX, RSI, volumeRatio, pricePosition, timeframe
|
||||
- Quality score + breakdown (what caused low score)
|
||||
- Block reason (QUALITY_SCORE_TOO_LOW, COOLDOWN, etc.)
|
||||
- Future: Price movement tracking (would it have hit TP1/TP2/SL?)
|
||||
|
||||
### Key Questions to Answer
|
||||
- Are we blocking good signals? (score 55-59 that would have won)
|
||||
- Are we letting bad signals through? (score 65-70 that lose)
|
||||
- Should threshold be 60? 65? 70? Symbol-specific?
|
||||
|
||||
### Timeline
|
||||
- **Phase 1:** 1-2 weeks (need 10-20 blocked signals)
|
||||
- **Phase 2:** 1 day (SQL analysis)
|
||||
- **Phase 3:** 2-3 hours (adjust thresholds)
|
||||
- **Total:** ~2-3 weeks
|
||||
|
||||
### Success Metrics
|
||||
- Win rate improves by 5%+ (60% → 65%)
|
||||
- Fewer losing trades in 50-60% drawdown range
|
||||
- Maintain or increase trade frequency
|
||||
|
||||
### SQL Queries Ready
|
||||
See `BLOCKED_SIGNALS_TRACKING.md` for full query reference
|
||||
|
||||
---
|
||||
|
||||
## 📐 Initiative 2: Position Scaling & Exit Strategy
|
||||
|
||||
**File:** [`POSITION_SCALING_ROADMAP.md`](./POSITION_SCALING_ROADMAP.md)
|
||||
|
||||
### Purpose
|
||||
Optimize HOW MUCH to close at each target and WHEN to move stops.
|
||||
|
||||
### Current Status
|
||||
✅ **Phase 5 COMPLETE** - TP2-as-Runner Implemented (Nov 11)
|
||||
- TP1 closes 75% (configurable via `TAKE_PROFIT_1_SIZE_PERCENT`)
|
||||
- TP2 activates trailing stop on remaining 25%
|
||||
- ATR-based dynamic trailing stop (Nov 11)
|
||||
|
||||
📊 **Phase 2-4 PENDING** - Need more trade data
|
||||
- Requires 50+ trades with full MFE/MAE tracking
|
||||
- Currently: 160 trades total, need more with v6 indicator data
|
||||
|
||||
### What's Being Collected
|
||||
Every trade tracks:
|
||||
- `maxFavorableExcursion` (MFE) - best profit % reached
|
||||
- `maxAdverseExcursion` (MAE) - worst drawdown % reached
|
||||
- `maxFavorablePrice` / `maxAdversePrice` - exact prices
|
||||
- Actual TP1/TP2/SL hit vs theoretical optimal exits
|
||||
|
||||
### Key Questions to Answer
|
||||
- Should TP1 be at 0.4% or 0.6%? (optimize via MFE data)
|
||||
- Should runner be 25% or 35%? (quality-based sizing)
|
||||
- When to move SL to breakeven? (after TP1 or earlier?)
|
||||
- Should high-quality signals (95+) keep 35% runner vs 25%?
|
||||
|
||||
### Timeline
|
||||
- **Phase 1:** ✅ COMPLETE
|
||||
- **Phase 2:** 2-3 weeks (collect 50+ trades)
|
||||
- **Phase 3:** 1 day (SQL analysis)
|
||||
- **Phase 4:** 2-3 hours (implement quality-based scaling)
|
||||
- **Total:** ~3-4 weeks
|
||||
|
||||
### Success Metrics
|
||||
- 15%+ increase in total P&L
|
||||
- Maintain 60%+ win rate
|
||||
- Average winner size increases
|
||||
- Runner exits capture extended moves
|
||||
|
||||
### Related Systems
|
||||
- ATR-based trailing stop (✅ implemented Nov 11)
|
||||
- TP2-as-runner activation (✅ implemented Nov 11)
|
||||
- Quality-based position sizing (🔜 after data collection)
|
||||
|
||||
---
|
||||
|
||||
## 📊 Initiative 3: ATR-Based Take Profit Levels
|
||||
|
||||
**File:** [`ATR_BASED_TP_ROADMAP.md`](./ATR_BASED_TP_ROADMAP.md)
|
||||
|
||||
### Purpose
|
||||
Replace fixed % targets with volatility-adaptive targets using ATR multipliers.
|
||||
|
||||
### Current Status
|
||||
📊 **Phase 1 (IN PROGRESS)** - ATR Data Collection
|
||||
- **Progress:** 1/50 trades with ATR tracking
|
||||
- **Started:** November 11, 2025 (with indicator v6)
|
||||
- **System:** `atrAtEntry` field saved with every trade
|
||||
|
||||
### Current System (Fixed %)
|
||||
```
|
||||
TP1: Entry + 0.4% (always)
|
||||
TP2: Entry + 0.7% (always)
|
||||
SL: Entry - 1.0% (always)
|
||||
```
|
||||
|
||||
### Proposed System (ATR Multipliers)
|
||||
```
|
||||
TP1: Entry + (ATR × 1.5) # Adaptive target
|
||||
TP2: Entry + (ATR × 2.5) # Adaptive target
|
||||
SL: Entry - (ATR × 2.0) # Adaptive stop
|
||||
```
|
||||
|
||||
### What's Being Collected
|
||||
Every trade now saves:
|
||||
- `atrAtEntry` (Float) - ATR % when trade opened
|
||||
- Entry price, exit price, realized P&L
|
||||
- MFE/MAE to determine if ATR-based targets would have hit
|
||||
|
||||
### Key Questions to Answer
|
||||
- Does 1.5x ATR TP1 hit more often than fixed 0.4%?
|
||||
- Do ATR-based targets improve P&L by 10%+?
|
||||
- What multipliers work best? (1.5x? 2.0x? Symbol-specific?)
|
||||
- Should we use hybrid? (max of fixed % OR ATR-based)
|
||||
|
||||
### Example Comparison
|
||||
**Current trade (ATR=0.26%, entry=$160.62):**
|
||||
- Fixed 0.4% TP1: $161.27 (+$0.64)
|
||||
- 1.5x ATR TP1: $161.25 (+$0.63) ← Almost identical!
|
||||
|
||||
**But in volatile market (ATR=0.50%):**
|
||||
- Fixed 0.4% TP1: $161.27 (same)
|
||||
- 1.5x ATR TP1: $161.83 (+0.75%) ← 88% wider, more room!
|
||||
|
||||
### Timeline
|
||||
- **Phase 1:** 2-4 weeks (collect 50+ trades with ATR)
|
||||
- **Phase 2:** 1-2 days (backtest analysis)
|
||||
- **Phase 3:** 2-3 hours (implementation)
|
||||
- **Phase 4:** 2-3 weeks (A/B testing)
|
||||
- **Total:** ~6-8 weeks
|
||||
|
||||
### Success Metrics
|
||||
- TP1 hit rate ≥ 75% (vs current ~70%)
|
||||
- Win rate maintained at 60%+
|
||||
- Total P&L improvement of 10%+
|
||||
- Better performance in volatile vs calm markets
|
||||
|
||||
### SQL Backtest Ready
|
||||
Complete backtest query in `ATR_BASED_TP_ROADMAP.md` (lines 100-150)
|
||||
|
||||
---
|
||||
|
||||
## 📅 Unified Timeline & Priorities
|
||||
|
||||
### Current Week (Nov 12-18, 2025)
|
||||
**Focus:** Data collection for all three initiatives
|
||||
- ✅ Systems deployed and collecting data automatically
|
||||
- 🔄 Execute 10-15 trades (indicator v6 in production)
|
||||
- 📊 Monitor: 0 blocked signals, 1 ATR-tracked trade, 160 total trades
|
||||
|
||||
### Week 2-3 (Nov 19 - Dec 2)
|
||||
**Focus:** Signal Quality Analysis (fastest ROI)
|
||||
- Should have 10-20 blocked signals by then
|
||||
- Run SQL analysis on blocked vs executed trades
|
||||
- Adjust quality thresholds if data shows improvement
|
||||
- **Expected Impact:** +5% win rate, fewer bad trades
|
||||
|
||||
### Week 4-5 (Dec 3-16)
|
||||
**Focus:** Continue data collection
|
||||
- Target: 30-40 total trades with v6 + ATR data
|
||||
- Monitor performance with any signal quality changes
|
||||
- Begin preliminary ATR-based backtest analysis
|
||||
|
||||
### Week 6-8 (Dec 17 - Jan 6)
|
||||
**Focus:** Position Scaling & ATR-based TP Analysis
|
||||
- Should have 50+ trades by then
|
||||
- Run comprehensive backtests on both initiatives
|
||||
- Implement changes with highest expected value
|
||||
- A/B test for 2-3 weeks
|
||||
|
||||
### Phase 1 Complete (Late January 2026)
|
||||
**Target:** All three optimizations deployed and validated
|
||||
- Signal quality: Optimized thresholds
|
||||
- Position scaling: Quality-based runner sizing
|
||||
- ATR-based TP: Volatility-adaptive targets (if backtest successful)
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Expected Combined Impact
|
||||
|
||||
### Conservative Estimate
|
||||
If each initiative improves performance by 10% independently:
|
||||
- Signal Quality: +5% win rate → fewer losses
|
||||
- Position Scaling: +15% average win size → bigger winners
|
||||
- ATR-based TP: +10% total P&L → better hit rates
|
||||
|
||||
**Combined:** ~35-40% improvement in total P&L over 3 months
|
||||
|
||||
### Impact on Phase 1 Goal ($106 → $2,500)
|
||||
**Current trajectory:** 20-30% monthly returns = 6-7 months
|
||||
**With optimizations:** 25-35% monthly returns = 4-5 months
|
||||
|
||||
---
|
||||
|
||||
## 📊 Data Requirements Summary
|
||||
|
||||
| Initiative | Data Needed | Current Progress | Est. Completion |
|
||||
|------------|-------------|------------------|-----------------|
|
||||
| Signal Quality | 10-20 blocked signals | 0/20 (0%) | ~2 weeks |
|
||||
| Position Scaling | 50+ trades w/ MFE/MAE | 160/50 (✅) but need v6 data | ~3 weeks |
|
||||
| ATR-based TP | 50+ trades w/ ATR | 1/50 (2%) | ~4 weeks |
|
||||
|
||||
**Bottleneck:** Trade frequency (3-5 signals/day = 10-17 days for 50 trades)
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Technical Implementation Status
|
||||
|
||||
### ✅ Already Implemented (Ready for data)
|
||||
- `atrAtEntry` field in database (Nov 11)
|
||||
- `indicatorVersion` tracking (Nov 12)
|
||||
- ATR-based trailing stop (Nov 11)
|
||||
- TP2-as-runner system (Nov 11)
|
||||
- BlockedSignal table & auto-logging (Nov 11)
|
||||
- MFE/MAE tracking (existing)
|
||||
- Signal quality scoring v4 (Nov 11)
|
||||
|
||||
### 🔜 Needs Implementation (After data analysis)
|
||||
- Quality threshold adjustments
|
||||
- Quality-based position sizing
|
||||
- ATR-based TP/SL calculation (optional toggle)
|
||||
- Hybrid target system (max of fixed % or ATR)
|
||||
|
||||
### 📝 Configuration Ready
|
||||
All systems have ENV variables and config structure ready:
|
||||
- `MIN_SIGNAL_QUALITY_SCORE` (adjustable after analysis)
|
||||
- `TAKE_PROFIT_1_SIZE_PERCENT` (70% default, adjustable)
|
||||
- `USE_ATR_BASED_TARGETS` (false, will enable after backtest)
|
||||
- `TP1_ATR_MULTIPLIER`, `TP2_ATR_MULTIPLIER`, `SL_ATR_MULTIPLIER`
|
||||
|
||||
---
|
||||
|
||||
## 📈 Progress Tracking
|
||||
|
||||
### Weekly Check-in Questions
|
||||
1. How many trades executed this week?
|
||||
2. How many blocked signals collected?
|
||||
3. Any patterns emerging in blocked signals?
|
||||
4. What's the current win rate vs target 60%?
|
||||
5. Are MFE/MAE averages improving?
|
||||
|
||||
### Monthly Review Questions
|
||||
1. Do we have enough data for next optimization phase?
|
||||
2. What's the biggest win/loss this month and why?
|
||||
3. Is indicator v6 outperforming v5? (need 20+ trades each)
|
||||
4. Should we adjust any thresholds based on patterns?
|
||||
5. Are we on track for Phase 1 goal ($2,500)?
|
||||
|
||||
---
|
||||
|
||||
## 🚨 Risk Management
|
||||
|
||||
### Don't Optimize Too Early
|
||||
- ❌ Bad: Change thresholds after 5 trades and 2 blocked signals
|
||||
- ✅ Good: Wait for 20 blocked signals + 50 trades minimum
|
||||
- Statistical significance matters!
|
||||
|
||||
### Keep Historical Baseline
|
||||
- Always compare against "what would the old system do?"
|
||||
- Track `signalQualityVersion` and `indicatorVersion` for this
|
||||
- Can revert if changes make things worse
|
||||
|
||||
### A/B Test Before Full Deploy
|
||||
- Test new thresholds on 50% of signals (coin flip)
|
||||
- Compare results after 20-30 trades
|
||||
- Only deploy if statistically better (p < 0.05)
|
||||
|
||||
---
|
||||
|
||||
## 📚 Related Documentation
|
||||
|
||||
**Core System:**
|
||||
- `copilot-instructions.md` - Full system architecture
|
||||
- `TRADING_GOALS.md` - 8-phase financial roadmap ($106 → $1M+)
|
||||
- `ATR_TRAILING_STOP_FIX.md` - Dynamic trailing stop implementation
|
||||
|
||||
**Data Analysis:**
|
||||
- `BLOCKED_SIGNALS_TRACKING.md` - SQL queries for signal analysis
|
||||
- `docs/analysis/SIGNAL_QUALITY_VERSION_ANALYSIS.sql` - Version comparison queries
|
||||
|
||||
**Implementation Guides:**
|
||||
- `SIGNAL_QUALITY_SETUP_GUIDE.md` - How signal scoring works
|
||||
- `PERCENTAGE_SIZING_FEATURE.md` - Position sizing system
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Next Actions
|
||||
|
||||
### Immediate (This Week)
|
||||
1. ✅ Deploy indicator v6 to TradingView production
|
||||
2. 🔄 Execute 10-15 trades to start data collection
|
||||
3. 📊 Monitor blocked signals (target: 2-3 this week)
|
||||
4. 🎯 Verify current trade closes correctly with new fixes
|
||||
|
||||
### Short Term (2-3 Weeks)
|
||||
1. Collect 10-20 blocked signals
|
||||
2. Run signal quality analysis
|
||||
3. Adjust MIN_SIGNAL_QUALITY_SCORE if data shows improvement
|
||||
4. Continue collecting ATR data (target: 30-40 trades)
|
||||
|
||||
### Medium Term (4-8 Weeks)
|
||||
1. Run position scaling backtest (50+ trades)
|
||||
2. Run ATR-based TP backtest (50+ trades)
|
||||
3. Implement quality-based position sizing
|
||||
4. A/B test ATR-based targets vs fixed %
|
||||
5. Deploy winning strategies
|
||||
|
||||
### Long Term (Phase 1 Complete)
|
||||
1. Document all optimizations in `copilot-instructions.md`
|
||||
2. Prepare for Phase 2 ($2,500 → $10,000)
|
||||
3. Consider new optimizations:
|
||||
- Time-of-day filtering
|
||||
- Symbol-specific thresholds
|
||||
- Volatility regime detection
|
||||
- Machine learning for signal scoring
|
||||
|
||||
---
|
||||
|
||||
**Bottom Line:** Three complementary optimizations, all data-driven, all on track. Focus on collecting clean data now, analyze when we have enough, implement what works. No premature optimization. 📊🚀
|
||||
Reference in New Issue
Block a user