trading_bot_v4/docs/roadmaps/OPTIMIZATION_MASTER_ROADMAP.md

Trading Bot Optimization - Master Roadmap

Last Updated: November 26, 2025
Current Capital: $540 USDC (zero debt, 100% health, 15x SOL leverage)
Phase 1 Goal: $106 → $2,500 (60%+ win rate, aggressive compounding)

🎯 Recent Progress:

• ✅ HA Infrastructure COMPLETE (Nov 25, 2025): Automatic DNS failover with Hostinger secondary server validated in production. Zero-downtime failover/failback operational.
• ✅ v8 Indicator DEPLOYED (Nov 22, 2025): 8 trades completed with perfect quality score separation (winners ≥95, losers ≤90). Quality threshold raised to 91.
• ✅ Multi-Timeframe Quality Scoring (Nov 26, 2025): All timeframes now get real quality scores (not hardcoded 0), enabling cross-timeframe win rate comparison.

---

🏗️ Infrastructure: High Availability Setup

File: HA_SETUP_ROADMAP.md

Status: ✅ COMPLETE - PRODUCTION READY

Completed: November 25, 2025
Cost: ~$20-30/month (secondary server + monitoring)
Uptime: 99.9% guaranteed with automatic failover

Implementation

• Primary Server: srvdocker02 (95.216.52.28) - trading-bot-v4:3001
• Secondary Server: Hostinger (72.62.39.24) - trading-bot-v4-secondary:3001
• Database: PostgreSQL streaming replication (asynchronous, <1s lag)
• Monitoring: dns-failover-monitor systemd service (30s checks, 3 failure threshold)
• SSL: nginx with HTTPS on both servers
• Firewall: pfSense health check rules configured

Live Test Results (Nov 25, 21:53-22:00 CET)

• ✅ Detection Time: 90 seconds (3 × 30s health checks)
• ✅ Failover Execution: <1 second (DNS update via INWX API)
• ✅ Downtime: 0 seconds (seamless secondary takeover)
• ✅ Failback: Automatic and immediate when primary recovers
• ✅ Data Integrity: Zero trade loss, database fully replicated

Why This Matters

• 24/7 Operations: Bot continues trading even if primary server crashes
• Peace of Mind: Automatic recovery while user sleeps/travels
• Financial Protection: No missed trades during infrastructure issues
• Enterprise-Grade: Same HA approach used by exchanges and financial platforms

Documentation

• Complete deployment guide: docs/DEPLOY_SECONDARY_MANUAL.md (689 lines)
• Architecture diagrams, setup procedures, monitoring, troubleshooting
• Git commit: 99dc736 (November 25, 2025)

---

Overview: Three Parallel Data-Driven Optimizations

All three initiatives follow the same pattern:

1. Collect data with current system (20-50 trades)
2. Analyze patterns via SQL backtesting
3. Implement changes with A/B testing
4. Deploy if successful (10%+ improvement required)

---

🎯 Initiative 1: Signal Quality Optimization

File: SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md

Purpose

Filter out bad trades BEFORE entry by optimizing quality score thresholds.

Current Status

✅ v8 Indicator DEPLOYED AND VALIDATED (Nov 18-22, 2025)

• Money Line v8: Sticky trend detection with 0.6% flip threshold
• Live Results: 8 trades completed (57.1% WR, +$262.70 total P&L)
• Perfect Separation: ALL winners quality ≥95, ALL losers quality ≤90
• Threshold Raised: 91 minimum (from 60) based on data validation
• Status: Production system, quality 91+ filter active
• Tracking: Database tags trades as indicatorVersion='v8' for comparison

✅ Phase 1.5 COMPLETE - Signal Frequency Penalties Deployed (Nov 14)

• Overtrading penalty: 3+ signals in 30min → -20 points
• Flip-flop penalty: Opposite direction <15min → -25 points
• Alternating pattern: Last 3 trades flip → -30 points
• Impact: Eliminates tight-range flip-flops (like $141-145 consolidation)

📊 Phase 1 (IN PROGRESS) - Blocked Signals Collection

• Progress: 8/20 blocked signals collected (6 complete with price tracking)
• Started: November 11, 2025
• Quality Threshold: Raised to 91 (Nov 21) after perfect separation validation
• System: Automatically saves blocked signals to database with full metrics

What's Being Collected

Every blocked signal saves:

• Signal metrics: ATR, ADX, RSI, volumeRatio, pricePosition, timeframe
• Quality score + breakdown (what caused low score)
• Block reason (QUALITY_SCORE_TOO_LOW, COOLDOWN, etc.)
• Frequency penalties applied (overtrading, flip-flop, alternating)
• Future: Price movement tracking (would it have hit TP1/TP2/SL?)

Key Questions to Answer

• Are frequency penalties catching flip-flops effectively?
• Are we blocking good signals? (score 55-59 that would have won)
• Are we letting bad signals through? (score 65-70 that lose)
• Should threshold be 60? 65? 70? Symbol-specific?

Next Phases (Planned)

• Phase 6: TradingView range compression metrics (Nov 2025)
  • Detects compressed ranges and ADX-momentum mismatches
  • ~1-2 hour implementation
• Phase 7: Volume profile integration (Dec 2025 - Q1 2026)
  • Uses Volume S/R Zones V2 for consolidation detection
  • Most powerful but most complex

Timeline

• Phase 1.5: ✅ Complete (Nov 14, 2025)
• Phase 1: 1-2 weeks (need 10-20 blocked signals)
• Phase 2: 1 day (SQL analysis)
• Phase 3: 2-3 hours (adjust thresholds)
• Phase 6: 1-2 hours (TradingView alert modifications)
• Phase 7: 2-3 hours (volume profile integration)
• Total: ~3-4 weeks

Success Metrics

• Win rate improves by 5-10%+ (current: ~45% → target: 55-60%)
• Eliminate flip-flop losses in consolidation zones
• Fewer losing trades in 50-60% drawdown range
• Maintain or increase trade frequency on valid trends

SQL Queries Ready

See BLOCKED_SIGNALS_TRACKING.md for full query reference

---

📐 Initiative 2: Position Scaling & Exit Strategy

File: POSITION_SCALING_ROADMAP.md

Purpose

Optimize HOW MUCH to close at each target and WHEN to move stops.

Current Status

✅ Phase 5 COMPLETE - TP2-as-Runner Implemented (Nov 11)

• TP1 closes 75% (configurable via TAKE_PROFIT_1_SIZE_PERCENT)
• TP2 activates trailing stop on remaining 25%
• ATR-based dynamic trailing stop (Nov 11)

📊 Phase 2-4 PENDING - Need more trade data

• Requires 50+ trades with full MFE/MAE tracking
• Currently: 170+ trades total (8 v8 trades, collecting data)

What's Being Collected

Every trade tracks:

• maxFavorableExcursion (MFE) - best profit % reached
• maxAdverseExcursion (MAE) - worst drawdown % reached
• maxFavorablePrice / maxAdversePrice - exact prices
• Actual TP1/TP2/SL hit vs theoretical optimal exits

Key Questions to Answer

• Should TP1 be at 0.4% or 0.6%? (optimize via MFE data)
• Should runner be 25% or 35%? (quality-based sizing)
• When to move SL to breakeven? (after TP1 or earlier?)
• Should high-quality signals (95+) keep 35% runner vs 25%?

Timeline

• Phase 1: ✅ COMPLETE
• Phase 2: 2-3 weeks (collect 50+ trades)
• Phase 3: 1 day (SQL analysis)
• Phase 4: 2-3 hours (implement quality-based scaling)
• Total: ~3-4 weeks

Success Metrics

• 15%+ increase in total P&L
• Maintain 60%+ win rate
• Average winner size increases
• Runner exits capture extended moves

Related Systems

• ATR-based trailing stop (✅ implemented Nov 11)
• TP2-as-runner activation (✅ implemented Nov 11)
• Quality-based position sizing (🔜 after data collection)

---

📊 Initiative 3: ATR-Based Take Profit Levels

File: ATR_BASED_TP_ROADMAP.md

Purpose

Replace fixed % targets with volatility-adaptive targets using ATR multipliers.

Current Status

📊 Phase 1 (IN PROGRESS) - ATR Data Collection

• Progress: 8/50 trades with ATR tracking (v8 indicator)
• Started: November 11, 2025 (with indicator v6)
• Median ATR: 0.43 for SOL-PERP (from 162 historical trades)
• System: atrAtEntry field saved with every trade

Current System (Fixed %)

TP1: Entry + 0.4% (always)
TP2: Entry + 0.7% (always)
SL:  Entry - 1.0% (always)

Proposed System (ATR Multipliers)

TP1: Entry + (ATR × 1.5)  # Adaptive target
TP2: Entry + (ATR × 2.5)  # Adaptive target
SL:  Entry - (ATR × 2.0)  # Adaptive stop

What's Being Collected

Every trade now saves:

• atrAtEntry (Float) - ATR % when trade opened
• Entry price, exit price, realized P&L
• MFE/MAE to determine if ATR-based targets would have hit

Key Questions to Answer

• Does 1.5x ATR TP1 hit more often than fixed 0.4%?
• Do ATR-based targets improve P&L by 10%+?
• What multipliers work best? (1.5x? 2.0x? Symbol-specific?)
• Should we use hybrid? (max of fixed % OR ATR-based)

Example Comparison

Current trade (ATR=0.26%, entry=$160.62):

• Fixed 0.4% TP1: $161.27 (+$0.64)
• 1.5x ATR TP1: $161.25 (+$0.63) ← Almost identical!

But in volatile market (ATR=0.50%):

• Fixed 0.4% TP1: $161.27 (same)
• 1.5x ATR TP1: $161.83 (+0.75%) ← 88% wider, more room!

Timeline

• Phase 1: 2-4 weeks (collect 50+ trades with ATR)
• Phase 2: 1-2 days (backtest analysis)
• Phase 3: 2-3 hours (implementation)
• Phase 4: 2-3 weeks (A/B testing)
• Total: ~6-8 weeks

Success Metrics

• TP1 hit rate ≥ 75% (vs current ~70%)
• Win rate maintained at 60%+
• Total P&L improvement of 10%+
• Better performance in volatile vs calm markets

SQL Backtest Ready

Complete backtest query in ATR_BASED_TP_ROADMAP.md (lines 100-150)

---

📅 Unified Timeline & Priorities

Current Week (Nov 12-18, 2025)

Focus: Data collection for all three initiatives

• ✅ Systems deployed and collecting data automatically
• 🔄 Execute 10-15 trades (indicator v6 in production)
• 📊 Monitor: 0 blocked signals, 1 ATR-tracked trade, 160 total trades

Week 2-3 (Nov 19 - Dec 2)

Focus: Signal Quality Analysis (fastest ROI)

• Should have 10-20 blocked signals by then
• Run SQL analysis on blocked vs executed trades
• Adjust quality thresholds if data shows improvement
• Expected Impact: +5% win rate, fewer bad trades

Week 4-5 (Dec 3-16)

Focus: Continue data collection

• Target: 30-40 total trades with v6 + ATR data
• Monitor performance with any signal quality changes
• Begin preliminary ATR-based backtest analysis

Week 6-8 (Dec 17 - Jan 6)

Focus: Position Scaling & ATR-based TP Analysis

• Should have 50+ trades by then
• Run comprehensive backtests on both initiatives
• Implement changes with highest expected value
• A/B test for 2-3 weeks

Phase 1 Complete (Late January 2026)

Target: All three optimizations deployed and validated

• Signal quality: Optimized thresholds
• Position scaling: Quality-based runner sizing
• ATR-based TP: Volatility-adaptive targets (if backtest successful)

---

🎯 Expected Combined Impact

Conservative Estimate

If each initiative improves performance by 10% independently:

• Signal Quality: +5% win rate → fewer losses
• Position Scaling: +15% average win size → bigger winners
• ATR-based TP: +10% total P&L → better hit rates

Combined: ~35-40% improvement in total P&L over 3 months

Impact on Phase 1 Goal ($106 → $2,500)

Current trajectory: 20-30% monthly returns = 6-7 months With optimizations: 25-35% monthly returns = 4-5 months

---

📊 Data Requirements Summary

Initiative	Data Needed	Current Progress	Est. Completion
Signal Quality	10-20 blocked signals	0/20 (0%)	~2 weeks
Position Scaling	50+ trades w/ MFE/MAE	160/50 (✅) but need v6 data	~3 weeks
ATR-based TP	50+ trades w/ ATR	1/50 (2%)	~4 weeks

Bottleneck: Trade frequency (3-5 signals/day = 10-17 days for 50 trades)

---

🔧 Technical Implementation Status

✅ Already Implemented (Ready for data)

• atrAtEntry field in database (Nov 11)
• indicatorVersion tracking (Nov 12)
• ATR-based trailing stop (Nov 11)
• TP2-as-runner system (Nov 11)
• BlockedSignal table & auto-logging (Nov 11)
• MFE/MAE tracking (existing)
• Signal quality scoring v4 (Nov 11)

🔜 Needs Implementation (After data analysis)

• Quality threshold adjustments
• Quality-based position sizing
• ATR-based TP/SL calculation (optional toggle)
• Hybrid target system (max of fixed % or ATR)

📝 Configuration Ready

All systems have ENV variables and config structure ready:

• MIN_SIGNAL_QUALITY_SCORE (adjustable after analysis)
• TAKE_PROFIT_1_SIZE_PERCENT (70% default, adjustable)
• USE_ATR_BASED_TARGETS (false, will enable after backtest)
• TP1_ATR_MULTIPLIER, TP2_ATR_MULTIPLIER, SL_ATR_MULTIPLIER

---

📈 Progress Tracking

Weekly Check-in Questions

1. How many trades executed this week?
2. How many blocked signals collected?
3. Any patterns emerging in blocked signals?
4. What's the current win rate vs target 60%?
5. Are MFE/MAE averages improving?

Monthly Review Questions

1. Do we have enough data for next optimization phase?
2. What's the biggest win/loss this month and why?
3. Is indicator v6 outperforming v5? (need 20+ trades each)
4. Should we adjust any thresholds based on patterns?
5. Are we on track for Phase 1 goal ($2,500)?

---

🚨 Risk Management

Don't Optimize Too Early

• ❌ Bad: Change thresholds after 5 trades and 2 blocked signals
• ✅ Good: Wait for 20 blocked signals + 50 trades minimum
• Statistical significance matters!

Keep Historical Baseline

• Always compare against "what would the old system do?"
• Track signalQualityVersion and indicatorVersion for this
• Can revert if changes make things worse

A/B Test Before Full Deploy

• Test new thresholds on 50% of signals (coin flip)
• Compare results after 20-30 trades
• Only deploy if statistically better (p < 0.05)

---

📚 Related Documentation

Core System:

• copilot-instructions.md - Full system architecture
• TRADING_GOALS.md - 8-phase financial roadmap ($106 → $1M+)
• ATR_TRAILING_STOP_FIX.md - Dynamic trailing stop implementation

Data Analysis:

• BLOCKED_SIGNALS_TRACKING.md - SQL queries for signal analysis
• docs/analysis/SIGNAL_QUALITY_VERSION_ANALYSIS.sql - Version comparison queries

Implementation Guides:

• SIGNAL_QUALITY_SETUP_GUIDE.md - How signal scoring works
• PERCENTAGE_SIZING_FEATURE.md - Position sizing system

---

🎯 Next Actions

Immediate (This Week)

1. ✅ Deploy indicator v6 to TradingView production
2. 🔄 Execute 10-15 trades to start data collection
3. 📊 Monitor blocked signals (target: 2-3 this week)
4. 🎯 Verify current trade closes correctly with new fixes

Short Term (2-3 Weeks)

1. Collect 10-20 blocked signals
2. Run signal quality analysis
3. Adjust MIN_SIGNAL_QUALITY_SCORE if data shows improvement
4. Continue collecting ATR data (target: 30-40 trades)

Medium Term (4-8 Weeks)

1. Run position scaling backtest (50+ trades)
2. Run ATR-based TP backtest (50+ trades)
3. Implement quality-based position sizing
4. A/B test ATR-based targets vs fixed %
5. Deploy winning strategies

Long Term (Phase 1 Complete)

1. Document all optimizations in copilot-instructions.md
2. Prepare for Phase 2 ($2,500 → $10,000)
3. Consider new optimizations:
   • Time-of-day filtering
   • Symbol-specific thresholds
   • Volatility regime detection
   • Machine learning for signal scoring

---

🚀 v9 Development Ideas (Nov 22, 2025 - Data Analysis)

Based on 8 v8 trades + 36 v5/v6 archived trades pattern analysis

1. Directional Filter (HIGHEST PRIORITY)

Pattern Discovered:

• v8 LONGS: 100% WR (3/3), +$565.03, Quality 98.3 avg, 174% avg MFE
• v8 SHORTS: 40% WR (2/5), -$283.54, Quality 91.0 avg, 23% avg MFE
• v5/v6: Shorts consistently outperform longs (50-60% WR vs 20-40%)

Hypothesis: Longs perform better across all indicator versions (44 total trades)

v9 Options:

• Conservative: Configurable DIRECTIONAL_BIAS setting (long_only, short_only, both)
• Aggressive: Smart direction filter - only trade direction with 7-day rolling WR ≥60%
• Expected Impact: Eliminate 60% of losses if pattern holds

Data Needed: 20 more v8 trades to validate (target: 28 total trades)

Decision Point: After trade #28, analyze long/short performance split

---

2. Time-of-Day Filter (MODERATE PRIORITY)

Pattern Discovered:

• 00-06 UTC (Asia): 66.7% WR, +$241.43 (3 trades)
• 18-24 UTC (After): 100% WR, +$257.56 (1 trade)
• 06-12 UTC (EU): 0% WR, -$138.35 (1 trade)
• 12-18 UTC (US): 66.7% WR, -$79.15 (3 trades)

Hypothesis: Asia/After-hours sessions outperform EU/US overlap

v9 Options:

• Configurable preferred trading hours (e.g., TRADING_HOURS=0-6,18-24)
• Block signals during low-performing sessions
• Expected Impact: ~15-20% improvement if pattern holds

Data Needed: 50+ trades to validate (sample size currently too small)

Decision Point: After 50 v8 trades, re-analyze time-of-day patterns

---

3. Quality-Based Emergency SL (SAFETY IMPROVEMENT)

Pattern Discovered:

• Trade cmi92gky: Quality 90, -$386.62 loss (-411% MAE)
• Only emergency exit in 8 trades, but 137% of total losses
• Emergency at -2% may be too generous for borderline quality signals

Hypothesis: Low-quality signals (90-94) need tighter emergency stops

v9 Options:

• Quality ≥95: Emergency at -2.0% (strong signal, give room)
• Quality 91-94: Emergency at -1.5% (moderate signal, tighter stop)
• Quality <91: BLOCKED (already implemented)
• Expected Impact: Cut worst-case losses by 25%

Data Needed: 10+ more trades in 91-94 quality range to validate

Decision Point: After 5+ emergency exits, analyze quality vs loss magnitude

---

4. Perfect Quality Threshold (ULTIMATE FILTER)

Pattern Discovered:

• Quality ≥95: 5 trades, 100% WR, +$906.39 (+$181/trade avg)
• Quality ≤90: 3 trades, 0% WR, -$624.90 (-$208/trade avg)
• Perfect separation at 91 threshold validated

Hypothesis: Raising to 95 after data collection = 100% WR

v9 Options:

• Keep threshold 91 until trade #28 (data collection)
• Raise to 95 after 20 more trades if 95+ pattern holds
• Zero tolerance for borderline signals
• Expected Impact: Potential 100% WR if pattern continues

Data Needed: 20 more trades (12 with quality ≥95 expected)

Decision Point: After trade #28, compare 91-94 vs 95+ performance

---

5. MFE/MAE-Based Position Sizing (ADVANCED)

Pattern Discovered:

• Winners: 137% avg MFE, -27% avg MAE (5:1 upside/downside)
• Losers: 10% avg MFE, -176% avg MAE (1:18 ratio)
• Winners move in our favor quickly, losers reverse hard

Hypothesis: Scale-in on early confirmation, fast-exit on early reversal

v9 Options:

• Open 50% position initially
• If profit ≥+0.5% within 5 minutes: Scale-in 50% more
• If loss ≥-0.3% within 5 minutes: Close entire position (fast exit)
• Expected Impact: Reduce loss exposure 50%, increase winner exposure 50%

Data Needed: 50+ trades with minute-by-minute price tracking

Decision Point: Phase 2 or Phase 3 (requires infrastructure changes)

---

Development Timeline

Current Phase (Trades 9-28): Continue data collection, threshold 91

• Validate directional bias pattern
• Collect time-of-day data
• Monitor quality ≥95 performance
• Track emergency exits by quality tier

After Trade #28: Analyze patterns, decide v9 features

• If long bias validates → Implement directional filter (v9a)
• If time patterns validate → Add session filter (v9b)
• If quality 95+ = 100% → Raise threshold (v9c)

After Trade #50: Advanced features

• MFE/MAE-based scaling
• Machine learning quality scoring
• Adaptive emergency SL

Priority Order (Impact × Ease):

1. Directional Filter - Highest impact if validated
2. Emergency SL by Quality - High safety, easy implementation
3. Raise Threshold to 95 - Zero effort, high impact if pattern holds
4. Time-of-Day Filter - Moderate impact, needs more data
5. MFE/MAE Scaling - Advanced, requires infrastructure

---

Bottom Line: Three complementary optimizations, all data-driven, all on track. Focus on collecting clean data now, analyze when we have enough, implement what works. No premature optimization. 📊🚀

v9 Strategy: Conservative approach - collect 20 more trades (target: 28 total), then make data-driven decisions about directional bias, quality thresholds, and safety improvements. Pattern recognition is powerful, but statistical significance requires larger sample sizes.