docs: Major documentation reorganization + ENV variable reference

**Documentation Structure:** - Created docs/ subdirectory organization (analysis/, architecture/, bugs/, cluster/, deployments/, roadmaps/, setup/, archived/) - Moved 68 root markdown files to appropriate categories - Root directory now clean (only README.md remains) - Total: 83 markdown files now organized by purpose **New Content:** - Added comprehensive Environment Variable Reference to copilot-instructions.md - 100+ ENV variables documented with types, defaults, purpose, notes - Organized by category: Required (Drift/RPC/Pyth), Trading Config (quality/ leverage/sizing), ATR System, Runner System, Risk Limits, Notifications, etc. - Includes usage examples (correct vs wrong patterns) **File Distribution:** - docs/analysis/ - Performance analyses, blocked signals, profit projections - docs/architecture/ - Adaptive leverage, ATR trailing, indicator tracking - docs/bugs/ - CRITICAL_*.md, FIXES_*.md bug reports (7 files) - docs/cluster/ - EPYC setup, distributed computing docs (3 files) - docs/deployments/ - *_COMPLETE.md, DEPLOYMENT_*.md status (12 files) - docs/roadmaps/ - All *ROADMAP*.md strategic planning files (7 files) - docs/setup/ - TradingView guides, signal quality, n8n setup (8 files) - docs/archived/2025_pre_nov/ - Obsolete verification checklist (1 file) **Key Improvements:** - ENV variable reference: Single source of truth for all configuration - Common Pitfalls #68-71: Already complete, verified during audit - Better findability: Category-based navigation vs 68 files in root - Preserves history: All files git mv (rename), not copy/delete - Zero broken functionality: Only documentation moved, no code changes **Verification:** - 83 markdown files now in docs/ subdirectories - Root directory cleaned: 68 files → 0 files (except README.md) - Git history preserved for all moved files - Container running: trading-bot-v4 (no restart needed) **Next Steps:** - Create README.md files in each docs subdirectory - Add navigation index - Update main README.md with new structure - Consolidate duplicate deployment docs - Archive truly obsolete files (old SQL backups) See: docs/analysis/CLEANUP_PLAN.md for complete reorganization strategy
2025-12-04 08:29:59 +01:00
parent e48332e347
commit 4c36fa2bc3
61 changed files with 520 additions and 37 deletions
--- a/docs/roadmaps/POSITION_SCALING_ROADMAP.md
+++ b/docs/roadmaps/POSITION_SCALING_ROADMAP.md
@@ -0,0 +1,389 @@
+# Position Scaling & Exit Optimization Roadmap
+
+## Current State (October 31, 2025)
+- **Total Trades:** 26 completed
+- **P&L:** +$27.12 (38% win rate)
+- **Shorts:** 6.6x more profitable than longs (+$2.46 vs -$0.37 avg)
+- **Current Strategy:** 
+  - TP1 at +1.5%: Close 75%
+  - TP2 at +3.0%: Close 80% of remaining (20% total)
+  - **Runner: 5% of position with 0.3% trailing stop ✅ ALREADY IMPLEMENTED**
+- **Problem:** Small runner size (5%) + tight trailing stop (0.3%) may be suboptimal
+- **Data Quality:** Ready to collect signal quality scores for correlation analysis
+
+---
+
+## Phase 1: Data Collection (CURRENT PHASE) 🔄
+**Goal:** Gather 20-50 trades with quality scores before making strategy changes
+
+### Data Points to Track:
+- [ ] Signal quality score (0-100) for each trade
+- [ ] Max Favorable Excursion (MFE) vs exit price differential
+- [ ] ATR at entry vs actual price movement distance
+- [ ] Time duration for winning trades (identify "quick wins" vs "runners")
+- [ ] Quality score correlation with:
+  - [ ] Win rate
+  - [ ] Average P&L
+  - [ ] MFE (runner potential)
+  - [ ] Trade duration
+
+### Analysis Queries (Run after 20+ scored trades):
+```sql
+-- Quality score vs performance
+SELECT 
+  CASE 
+    WHEN "signalQualityScore" >= 80 THEN 'High (80-100)'
+    WHEN "signalQualityScore" >= 70 THEN 'Medium (70-79)'
+    ELSE 'Low (60-69)'
+  END as quality_tier,
+  COUNT(*) as trades,
+  ROUND(AVG("realizedPnL")::numeric, 2) as avg_pnl,
+  ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe,
+  ROUND(100.0 * SUM(CASE WHEN "realizedPnL" > 0 THEN 1 ELSE 0 END) / COUNT(*)::numeric, 1) as win_rate
+FROM "Trade"
+WHERE "signalQualityScore" IS NOT NULL AND "exitReason" IS NOT NULL
+GROUP BY quality_tier
+ORDER BY quality_tier;
+
+-- ATR correlation with movement
+SELECT 
+  direction,
+  ROUND(AVG(atr)::numeric, 2) as avg_atr,
+  ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe,
+  ROUND(AVG(ABS("exitPrice" - "entryPrice") / "entryPrice" * 100)::numeric, 2) as avg_move_pct
+FROM "Trade"
+WHERE atr IS NOT NULL AND "exitReason" IS NOT NULL
+GROUP BY direction;
+
+-- Runner potential analysis (how many went beyond TP2?)
+SELECT 
+  "exitReason",
+  COUNT(*) as count,
+  ROUND(AVG("maxFavorableExcursion")::numeric, 2) as avg_mfe,
+  ROUND(AVG("realizedPnL")::numeric, 2) as avg_pnl,
+  -- MFE > 3% indicates runner potential
+  SUM(CASE WHEN "maxFavorableExcursion" > 3.0 THEN 1 ELSE 0 END) as runner_potential_count
+FROM "Trade"
+WHERE "exitReason" IS NOT NULL
+GROUP BY "exitReason"
+ORDER BY count DESC;
+```
+
+---
+
+## Phase 2: ATR-Based Dynamic Targets ⏳
+**Prerequisites:** ✅ 20+ trades with ATR data collected
+
+### Implementation Tasks:
+- [ ] **Add ATR normalization function** (`lib/trading/scaling-strategy.ts`)
+  - [ ] Calculate normalized ATR factor: `(current_ATR / baseline_ATR)`
+  - [ ] Baseline ATR = 2.0 for SOL-PERP (adjust based on data)
+  
+- [ ] **Update TP calculation in `lib/drift/orders.ts`**
+  - [ ] TP1: `entry + (1.5% × ATR_factor)` instead of fixed 1.5%
+  - [ ] TP2: `entry + (3.0% × ATR_factor)` instead of fixed 3.0%
+  
+- [ ] **Modify Position Manager monitoring loop**
+  - [ ] Store `atrFactor` in `ActiveTrade` interface
+  - [ ] Adjust dynamic SL movements by ATR factor
+  - [ ] Update breakeven trigger: `+0.5% × ATR_factor`
+  - [ ] Update profit lock trigger: `+1.2% × ATR_factor`
+
+- [ ] **Testing:**
+  - [ ] Backtest on historical trades with ATR data
+  - [ ] Calculate improvement: old fixed % vs new ATR-adjusted
+  - [ ] Run 10 test trades before production
+
+**Expected Outcome:** Wider targets in high volatility, tighter in low volatility
+
+---
+
+## Phase 3: Signal Quality-Based Scaling ⏳
+**Prerequisites:** ✅ Phase 2 complete, ✅ 30+ trades with quality scores, ✅ Clear correlation proven
+
+### Implementation Tasks:
+- [ ] **Create quality tier configuration** (`config/trading.ts`)
+  ```typescript
+  export interface QualityTierConfig {
+    minScore: number
+    maxScore: number
+    tp1Percentage: number  // How much to take off
+    tp2Percentage: number
+    runnerPercentage: number
+    atrMultiplierTP1: number
+    atrMultiplierTP2: number
+    trailingStopATR: number
+  }
+  ```
+
+- [ ] **Define three tiers based on data analysis:**
+  - [ ] **High Quality (80-100):** Aggressive runner strategy
+    - TP1: 50% off at 2.0×ATR
+    - TP2: 25% off at 4.0×ATR
+    - Runner: 25% with 2.5×ATR trailing stop
+  
+  - [ ] **Medium Quality (70-79):** Balanced (current-ish)
+    - TP1: 75% off at 1.5×ATR
+    - TP2: 25% off at 3.0×ATR
+    - Runner: None (full exit at TP2)
+  
+  - [ ] **Low Quality (60-69):** Conservative quick exit
+    - TP1: 100% off at 1.0×ATR
+    - TP2: None
+    - Runner: None
+
+- [ ] **Update `placeExitOrders()` function**
+  - [ ] Accept `qualityScore` parameter
+  - [ ] Select tier config based on score
+  - [ ] Place orders according to tier rules
+  - [ ] Only place TP2 order if tier has runner
+
+- [ ] **Update Position Manager**
+  - [ ] Store `qualityScore` in `ActiveTrade`
+  - [ ] Apply tier-specific trailing stop logic
+  - [ ] Handle partial closes (50%, 75%, or 100%)
+
+- [ ] **Database tracking:**
+  - [ ] Add `scalingTier` field to Trade model (high/medium/low)
+  - [ ] Track which tier was used for each trade
+
+**Expected Outcome:** High quality signals let winners run, low quality signals take quick profits
+
+---
+
+## Phase 4: Direction-Based Optimization ⏳
+**Prerequisites:** ✅ Phase 3 complete, ✅ Directional edge confirmed in 50+ trades
+
+### Implementation Tasks:
+- [ ] **Analyze directional performance** (Re-run after 50 trades)
+  - [ ] Compare long vs short win rates
+  - [ ] Compare long vs short avg P&L
+  - [ ] Compare long vs short MFE (runner potential)
+  - [ ] **Decision:** If shorts still 3x+ better, implement direction bias
+
+- [ ] **Direction-specific configs** (`config/trading.ts`)
+  ```typescript
+  export interface DirectionConfig {
+    shortTP1Pct: number  // If shorts have edge, wider targets
+    shortTP2Pct: number
+    shortRunnerPct: number
+    longTP1Pct: number   // If longs struggle, tighter defensive
+    longTP2Pct: number
+    longRunnerPct: number
+  }
+  ```
+
+- [ ] **Implementation in order placement:**
+  - [ ] Check `direction` field
+  - [ ] Apply direction-specific multipliers on top of quality tier
+  - [ ] Example: Short with high quality = 2.0×ATR × 1.2 (direction bonus)
+
+- [ ] **A/B Testing:**
+  - [ ] Run 20 trades with direction bias
+  - [ ] Run 20 trades without (control group)
+  - [ ] Compare results before full rollout
+
+**Expected Outcome:** Shorts get wider targets if edge persists, longs stay defensive
+
+---
+
+## Phase 5: Optimize Runner Size & Trailing Stop ⏳
+**Prerequisites:** ✅ Phase 3 complete, ✅ Runner data collected (10+ trades with runners)
+
+**Current Implementation:** ✅ Runner with trailing stop already exists!
+- Runner size: 5% (configurable via `TAKE_PROFIT_2_SIZE_PERCENT=80`)
+- Trailing stop: 0.3% fixed (configurable via `TRAILING_STOP_PERCENT=0.3`)
+
+### Implementation Tasks:
+- [ ] **Analyze runner performance from existing trades**
+  - [ ] Query trades where runner was active (TP2 hit)
+  - [ ] Calculate: How many runners hit trailing stop vs kept going?
+  - [ ] Calculate: Average runner profit vs optimal exit
+  - [ ] Calculate: Was 0.3% trailing stop too tight? (got stopped out too early?)
+
+- [ ] **Optimize runner size by quality tier:**
+  - [ ] High quality (80-100): 25% runner (TP2 closes 0%, all becomes runner)
+  - [ ] Medium quality (70-79): 10% runner (TP2 closes 60% of remaining)
+  - [ ] Low quality (60-69): 5% runner (current behavior)
+
+- [ ] **Make trailing stop ATR-based:**
+  - [ ] Change from fixed 0.3% to `(1.5 × ATR)` or `(2.0 × ATR)`
+  - [ ] Add `trailingStopATRMultiplier` config option
+  - [ ] Update Position Manager to use ATR-based trailing distance
+  - [ ] Store ATR value in ActiveTrade for dynamic calculations
+
+- [ ] **Add runner-specific analytics:**
+  - [ ] Dashboard widget: Runner performance stats
+  - [ ] Show: Total profit from runners vs TP1/TP2
+  - [ ] Show: Average runner hold time
+  - [ ] Show: Runner win rate
+
+- [ ] **Testing:**
+  - [ ] Backtest: Simulate larger runners (10-25%) on historical TP2 trades
+  - [ ] Backtest: Simulate ATR-based trailing stop vs fixed 0.3%
+  - [ ] A/B test: Run 10 trades with optimized settings before full rollout
+
+**Expected Outcome:** Capture more profit from extended moves, reduce premature trailing stop exits
+
+---
+
+## Phase 6: Advanced ML-Based Exit Prediction (Future) 🔮
+**Prerequisites:** ✅ 100+ trades with all metrics, ✅ Phases 1-5 complete
+
+### Research Tasks:
+- [ ] **Feature engineering:**
+  - [ ] Input features: ATR, ADX, RSI, volumeRatio, pricePosition, quality score, direction, timeframe
+  - [ ] Target variable: Did trade reach 2×TP2? (binary classification)
+  - [ ] Additional target: Max profit % reached (regression)
+
+- [ ] **Model training:**
+  - [ ] Split data: 70% train, 30% test
+  - [ ] Try models: Logistic Regression, Random Forest, XGBoost
+  - [ ] Evaluate: Precision, Recall, F1 for runner prediction
+  - [ ] Cross-validation with time-based splits (avoid lookahead bias)
+
+- [ ] **Integration:**
+  - [ ] `/api/trading/predict-runner` endpoint
+  - [ ] Call during trade execution to get runner probability
+  - [ ] Adjust runner size based on probability: 0-15% runner if low, 25-35% if high
+
+- [ ] **Monitoring:**
+  - [ ] Track model accuracy over time
+  - [ ] Retrain monthly with new data
+  - [ ] A/B test: ML-based vs rule-based scaling
+
+**Expected Outcome:** AI predicts which trades have runner potential before entry
+
+---
+
+## Quick Wins (Can Do Anytime) ⚡
+
+- [ ] **Manual runner management for current trade**
+  - [ ] Move SL to +30% profit lock on current +41% SOL position
+  - [ ] Monitor manually until trend breaks
+  - [ ] Document outcome: How much did runner capture?
+
+- [ ] **Add "runner stats" to analytics dashboard**
+  - [ ] Show: How many trades went beyond TP2?
+  - [ ] Show: Average MFE for TP2 exits
+  - [ ] Show: Estimated missed profit from not having runners
+
+- [ ] **Database views for common queries**
+  - [ ] Create `vw_quality_performance` view
+  - [ ] Create `vw_runner_potential` view
+  - [ ] Create `vw_directional_edge` view
+
+- [ ] **Alerts for exceptional trades**
+  - [ ] Telegram notification when MFE > 5% (runner candidate)
+  - [ ] Telegram notification when quality score > 90 (premium setup)
+
+---
+
+## Decision Gates 🚦
+
+**Before Phase 2 (ATR-based):**
+- ✅ Have 20+ trades with ATR data
+- ✅ ATR values look reasonable (0.5 - 3.5 range)
+- ✅ Clear volatility variation observed
+
+**Before Phase 3 (Quality tiers):**
+- ✅ Have 30+ trades with quality scores
+- ✅ Statistical significance: High quality scores show measurably better outcomes
+- ✅ Correlation coefficient > 0.3 between quality and P&L
+
+**Before Phase 4 (Direction bias):**
+- ✅ Have 50+ trades (25+ each direction)
+- ✅ Directional edge persists (3x+ performance gap)
+- ✅ Edge is consistent across different market conditions
+
+**Before Phase 5 (Runners):**
+- ✅ 30%+ of trades show MFE > TP2
+- ✅ Average MFE significantly higher than TP2 level
+- ✅ Phases 2-3 stable and profitable
+
+**Before Phase 6 (ML):**
+- ✅ 100+ trades with complete feature data
+- ✅ Proven improvement from Phases 1-5
+- ✅ Computational resources available (training time)
+
+---
+
+## Notes & Observations
+
+### Current Trade Example (Oct 31, 2025):
+- Entry: $182.73
+- Current: $186.56 (+2.1%, +$11.30)
+- **Actual P&L: +41% unrealized** 🚀
+- **Status:** TP1 hit (closed 75%), TP2 hit (closed 20%), 5% runner still active with trailing stop
+- **Lesson:** The 5% runner captured this move! But could a larger runner (10-25%) capture even more?
+- **Trailing stop:** 0.3% below peak might be too tight, ATR-based (1.5-2.0×ATR) might work better
+
+### Key Metrics to Watch:
+- Win rate by quality tier
+- Average MFE vs exit price gap
+- Correlation between ATR and price movement
+- Shorts vs longs performance delta
+- Percentage of trades that go beyond TP2
+
+### Strategy Validation:
+Run this after each phase to validate improvement:
+```sql
+-- Compare old vs new strategy performance
+SELECT 
+  'Phase X' as phase,
+  COUNT(*) as trades,
+  ROUND(AVG("realizedPnL")::numeric, 2) as avg_pnl,
+  ROUND(SUM("realizedPnL")::numeric, 2) as total_pnl,
+  ROUND(100.0 * SUM(CASE WHEN "realizedPnL" > 0 THEN 1 ELSE 0 END) / COUNT(*)::numeric, 1) as win_rate
+FROM "Trade"
+WHERE "createdAt" >= '[phase_start_date]'
+  AND "exitReason" IS NOT NULL;
+```
+
+---
+
+## Timeline Estimate
+
+- **Phase 1 (Data Collection):** 2-4 weeks (depends on signal frequency)
+- **Phase 2 (ATR-based):** 3-5 days implementation + 1 week testing
+- **Phase 3 (Quality tiers):** 5-7 days implementation + 2 weeks testing
+- **Phase 4 (Direction bias):** 2-3 days implementation + 1 week testing
+- **Phase 5 (Runners):** 7-10 days implementation + 2 weeks testing
+- **Phase 6 (ML):** 2-3 weeks research + implementation
+
+**Total estimated time:** 3-4 months from start to Phase 5 complete
+
+---
+
+## Success Criteria
+
+**Phase 2 Success:**
+- [ ] Average P&L increases by 10%+ vs fixed targets
+- [ ] Win rate stays stable or improves
+- [ ] No increase in max drawdown
+
+**Phase 3 Success:**
+- [ ] High quality trades show 20%+ better P&L than low quality
+- [ ] Overall P&L increases by 15%+ vs Phase 2
+- [ ] Quality filtering prevents some losing trades
+
+**Phase 4 Success:**
+- [ ] Directional P&L gap narrows (improve weak direction)
+- [ ] Or: Strong direction P&L improves further (if edge is real)
+
+**Phase 5 Success:**
+- [ ] Runners capture 20%+ more profit on winning trades
+- [ ] Total P&L increases by 25%+ vs Phase 4
+- [ ] Runners don't create new losing trades
+
+**Overall Success (All Phases):**
+- [ ] 2x total P&L vs baseline strategy
+- [ ] 50%+ win rate (up from 38%)
+- [ ] Average winner > 2× average loser
+- [ ] Profit factor > 2.0
+
+---
+
+**Status:** Phase 1 (Data Collection) - Active 🔄
+**Last Updated:** October 31, 2025
+**Next Review:** After 20 trades with quality scores collected