diff --git a/cluster/exploration.db b/cluster/exploration.db index f923611..7f643ff 100644 Binary files a/cluster/exploration.db and b/cluster/exploration.db differ diff --git a/docs/1MIN_SIMPLIFIED_FEED.md b/docs/1MIN_SIMPLIFIED_FEED.md new file mode 100644 index 0000000..7b33bb9 --- /dev/null +++ b/docs/1MIN_SIMPLIFIED_FEED.md @@ -0,0 +1,157 @@ +# 1-Minute Simplified Price Feed (Dec 4, 2025) + +## Problem Statement + +**Issue:** TradingView alert queue getting overwhelmed by 1-minute signal flood +- **Volume:** 60 alerts/hour (one per minute) from 1-minute data feed +- **Impact:** 5-minute trading signals occasionally skipped (happened 2× so far) +- **Root cause:** 1-minute alerts sending full trading signal format with all indicators + +## Solution: Lightweight Price-Only Feed + +**Change:** Reduce 1-minute alert payload by ~70% + +### Before (Full Format) +``` +SOLUSDT buy 1 @ 142.08 | ATR:0.65 | ADX:14.3 | RSI:51.3 | VOL:0.87 | POS:59.3 | IND:v9 +``` +**Size:** ~80 characters +**Processing:** n8n parses 7 fields, bot calculates quality score, saves to BlockedSignal + MarketData + +### After (Minimal Format) +``` +SOLUSDT buy 1 @ 142.08 +``` +**Size:** ~22 characters (~73% reduction) +**Processing:** n8n parses 3 fields (symbol/timeframe/price), bot routes to BlockedSignal only + +## Why This Works + +### What We Actually Need from 1-Minute Data +1. ✅ **Price updates** - For market data cache (5-minute expiry) +2. ✅ **Symbol identification** - Routing to correct market +3. ✅ **Timeframe marker** - `timeframe="1"` triggers data collection path +4. ❌ **ATR/ADX/RSI/VOL/POS** - NOT used in real-time (recalculated from DB if needed) +5. ❌ **Direction** - Meaningless for data-only signals (see copilot-instructions.md) + +### What We Already Have +- **5-minute signals** include full indicator context for trading decisions +- **BlockedSignalTracker** recalculates TP1/TP2/SL from database data (not live indicators) +- **Market data cache** only needs `currentPrice` + `timestamp` for validation (5-minute expiry) +- **MarketData table** can store NULL for indicators (not critical for analysis) + +### What Gets Removed +```typescript +// OLD: Execute endpoint saves full metrics to MarketData table +await prisma.marketData.create({ + data: { + atr: Number(body.atr) || 0, // ❌ Remove + adx: Number(body.adx) || 0, // ❌ Remove + rsi: Number(body.rsi) || 50, // ❌ Remove + volumeRatio: Number(body.volumeRatio) || 1.0, // ❌ Remove + pricePosition: Number(body.pricePosition) || 50, // ❌ Remove + maGap: Number(body.maGap) || undefined, // ❌ Remove + + // Keep essentials + symbol: driftSymbol, // ✅ Keep + timeframe: '1', // ✅ Keep + price: currentPrice, // ✅ Keep + timestamp: new Date() // ✅ Keep (auto-generated) + } +}) +``` + +## Implementation Plan + +### Phase 1: Update TradingView Alert (IMMEDIATE) +1. Replace `moneyline_1min_data_feed.pinescript` with `moneyline_1min_price_feed.pinescript` +2. Update TradingView alert to use new script +3. Test: Verify alerts still route to BlockedSignal correctly + +### Phase 2: Update Execute Endpoint (OPTIONAL) +Since we're only sending price, make execute endpoint resilient to missing fields: +```typescript +// Lines 193-207: MarketData.create() +// Change required fields to optional with safe defaults +atr: Number(body.atr) || null, // NULL if missing +adx: Number(body.adx) || null, +rsi: Number(body.rsi) || null, +// ... etc +``` + +### Phase 3: Monitor Results +- Track 5-minute signal reliability (should go from 98% → 100%) +- Verify 1-minute data still saves to BlockedSignal +- Check TradingView alert queue no longer shows congestion + +## Expected Impact + +### Benefits +1. **70% reduction** in TradingView alert payload size +2. **Priority preservation** - 5-minute trading signals less likely to be dropped +3. **Faster processing** - Less n8n parsing, less bot processing +4. **Same functionality** - No loss of critical features + +### Tradeoffs +1. MarketData table will have NULL indicators (acceptable - not used for decisions) +2. Historical 1-minute indicator data less complete (acceptable - we have 5-minute full data) + +### No Impact On +- ✅ 5-minute trading signals (still full format with all indicators) +- ✅ BlockedSignalTracker (calculates from price movements, not live indicators) +- ✅ Market data cache (only needs price + timestamp) +- ✅ Smart Entry Validation (uses 5-minute cached data, not 1-minute) + +## Files Changed + +### TradingView Pine Script +- **NEW:** `workflows/trading/moneyline_1min_price_feed.pinescript` - Simplified version +- **OLD:** `workflows/trading/moneyline_1min_data_feed.pinescript` - Full format (archived) + +### n8n Workflow +- **UNCHANGED:** `workflows/trading/parse_signal_enhanced.json` - Already handles minimal format + - Extracts: `symbol`, `timeframe`, `signalPrice` (from `@ price` pattern) + - Defaults: ATR/ADX/RSI/etc to 0 if missing + +### Bot Code +- **UNCHANGED:** `app/api/trading/execute/route.ts` - Already handles missing fields with fallbacks + - Lines 193-207: Uses `Number(body.field) || default` pattern + +## Migration Steps + +1. ✅ **Read this document** - Understand rationale +2. ⏳ **Update TradingView alert** - Replace with `moneyline_1min_price_feed.pinescript` +3. ⏳ **Test with 1-2 alerts** - Verify BlockedSignal still populates +4. ⏳ **Monitor for 24 hours** - Check 5-minute signal reliability +5. ⏳ **Archive old script** - Move `moneyline_1min_data_feed.pinescript` to archive/ + +## Rollback Plan + +If issues arise: +1. Switch TradingView alert back to `moneyline_1min_data_feed.pinescript` +2. No code changes needed - bot handles both formats + +## Decision Criteria + +**Use simplified feed IF:** +- 5-minute signal drops are occurring regularly (2+ per week) +- TradingView alert queue shows congestion +- 1-minute indicator data not being used for analysis + +**Keep full feed IF:** +- 5-minute signals are 100% reliable +- Future analysis requires minute-by-minute indicator tracking +- No performance issues observed + +## Current Status (Dec 4, 2025) + +- ✅ Simplified script created +- ⏳ TradingView alert update pending +- ⏳ Monitoring results pending +- **User decision:** Test simplified feed for 24-48 hours + +## References + +- Copilot Instructions: Section "1-Minute Data Collection System" +- Original Implementation: `docs/1MIN_DATA_COLLECTION_SIMPLE.md` +- BlockedSignalTracker: `lib/analysis/blocked-signal-tracker.ts` diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..361ee5d --- /dev/null +++ b/docs/README.md @@ -0,0 +1,161 @@ +# Trading Bot v4 Documentation + +**Comprehensive documentation for the Trading Bot v4 autonomous trading system.** + +Last Updated: December 4, 2025 + +--- + +## 📂 Documentation Structure + +### **[analysis/](analysis/)** - Performance Analysis & Optimization Studies +Performance analyses, blocked signals tracking, profit projections, and diagnostic results. Data-driven insights for system optimization. + +### **[architecture/](architecture/)** - System Architecture & Design +Adaptive leverage system, ATR trailing stops, indicator version tracking, and position synchronization. Core technical design documentation. + +### **[bugs/](bugs/)** - Bug Reports & Critical Fixes +CRITICAL_*.md and FIXES_*.md bug reports. Historical record of critical incidents and their resolutions. + +### **[cluster/](cluster/)** - Distributed Computing & EPYC Cluster +EPYC cluster setup, distributed backtesting coordination, dual sweep configurations. Multi-server infrastructure documentation. + +### **[deployments/](deployments/)** - Deployment Status & Verification +*_COMPLETE.md and DEPLOYMENT_*.md status files. Verification reports for major feature deployments. + +### **[roadmaps/](roadmaps/)** - Strategic Planning & Roadmaps +All *ROADMAP*.md files. Long-term strategic planning for system enhancements across multiple dimensions. + +### **[setup/](setup/)** - Setup Guides & Configuration +TradingView alert setup, n8n workflows, signal quality configuration, percentage sizing features. + +### **[archived/](archived/)** - Historical Documentation +Obsolete verification checklists and documentation from previous development phases. + +--- + +## 🚀 Quick Start + +**New Developers:** +1. Read [TRADING_GOALS.md](TRADING_GOALS.md) - Understand financial objectives +2. Review [roadmaps/OPTIMIZATION_MASTER_ROADMAP.md](roadmaps/OPTIMIZATION_MASTER_ROADMAP.md) - Strategic vision +3. Check [setup/QUICK_SETUP_CARD.md](setup/QUICK_SETUP_CARD.md) - Quick configuration reference +4. Study [architecture/ADAPTIVE_LEVERAGE_SYSTEM.md](architecture/ADAPTIVE_LEVERAGE_SYSTEM.md) - Core risk management + +**Debugging Issues:** +1. Search [bugs/](bugs/) directory for similar incidents +2. Review Common Pitfalls in `.github/copilot-instructions.md` +3. Check [deployments/](deployments/) for recent changes +4. Consult [analysis/DIAGNOSTIC_RESULTS_SUMMARY.md](analysis/DIAGNOSTIC_RESULTS_SUMMARY.md) + +**Feature Development:** +1. Check [roadmaps/](roadmaps/) for planned enhancements +2. Review [architecture/](architecture/) for existing patterns +3. Read deployment verification docs in [deployments/](deployments/) +4. Follow verification procedures before marking features "complete" + +--- + +## 📊 Current System Status + +**Production Configuration (Dec 4, 2025):** +- **Capital:** $540 USDC (100% health, zero debt) +- **Quality Thresholds:** LONG ≥90, SHORT ≥95 +- **Adaptive Leverage:** 10x for high-quality (≥95), 5x for borderline (90-94) +- **Indicator:** v9 Money Line with MA Gap + Momentum SHORT Filter +- **Position Sizing:** SOL-PERP at 100% allocation, 5x leverage +- **Runner System:** TP1 closes 60%, TP2 activates trailing stop on 40% runner +- **Smart Entry:** Enabled (waits for 0.3% pullback confirmation) + +See [TRADING_GOALS.md](TRADING_GOALS.md) for complete financial roadmap ($106 → $100k+) + +--- + +## 🔍 Finding Documentation + +**By Topic:** +- **Performance Data:** [analysis/](analysis/) +- **System Design:** [architecture/](architecture/) +- **Bug History:** [bugs/](bugs/) +- **Infrastructure:** [cluster/](cluster/) +- **Feature Status:** [deployments/](deployments/) +- **Future Plans:** [roadmaps/](roadmaps/) +- **Setup Guides:** [setup/](setup/) + +**By Date:** +- Most recent deployments in [deployments/](deployments/) +- Historical bugs in [bugs/](bugs/) +- Archived docs in [archived/](archived/) + +**By File Type:** +- `*ROADMAP.md` → [roadmaps/](roadmaps/) +- `CRITICAL_*.md` → [bugs/](bugs/) +- `*_COMPLETE.md` → [deployments/](deployments/) +- `*_ANALYSIS.md` → [analysis/](analysis/) + +--- + +## 📝 Documentation Standards + +**File Naming Conventions:** +- `UPPERCASE_WITH_UNDERSCORES.md` - Important permanent docs +- `lowercase-with-dashes.md` - Temporary or draft docs +- `*_COMPLETE.md` - Verified deployment status +- `CRITICAL_*.md` - Bug reports requiring immediate attention + +**Content Requirements:** +- **Date stamps:** All docs must include creation/update dates +- **Status indicators:** Use emojis (✅ ❌ 🔄 ⏳ 🎯) for quick status scanning +- **Code examples:** Always include working code snippets for complex features +- **Verification steps:** Document how to verify feature is working +- **Lessons learned:** Include "Why This Matters" and "Lessons Learned" sections + +**Update Frequency:** +- **Real-time:** [deployments/](deployments/) - Update immediately after deployment +- **Weekly:** [roadmaps/](roadmaps/) - Review progress, update status +- **Monthly:** [analysis/](analysis/) - Performance review, optimization analysis +- **As-needed:** [bugs/](bugs/), [setup/](setup/), [architecture/](architecture/) + +--- + +## 🤝 Contributing + +**Before Committing:** +1. Update relevant documentation in appropriate subdirectory +2. Add entry to `.github/copilot-instructions.md` if system behavior changes +3. Create deployment verification doc in [deployments/](deployments/) +4. Update roadmap status in [roadmaps/](roadmaps/) if completing phase +5. Git commit with descriptive message (`docs:`, `feat:`, `fix:`, `critical:`) + +**Documentation Mandate:** +- **EVERY git commit requires documentation update** - Code without docs = incomplete work +- See `.github/copilot-instructions.md` "DOCUMENTATION + GIT COMMIT: INSEPARABLE WORKFLOW" +- Common Pitfalls, new ENV variables, configuration changes → Document immediately +- Bug fixes → Add to Common Pitfalls with full incident details + +--- + +## 🔗 External References + +- **Main Repository:** `/home/icke/traderv4/` +- **Configuration:** `.github/copilot-instructions.md` (6,400+ lines) +- **Environment Variables:** `.env` (482 lines) +- **Database Schema:** `prisma/schema.prisma` +- **Primary Code:** `app/api/trading/`, `lib/trading/`, `lib/drift/` + +--- + +## 📞 Support + +**For Issues:** +1. Check [bugs/](bugs/) for similar problems +2. Review Common Pitfalls in `.github/copilot-instructions.md` +3. Consult [deployments/](deployments/) for recent changes +4. Search git history: `git log --grep="keyword"` + +**For Features:** +1. Check [roadmaps/](roadmaps/) for planned work +2. Review [architecture/](architecture/) for existing patterns +3. Study similar implementations in git history + +**Remember:** This is a **real money trading system** - every change affects financial outcomes. Always verify thoroughly before declaring anything "working" or "fixed". diff --git a/docs/analysis/README.md b/docs/analysis/README.md new file mode 100644 index 0000000..f4c4229 --- /dev/null +++ b/docs/analysis/README.md @@ -0,0 +1,156 @@ +# Performance Analysis & Optimization Studies + +**Data-driven insights for trading system optimization.** + +This directory contains performance analyses, blocked signals tracking, profit projections, and diagnostic results. All files focus on measuring system performance and identifying optimization opportunities. + +--- + +## 📊 Key Documents + +### **Performance Reviews** +- `PROFIT_PROJECTION_NOV24_2025.md` - Monthly profit projections and compound growth analysis +- `QUALITY_90_TP1_ONLY_ANALYSIS.md` - Analysis of quality 90+ signals with TP1-only exits +- `SIGNAL_QUALITY_TEST_RESULTS.md` - Signal quality scoring validation results + +### **Blocked Signals Analysis** +- `BLOCKED_SIGNALS_TRACKING.md` - System for tracking signals blocked by quality filters +- `BLOCKED_SIGNAL_ANALYSIS_DEC2.md` - December 2 analysis of blocked signals +- `BLOCKED_SIGNAL_CORRECTED_ANALYSIS_DEC2.md` - Corrected analysis after data quality fixes + +### **Adaptive Leverage** +- `LONG_ADAPTIVE_LEVERAGE_VERIFICATION.md` - Verification of LONG direction adaptive leverage +- `SHORT_ADAPTIVE_LEVERAGE_IMPLEMENTED.md` - SHORT direction adaptive leverage implementation + +### **System Health** +- `DIAGNOSTIC_RESULTS_SUMMARY.md` - Comprehensive diagnostic test results +- `SAFETY_ANALYSIS.md` - Risk management and safety parameter analysis +- `RECOVERY_PLAN.md` - System recovery procedures after incidents +- `ANALYTICS_STATUS_AND_NEXT_STEPS.md` - Analytics dashboard status and roadmap + +### **Strategic Planning** +- `CLEANUP_PLAN.md` - Documentation reorganization strategy (precursor to this structure) + +--- + +## 📈 Analysis Workflow + +**1. Data Collection:** +- Trades executed with quality scores, ATR, ADX, RSI metrics +- Blocked signals saved to BlockedSignal table for comparison +- Position Manager tracks MAE (Max Adverse Excursion) and MFE (Max Favorable Excursion) + +**2. Analysis Queries:** +```sql +-- Win rate by quality tier +SELECT + CASE + WHEN signalQualityScore >= 95 THEN '95-100' + WHEN signalQualityScore >= 90 THEN '90-94' + WHEN signalQualityScore >= 85 THEN '85-89' + ELSE '60-84' + END as tier, + COUNT(*) as trades, + ROUND(100.0 * SUM(CASE WHEN realizedPnL > 0 THEN 1 ELSE 0 END) / COUNT(*), 1) as win_rate, + ROUND(SUM(realizedPnL), 2) as total_pnl +FROM "Trade" +WHERE exitReason IS NOT NULL +GROUP BY tier +ORDER BY MIN(signalQualityScore) DESC; +``` + +**3. Optimization:** +- Compare blocked vs executed signal performance +- Identify quality threshold sweet spots +- Validate parameter changes with 50-100 trade samples +- Document findings in this directory + +--- + +## 🎯 Current Focus (Dec 2025) + +**Active Analyses:** +- Quality threshold optimization (LONG 90, SHORT 95 under evaluation) +- Adaptive leverage performance tracking (10x vs 5x tiers) +- Blocked signal win rate comparison (would-be winners vs actual blocks) +- Runner system effectiveness (40% runner with ATR trailing stop) + +**Upcoming:** +- Multi-timeframe comparison (5min vs 15min vs 1H) +- MA crossover pattern validation (ADX weak→strong hypothesis) +- Smart Entry Validation Queue effectiveness tracking + +--- + +## 📝 Adding New Analyses + +**Template Structure:** +```markdown +# [Analysis Title] + +**Date:** [YYYY-MM-DD] +**Data Range:** [Date range or trade count] +**Objective:** [What you're analyzing] + +## Executive Summary +[2-3 sentence key findings] + +## Methodology +[How data was collected, SQL queries, filters applied] + +## Results +[Tables, charts, statistics] + +## Insights +[What the data means, patterns discovered] + +## Recommendations +[Specific actions, parameter changes, threshold adjustments] + +## Verification Required +[How to test recommendations, sample sizes needed] +``` + +**Naming Convention:** +- `[TOPIC]_ANALYSIS_[DATE].md` for dated analyses +- `[FEATURE]_VERIFICATION.md` for feature validation +- `[METRIC]_TRACKING.md` for ongoing measurement + +--- + +## 🔍 Finding Specific Analyses + +**By Topic:** +- Quality scores → `SIGNAL_QUALITY_*`, `QUALITY_90_*` +- Leverage → `*_ADAPTIVE_LEVERAGE_*` +- Blocked signals → `BLOCKED_SIGNAL_*` +- System health → `DIAGNOSTIC_*`, `SAFETY_*`, `RECOVERY_*` + +**By Date:** +- All files include creation dates in content +- Most recent: December 2025 blocked signal analyses +- Historical: CLEANUP_PLAN.md (documentation strategy) + +--- + +## ⚠️ Important Notes + +**Data Integrity:** +- Always filter out `signalSource='manual'` for indicator analysis +- Use `WHERE timeframe='5'` for production signal analysis +- Exclude `blockReason='DATA_COLLECTION_ONLY'` for execution analysis + +**Sample Sizes:** +- Minimum 50 trades for statistical validity +- Minimum 20 blocked signals for threshold analysis +- 100+ trades preferred for major parameter changes + +**Real Money System:** +- Every analysis affects financial outcomes +- Always include "Expected Impact" section (dollar amounts) +- Document both upside potential and downside risk +- Require user approval for threshold changes >10 points + +--- + +See `../README.md` for overall documentation structure. \ No newline at end of file diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 0000000..391a87f --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,219 @@ +# System Architecture & Design + +**Core technical design documentation for Trading Bot v4 autonomous trading system.** + +This directory contains architectural decisions, system design patterns, and technical specifications for major system components. + +--- + +## 🏗️ Key Architecture Documents + +### **Adaptive Leverage System** +- `ADAPTIVE_LEVERAGE_SYSTEM.md` - **Complete implementation guide** + - Quality-based leverage tiers (10x for ≥95, 5x for 90-94) + - Direction-specific thresholds (LONG ≥95, SHORT ≥90) + - Settings UI with real-time collateral display + - Risk-adjusted position sizing based on signal confidence + - **Status:** ✅ ACTIVE in production (Dec 1, 2025) + +### **ATR Trailing Stop System** +- `ATR_TRAILING_STOP_FIX.md` - **Phase 7.3 adaptive trailing enhancement** + - Real-time 1-minute ADX monitoring (not static entry ADX) + - Dynamic trail multipliers: base 1.5× + ADX strength tier + acceleration bonus + - ADX acceleration (+7 points) → 1.3× multiplier + - Profit acceleration (>2%) → additional 1.3× multiplier + - Combined max: 3.16× multiplier for strong trending moves + - **Status:** ✅ DEPLOYED (Nov 27, 2025) + +### **Indicator Version Tracking** +- `INDICATOR_VERSION_TRACKING.md` - **TradingView strategy versioning** + - v9: Money Line with MA Gap + Momentum SHORT Filter (PRODUCTION) + - v8: Money Line Sticky Trend (ARCHIVED - 57.1% WR baseline) + - Historical versions (v5-v7) for comparison + - Database field tracking for performance analysis + - **Purpose:** A/B testing and strategy evolution + +### **Position Synchronization** +- `POSITION_SYNC_QUICK_REF.md` - **Position Manager sync procedures** + - Startup validation (orphan detection) + - External closure handling + - Ghost position cleanup + - Database-first state management + - **Critical for:** System restarts, container crashes + +--- + +## 🎯 Design Principles + +### **1. Database as Source of Truth** +- All state-changing operations write to database FIRST +- In-memory state (Position Manager) is derived from database +- Container restarts restore state from database +- External closures update database before removing from memory + +### **2. Dual-Layer Redundancy** +- **Layer 1:** On-chain orders (Drift Protocol TP/SL limit orders) +- **Layer 2:** Software monitoring (Position Manager every 2 seconds) +- If on-chain orders fail, Position Manager executes via market orders +- Both layers track to same database for complete history + +### **3. Singleton Pattern Everywhere** +```typescript +// CORRECT: Use getter functions +const driftService = await initializeDriftService() +const positionManager = await getInitializedPositionManager() +const prisma = getPrismaClient() + +// WRONG: Never create multiple instances +const driftService = new DriftService() // ❌ +``` + +### **4. Adaptive Systems Over Static Parameters** +- **ATR-based TP/SL:** Adapts to market volatility automatically +- **Adaptive Leverage:** Scales risk with signal quality +- **ADX-based Trailing:** Widens trail during trend acceleration +- **Smart Entry:** Validates price action before execution + +### **5. Direction-Specific Configuration** +- LONG signals: Different quality thresholds than SHORT +- SHORT signals: Higher baseline difficulty, lower thresholds +- Momentum filters: Different for LONG (breakout) vs SHORT (reversal) +- Risk management: ADX-based runner SL positioning + +--- + +## 📊 System Components + +### **Core Services (Singletons)** +- **DriftService:** Drift Protocol SDK wrapper, WebSocket subscriptions +- **Position Manager:** Software monitoring loop (2-second price checks) +- **Pyth Price Monitor:** Real-time oracle price feeds (WebSocket) +- **Stop Hunt Tracker:** 4-hour revenge window monitoring +- **Blocked Signal Tracker:** 30-minute price tracking for blocked signals +- **Smart Validation Queue:** Pullback confirmation system (30-second checks) + +### **Key Workflows** +1. **Trade Execution:** TradingView → n8n → check-risk → execute → database → Position Manager +2. **Exit Management:** Price check → TP/SL trigger → closePosition → database → Telegram notification +3. **Startup Recovery:** Database query → Drift validation → restore Position Manager → resume monitoring +4. **Ghost Detection:** Drift API check → database cleanup → Position Manager removal + +### **Data Flow** +``` +TradingView Alert (5-min candle) + ↓ [Webhook] +n8n Parse Signal Enhanced + ↓ [HTTP POST] +/api/trading/check-risk + ↓ [Quality score, cooldown, duplicates] +/api/trading/execute + ↓ [Market order, TP/SL placement] +Database (Trade table) + ↓ [State persistence] +Position Manager (monitoring) + ↓ [2-second price checks] +Exit Detection (TP1/TP2/SL/Trailing) + ↓ [Market close order] +Database Update (exit details) + ↓ [P&L, exit reason, timestamps] +Telegram Notification +``` + +--- + +## 🔧 Configuration Architecture + +### **Three-Layer Merge Priority** +1. **Per-symbol ENV vars** (highest): `SOLANA_*`, `ETHEREUM_*` +2. **Market config** (code): `SUPPORTED_MARKETS` in config/trading.ts +3. **Global ENV vars** (fallback): `MAX_POSITION_SIZE_USD`, `LEVERAGE` +4. **Default config** (lowest): `DEFAULT_TRADING_CONFIG` + +### **Key Configuration Files** +- `config/trading.ts` - Central config with merge logic +- `.env` - Environment variables (482 lines) +- `prisma/schema.prisma` - Database schema +- `.github/copilot-instructions.md` - System documentation (6,400+ lines) + +--- + +## 🚀 Adding New Architecture + +**When to Create Architecture Docs:** +- Major system component (Position Manager, Drift client, etc.) +- Risk management system (leverage, stops, entry validation) +- Data flow pattern (TradingView → execution → monitoring) +- Critical decision (database-first vs in-memory, singleton vs multiple instances) + +**Template Structure:** +```markdown +# [Component Name] + +**Purpose:** [One-line description] + +## Architecture Decision +[Why this design was chosen, alternatives considered] + +## Implementation +[Code structure, key classes, integration points] + +## Data Flow +[How data moves through the component] + +## Configuration +[ENV variables, defaults, merge priority] + +## Error Handling +[Failure scenarios, recovery procedures] + +## Verification +[How to test, expected behaviors, logs to watch] + +## Deployment +[Restart requirements, migration steps] + +## Lessons Learned +[What worked, what didn't, future improvements] +``` + +--- + +## 🔍 Finding Architecture Info + +**By Component:** +- Leverage system → `ADAPTIVE_LEVERAGE_SYSTEM.md` +- Trailing stops → `ATR_TRAILING_STOP_FIX.md` +- Indicator versions → `INDICATOR_VERSION_TRACKING.md` +- Position sync → `POSITION_SYNC_QUICK_REF.md` + +**By Pattern:** +- Singleton → All architecture docs mention singleton pattern +- Adaptive systems → ATR, Leverage, Trailing Stop docs +- Database-first → Position Sync, all architecture patterns +- Direction-specific → Leverage, Indicator docs + +--- + +## ⚠️ Critical Architecture Constraints + +**Real Money System:** +- Every architectural decision affects financial outcomes +- Database integrity is non-negotiable (write FIRST, read to restore) +- Position Manager MUST have TP/SL backup (on-chain orders can fail) +- Container restarts MUST restore full state (database = source of truth) + +**Performance Requirements:** +- Position Manager: 2-second monitoring loop (not slower) +- Price feeds: WebSocket real-time (not HTTP polling) +- Order placement: <1 second execution (market orders) +- Database writes: Non-blocking (async, don't wait for confirmation) + +**Scalability Limits:** +- Maximum 20 simultaneous positions tracked +- Maximum 10 trades per hour per symbol +- Drift SDK: 50 RPC calls/second burst, 10/second sustained +- PostgreSQL: 10 concurrent connections max + +--- + +See `../README.md` for overall documentation structure. diff --git a/docs/archived/README.md b/docs/archived/README.md new file mode 100644 index 0000000..2a3e072 --- /dev/null +++ b/docs/archived/README.md @@ -0,0 +1,259 @@ +# Archived Documentation + +**Historical documentation for deprecated features and old system versions.** + +This directory contains documentation for features that have been removed, deprecated, or replaced by newer systems. Kept for historical reference and learning from past decisions. + +--- + +## 📦 What's Archived Here + +**Archived files are kept for:** +1. **Historical reference** - Understanding past design decisions +2. **Learning from failures** - Why certain approaches didn't work +3. **Baseline comparisons** - Comparing new systems to old +4. **Recovery information** - If rollback ever needed (rare) + +**Archived files are NOT:** +- ❌ Active in production +- ❌ Maintained or updated +- ❌ Recommended for new implementations +- ❌ Complete or fully accurate anymore + +--- + +## 🗄️ Currently Archived (Examples) + +### **Indicator Versions (Replaced by v9)** + +**v5: Buy/Sell Signal Strategy** +- **Period:** Pre-Nov 12, 2024 +- **Performance:** 36.4% WR, +$25.47 total +- **Why replaced:** Poor win rate, basic signal detection +- **Successor:** v6 HalfTrend + BarColor + +**v6: HalfTrend + BarColor** +- **Period:** Nov 12-18, 2024 +- **Performance:** 48% WR, -$47.70 total +- **Why replaced:** Marginal improvement, still underperforming +- **Successor:** v8 Money Line Sticky Trend + +**v7: v6 with Toggles** +- **Period:** Nov 2024 (minimal data) +- **Performance:** Deprecated before significant data collected +- **Why replaced:** No fundamental improvements over v6 +- **Successor:** v8 Money Line Sticky Trend + +**v8: Money Line Sticky Trend** +- **Period:** Nov 18-26, 2024 +- **Performance:** 57.1% WR, +$262.70 (8 trades) +- **Why replaced:** SHORT failures (5 oversold disasters), weak chop entry +- **Key issues:** + * RSI filter for SHORTs caught oversold (25-35 RSI) = wrong entries + * Quality 80 signal at ADX 20.7 (weak chop) = stopped out +- **Successor:** v9 Money Line with Momentum-Based SHORT Filter +- **Lesson learned:** Oversold doesn't mean reversal, need momentum at tops + +**v10: Adaptive Position Scoring (REMOVED)** +- **Period:** Nov 28, 2024 (1 day) +- **Performance:** Parameter insensitivity (72 configs = identical $498.12 P&L) +- **Why removed:** Bug in penalty logic, no edge over v9, added complexity +- **Key issues:** + * Price position penalty applied to 18.9% (should only apply to 40-60% chop) + * No performance improvement despite added complexity + * Violated "simpler is better" principle +- **Lesson learned:** Parameter insensitivity indicates no real edge, just noise +- **Status:** ✅ FULLY REMOVED (Nov 28, 2025) + +--- + +## 🎯 Current Production System + +**v9: Money Line with MA Gap + Momentum SHORT Filter** +- **Period:** Nov 26, 2025+ +- **Status:** ✅ PRODUCTION +- **Key Features:** + * MA Gap analysis (+5 to +15 quality points) + * Momentum-Based SHORT filter (ADX ≥23, Position ≥60% OR ≤40% + Vol ≥2.0x) + * Removed RSI filter for SHORTs (data showed RSI 50+ has 68.2% WR) + * Quality thresholds: LONG ≥90, SHORT ≥80 +- **File:** `workflows/trading/moneyline_v9_ma_gap.pinescript` +- **Documentation:** `../roadmaps/INDICATOR_V9_MA_GAP_ROADMAP.md` + +--- + +## 📊 Performance Comparison + +### **Win Rate Evolution** +``` +v5: 36.4% WR (baseline - poor) +v6: 48.0% WR (+11.6% improvement) +v8: 57.1% WR (+9.1% improvement) +v9: TBD (expected >60% with momentum filter) +``` + +### **Key Learnings from Archived Versions** + +**From v5 → v6:** +- Basic signal detection insufficient +- Need trend confirmation (HalfTrend) +- Color-based bar confirmation helps + +**From v6 → v8:** +- 0.6% flip threshold prevents whipsaw +- Momentum confirmation critical (not just price cross) +- Anti-whipsaw logic needed (MA spacing) + +**From v8 → v9:** +- **CRITICAL:** Don't short oversold (RSI 25-35) +- Short momentum at tops (Position ≥60%) +- ADX ≥23 filters weak chop (avoid ADX 20.7 disasters) +- RSI 50+ SHORTs = 68.2% WR (best performance!) +- Quality thresholds should differ by direction (LONG 90, SHORT 80) + +**From v10 attempt:** +- Parameter insensitivity = no edge +- Simpler systems often better than complex +- Bug-free simple logic > buggy complex logic +- Don't add complexity without proven edge + +--- + +## 🔍 Finding Archived Information + +**By Feature:** +- Old indicators → Check indicator version numbers (v5, v6, v7, v8, v10) +- Deprecated systems → Look for "REMOVED" or "DEPRECATED" in filenames +- Historical baselines → Compare performance metrics in docs + +**By Date:** +- Pre-Nov 12, 2024 → v5 era +- Nov 12-18, 2024 → v6 era +- Nov 18-26, 2024 → v8 era +- Nov 26, 2024+ → v9 era (current) + +**By Performance:** +- Worst: v5 (36.4% WR) +- Moderate: v6 (48% WR) +- Good: v8 (57.1% WR) +- Best: v9 (expected 60%+ WR) + +--- + +## ⚠️ Using Archived Documentation + +**DO:** +- ✅ Reference for historical context +- ✅ Learn from past mistakes +- ✅ Compare old vs new performance +- ✅ Understand why changes were made + +**DON'T:** +- ❌ Implement archived features in production +- ❌ Assume archived docs are accurate for current system +- ❌ Use archived parameters without validation +- ❌ Copy code from archived implementations + +--- + +## 📝 Archiving Process + +**When to Archive:** +1. Feature completely replaced by newer system +2. Performance proven inferior to current +3. No longer maintained or updated +4. Kept only for historical reference + +**How to Archive:** +1. Move file to `docs/archived/` directory +2. Add deprecation notice at top of file +3. Update references in active docs +4. Remove from main navigation +5. Keep in git history (never delete) + +**Deprecation Notice Template:** +```markdown +# ⚠️ ARCHIVED - [Feature Name] + +**Status:** DEPRECATED (Replaced by [Successor]) +**Date Archived:** [Date] +**Reason:** [Why replaced/removed] + +**For current system, see:** [Link to replacement docs] + +--- + +[Original content below...] +``` + +--- + +## 🗑️ What Gets Deleted vs Archived + +**Archived (Kept):** +- ✅ Historical indicator versions +- ✅ Deprecated but once-production features +- ✅ Performance baselines and comparisons +- ✅ Design decisions that didn't work out +- ✅ Learning experiences worth documenting + +**Deleted (Removed):** +- ❌ Temporary test files +- ❌ Duplicate documentation +- ❌ Incomplete drafts never used +- ❌ Outdated troubleshooting for removed systems +- ❌ Dead code with no historical value + +--- + +## 📚 Historical Analysis Resources + +**Performance Tracking:** +- Database: Trade table with `indicatorVersion` field +- SQL queries: `WHERE indicatorVersion IN ('v5', 'v6', 'v8')` +- Analytics UI: Compare tab shows version performance +- CSV exports: Historical backtest results + +**Commit History:** +```bash +# Find when feature was archived +git log --all --full-history -- "docs/archived/filename.md" + +# See original implementation +git show [commit_hash]:path/to/original/file.ts + +# Compare versions +git diff [old_commit] [new_commit] -- path/to/file +``` + +**Documentation Trail:** +- Copilot instructions: Common Pitfalls section documents issues +- Deployment docs: Show when features were removed +- Roadmaps: Track feature evolution over time + +--- + +## 🎓 Learning from History + +### **Successful Patterns** +- **Iterative improvement:** v5 → v6 → v8 → v9 steady progress +- **Data-driven decisions:** Each change validated with trade data +- **Simple foundations:** v9 builds on v8 core, removes complexity +- **Direction-specific logic:** LONG and SHORT need different filters + +### **Failed Patterns** +- **Premature complexity:** v10 added complexity without proven edge +- **Parameter insensitivity:** Identical results across configs = no edge +- **Rigid filters:** RSI <35 for SHORT caught wrong phase (oversold) +- **Ignoring direction differences:** Same thresholds for LONG/SHORT suboptimal + +### **Key Lessons** +1. **Simple > Complex:** v9 removed RSI filter, improved performance +2. **Test at scale:** 72 v10 configs with identical P&L = clear signal +3. **Direction matters:** LONGs and SHORTs need different strategies +4. **Data beats theory:** RSI 50+ SHORTs = 68.2% WR (counterintuitive) +5. **Fix bugs quickly:** v10 removed after 1 day when bug discovered + +--- + +See `../README.md` for overall documentation structure. diff --git a/docs/bugs/README.md b/docs/bugs/README.md new file mode 100644 index 0000000..28c5a26 --- /dev/null +++ b/docs/bugs/README.md @@ -0,0 +1,242 @@ +# Bug Reports & Critical Fixes + +**Historical record of critical incidents, bugs, and their resolutions.** + +This directory contains CRITICAL_*.md and FIXES_*.md bug reports. Every file documents a real incident that cost money, time, or could have caused financial loss if not caught. + +--- + +## 🚨 Critical Bug Reports + +### **Position Management** +- `CRITICAL_FIX_POSITION_SIZE_BUG.md` - Smart Entry using webhook percentage as signal price + - **Impact:** $89 position sizes instead of $2,300, 97% pullback calculations + - **Root Cause:** TradingView webhook sent percentage (70.80) not price ($142.50) + - **Fix:** Use Pyth oracle price instead of webhook signal.price + - **Date:** Dec 3, 2025 + +- `CRITICAL_INCIDENT_UNPROTECTED_POSITION.md` - Database-first pattern violation + - **Impact:** Position opened with NO database record, NO Position Manager tracking + - **Root Cause:** Position Manager added before database save + - **Fix:** Database save FIRST, then Position Manager add + - **Date:** Nov 13, 2025 + +- `CRITICAL_TP1_FALSE_DETECTION_BUG.md` - TP1 detection fails when on-chain orders fill fast + - **Impact:** Winners marked as "SL" exits, analytics incorrect + - **Root Cause:** External closure detected after both TP1 + runner closed + - **Fix:** Percentage-based exit reason inference + - **Date:** Nov 19, 2025 + +### **Trade Execution** +- `CRITICAL_MISSING_TRADES_NOV19.md` - Trades executed but not in database + - **Impact:** 3 trades missing from database, P&L values inflated 5-14× + - **Root Cause:** P&L compounding in external closure detection + - **Fix:** Don't mutate trade.realizedPnL during calculation + - **Date:** Nov 19, 2025 + +- `CRITICAL_ISSUES_FOUND.md` - Multiple critical discovery incidents + - Container restart killing positions + - Phantom detection killing runners + - Field name mismatches in startup validation + - **Dates:** Various Nov 2025 + +--- + +## 🔧 Fix Documentation + +### **Runner System** +- `FIXES_RUNNER_AND_CANCELLATION.md` - Runner stop loss gap fixes + - **Problem:** No SL protection between TP1 and TP2 + - **Solution:** Explicit runner SL check in monitoring loop + - **Impact:** Prevented unlimited risk exposure on 25-40% runner + - **Date:** Nov 15, 2025 + +### **Applied Fixes** +- `FIXES_APPLIED.md` - Collection of multiple bug fixes + - TP1 detection + - P&L calculation + - External closure handling + - Order cancellation + - **Period:** Nov 2025 + +--- + +## 📋 Common Bug Patterns + +### **1. P&L Compounding (Multiple Incidents)** +**Pattern:** Monitoring loop detects closure multiple times, accumulates P&L +**Symptoms:** Database shows 5-20× actual P&L, duplicate Telegram notifications +**Root Cause:** Async operations + monitoring loop = race condition +**Solution:** Delete from Map IMMEDIATELY (atomic operation) before any async work + +**Example Fixes:** +- Common Pitfall #49 (Nov 17, 2025) +- Common Pitfall #61 (Nov 22, 2025) +- Common Pitfall #67 (Dec 2, 2025) + +### **2. Database-First Violations** +**Pattern:** In-memory state updated before database write +**Symptoms:** Container restart loses tracking, positions unprotected +**Root Cause:** Developer assumes database always succeeds +**Solution:** Always database write FIRST, then update in-memory state + +**Example Fixes:** +- Common Pitfall #29 (Nov 13, 2025) +- CRITICAL_INCIDENT_UNPROTECTED_POSITION.md + +### **3. External Closure Detection** +**Pattern:** Position closed on-chain, Position Manager detects late +**Symptoms:** Duplicate updates, wrong exit reasons, ghost positions +**Root Cause:** Drift state propagation delay (5-10 seconds) +**Solution:** Verification wait + closingInProgress flag + ghost detection + +**Example Fixes:** +- Common Pitfall #47 (Nov 16, 2025) - Close verification gap +- Common Pitfall #56 (Nov 20, 2025) - Ghost orders +- Common Pitfall #57 (Nov 20, 2025) - P&L accuracy + +### **4. Unit Conversion Errors** +**Pattern:** Tokens vs USD, percentage vs decimal, on-chain vs display units +**Symptoms:** Position sizes off by 100×+, TP/SL at wrong prices +**Root Cause:** SDK returns different units than expected +**Solution:** Always log raw values, verify units explicitly + +**Example Fixes:** +- Common Pitfall #22 (Nov 12, 2025) - position.size is TOKENS not USD +- Common Pitfall #68 (Dec 3, 2025) - Signal price is percentage not USD + +--- + +## 🔍 Debugging Checklist + +**When investigating bugs:** + +1. **Check logs first:** +```bash +docker logs -f trading-bot-v4 | grep -E "CRITICAL|ERROR|Failed" +``` + +2. **Verify deployment:** +```bash +# Container start time MUST be > commit timestamp +docker inspect trading-bot-v4 --format='{{.State.StartedAt}}' +git log -1 --format='%ai %s' +``` + +3. **Query database state:** +```sql +-- Check for inconsistencies +SELECT * FROM "Trade" +WHERE exitReason IS NULL + AND (createdAt < NOW() - INTERVAL '1 hour') +ORDER BY createdAt DESC; +``` + +4. **Compare to Drift:** +```bash +# Get actual position from Drift +curl -X GET http://localhost:3001/api/trading/positions \ + -H "Authorization: Bearer $API_SECRET_KEY" +``` + +5. **Search Common Pitfalls:** +```bash +grep -n "symptom_keyword" .github/copilot-instructions.md | head -20 +``` + +--- + +## 📝 Creating Bug Reports + +**Required Sections:** +```markdown +# [Bug Title] (CRITICAL - Fixed [Date]) + +**Symptom:** [What user observed] + +**Root Cause:** [Technical explanation] + +**Real Incident ([Date]):** +* [Specific trade or event] +* [Expected behavior] +* [Actual behavior] +* [Financial impact] + +**Impact:** [Scope of problem] + +**Fix ([Date]):** +```typescript +// Code showing the fix +``` + +**Files changed:** [List of files] + +**Git commit:** [Commit hash] + +**Deployed:** [Deployment timestamp] + +**Verification Required:** +* [How to test fix works] +* [Expected logs/behavior] + +**Lessons Learned:** +1. [Key insight #1] +2. [Key insight #2] +``` + +**Naming Convention:** +- `CRITICAL_[TOPIC]_BUG.md` - Single critical bug +- `CRITICAL_[TOPIC]_[DATE].md` - Dated incident report +- `FIXES_[SYSTEM].md` - Multiple related fixes + +--- + +## ⚠️ Prevention Guidelines + +**From Past Incidents:** + +1. **Always verify deployment before declaring "fixed"** + - Container restart timestamp > commit timestamp + - Test with actual trade if possible + - Check logs for expected behavior change + +2. **Never trust SDK data formats** + - Log raw values first + - Verify units explicitly (tokens vs USD, % vs decimal) + - Check SDK docs are often wrong + +3. **Database writes before in-memory updates** + - Save to database FIRST + - Only then update Position Manager, caches, etc. + - If DB fails, don't proceed + +4. **Async operations need serialization** + - Delete from Map IMMEDIATELY (atomic lock) + - Don't check Map.has() then delete later (race condition) + - Use Map.delete() return value as lock + +5. **External closures need verification** + - Wait 5-10 seconds for Drift state propagation + - Query Drift to confirm position actually closed + - Keep monitoring if verification fails + +--- + +## 📞 Escalation + +**When to declare "CRITICAL":** +- Financial loss occurred or was narrowly avoided +- System produced incorrect data (P&L, positions, exits) +- Position left unprotected (no monitoring, no TP/SL) +- Bug could recur and cause future losses + +**Response Required:** +1. Stop trading immediately if data integrity affected +2. Document incident in this directory +3. Add to Common Pitfalls in copilot-instructions.md +4. Fix + deploy + verify within 24 hours +5. Update relevant architecture docs + +--- + +See `../README.md` for overall documentation structure. diff --git a/docs/cluster/README.md b/docs/cluster/README.md new file mode 100644 index 0000000..8e51091 --- /dev/null +++ b/docs/cluster/README.md @@ -0,0 +1,291 @@ +# Distributed Computing & EPYC Cluster + +**Infrastructure for large-scale parameter optimization and backtesting.** + +This directory contains documentation for the EPYC cluster setup, distributed backtesting coordination, and multi-server infrastructure. + +--- + +## 🖥️ Cluster Documentation + +### **EPYC Server Setup** +- `EPYC_SETUP_COMPREHENSIVE.md` - **Complete setup guide** + - Hardware: AMD EPYC 7282 16-Core Processor (Debian 12 Bookworm) + - Python environment: 3.11.2 with pandas 2.3.3, numpy 2.3.5 + - SSH configuration: Nested hop (master → worker1 → worker2) + - Package deployment: tar.gz transfer with virtual environment + - **Status:** ✅ OPERATIONAL (24 workers processing 65,536 combos) + +### **Distributed Architecture** +- `DUAL_SWEEP_README.md` - **Parallel sweep execution** + - Coordinator: Assigns chunks to workers + - Workers: Execute parameter combinations in parallel + - Database: SQLite exploration.db for state tracking + - Results: CSV files with top N configurations + - **Use case:** v9 exhaustive parameter optimization (Nov 28-29, 2025) + +### **Cluster Control** +- `CLUSTER_START_BUTTON_FIX.md` - **Web UI integration** + - Dashboard: http://localhost:3001/cluster + - Start/Stop buttons with status detection + - Database-first status (SSH supplementary) + - Real-time progress tracking + - **Status:** ✅ DEPLOYED (Nov 30, 2025) + +--- + +## 🏗️ Cluster Architecture + +### **Physical Infrastructure** +``` +Master Server (local development machine) + ├── Coordinator Process (assigns chunks) + ├── Database (exploration.db) + └── Web Dashboard (Next.js) + ↓ [SSH] +Worker1 (EPYC 10.10.254.106) + ├── 12 worker processes + ├── 64GB RAM + └── Direct SSH connection + ↓ [SSH ProxyJump] +Worker2 (EPYC 10.20.254.100) + ├── 12 worker processes + ├── 64GB RAM + └── Via worker1 hop +``` + +### **Data Flow** +``` +1. Coordinator creates chunks (2,000 combos each) + ↓ +2. Marks chunk status='pending' in database + ↓ +3. Worker queries database for pending chunks + ↓ +4. Coordinator assigns chunk to worker via SSH + ↓ +5. Worker updates status='running' + ↓ +6. Worker processes combinations in parallel + ↓ +7. Worker saves results to strategies table + ↓ +8. Worker updates status='completed' + ↓ +9. Coordinator assigns next pending chunk + ↓ +10. Dashboard shows real-time progress +``` + +### **Database Schema** +```sql +-- chunks table: Work distribution +CREATE TABLE chunks ( + id TEXT PRIMARY KEY, -- v9_chunk_000000 + start_combo INTEGER, -- 0, 2000, 4000, etc. + end_combo INTEGER, -- 2000, 4000, 6000, etc. + status TEXT, -- 'pending', 'running', 'completed' + assigned_worker TEXT, -- 'worker1', 'worker2' + started_at INTEGER, + completed_at INTEGER +); + +-- strategies table: Results storage +CREATE TABLE strategies ( + id INTEGER PRIMARY KEY, + chunk_id TEXT, + params TEXT, -- JSON of parameter values + pnl REAL, + win_rate REAL, + profit_factor REAL, + max_drawdown REAL, + total_trades INTEGER +); +``` + +--- + +## 🚀 Using the Cluster + +### **Starting a Sweep** +```bash +# 1. Prepare package on master +cd /home/icke/traderv4/backtester +tar -czf backtest_v9_sweep.tar.gz data/ backtester_core.py v9_moneyline_ma_gap.py moneyline_core.py + +# 2. Transfer to EPYC workers +scp backtest_v9_sweep.tar.gz root@10.10.254.106:/home/backtest/ +ssh root@10.10.254.106 "scp backtest_v9_sweep.tar.gz root@10.20.254.100:/home/backtest/" + +# 3. Extract on workers +ssh root@10.10.254.106 "cd /home/backtest && tar -xzf backtest_v9_sweep.tar.gz" +ssh root@10.10.254.106 "ssh root@10.20.254.100 'cd /home/backtest && tar -xzf backtest_v9_sweep.tar.gz'" + +# 4. Start via web dashboard or CLI +# Web: http://localhost:3001/cluster → Click "Start Cluster" +# CLI: cd /home/icke/traderv4/cluster && python v9_advanced_coordinator.py +``` + +### **Monitoring Progress** +```bash +# Dashboard +curl -s http://localhost:3001/api/cluster/status | jq + +# Database query +sqlite3 cluster/exploration.db "SELECT status, COUNT(*) FROM chunks GROUP BY status;" + +# Worker processes +ssh root@10.10.254.106 "ps aux | grep [p]ython | grep backtest | wc -l" +``` + +### **Collecting Results** +```bash +# Results saved to cluster/results/ +ls -lh cluster/results/sweep_v9_*.csv + +# Top 100 configurations by P&L +sqlite3 cluster/exploration.db "SELECT params, pnl, win_rate FROM strategies ORDER BY pnl DESC LIMIT 100;" +``` + +--- + +## 🔧 Configuration + +### **Coordinator Settings** +```python +# cluster/v9_advanced_coordinator.py +WORKERS = { + 'worker1': {'host': '10.10.254.106', 'port': 22}, + 'worker2': {'host': '10.20.254.100', 'port': 22, 'proxy_jump': '10.10.254.106'} +} + +CHUNK_SIZE = 2000 # Combinations per chunk +MAX_WORKERS = 24 # 12 per server +CHECK_INTERVAL = 60 # Status check frequency (seconds) +``` + +### **Worker Settings** +```python +# cluster/distributed_worker.py +NUM_PROCESSES = 12 # Parallel backtests +BATCH_SIZE = 100 # Save results every N combos +TIMEOUT = 120 # Per-combo timeout (seconds) +``` + +### **SSH Configuration** +```bash +# ~/.ssh/config +Host worker1 + HostName 10.10.254.106 + User root + StrictHostKeyChecking no + ServerAliveInterval 30 + +Host worker2 + HostName 10.20.254.100 + User root + ProxyJump worker1 + StrictHostKeyChecking no + ServerAliveInterval 30 +``` + +--- + +## 🐛 Common Issues + +### **SSH Timeout Errors** +**Symptom:** "SSH command timed out for worker2" +**Root Cause:** Nested hop requires 60s timeout (not 30s) +**Fix:** Common Pitfall #64 - Increase subprocess timeout +```python +result = subprocess.run(ssh_cmd, timeout=60) # Not 30 +``` + +### **Database Lock Errors** +**Symptom:** "database is locked" +**Root Cause:** Multiple workers writing simultaneously +**Fix:** Use WAL mode + increase busy_timeout +```python +connection.execute('PRAGMA journal_mode=WAL') +connection.execute('PRAGMA busy_timeout=10000') +``` + +### **Worker Not Processing** +**Symptom:** Chunk status='running' but no worker processes +**Root Cause:** Worker crashed or SSH session died +**Fix:** Cleanup database + restart +```bash +# Mark stuck chunks as pending +sqlite3 exploration.db "UPDATE chunks SET status='pending', assigned_worker=NULL WHERE status='running' AND started_at < (strftime('%s','now') - 600);" +``` + +### **Status Shows "Idle" When Running** +**Symptom:** Dashboard shows idle despite workers running +**Root Cause:** SSH detection timing out, database not queried first +**Fix:** Database-first status detection (Common Pitfall #71) +```typescript +// Check database BEFORE SSH +const hasRunningChunks = explorationData.chunks.running > 0 +if (hasRunningChunks) clusterStatus = 'active' +``` + +--- + +## 📊 Performance Metrics + +**v9 Exhaustive Sweep (65,536 combos):** +- **Duration:** ~29 hours (24 workers) +- **Speed:** 1.60s per combo (4× faster than 6 local workers) +- **Throughput:** ~37 combos/minute across cluster +- **Data processed:** 139,678 OHLCV rows × 65,536 combos = 9.16B calculations +- **Results:** Top 100 saved to CSV (~10KB file) + +**Cost Analysis:** +- Local (6 workers): 72 hours estimated +- EPYC (24 workers): 29 hours actual +- **Time savings:** 43 hours (60% faster) +- **Resource utilization:** 64 cores utilized vs 6 local + +--- + +## 📝 Adding Cluster Features + +**When to Use Cluster:** +- Parameter sweeps >10,000 combinations +- Backtests requiring >24 hours on local machine +- Multi-strategy comparison (need parallel execution) +- Production validation (test many configs simultaneously) + +**Scaling Guidelines:** +- **<1,000 combos:** Local machine sufficient +- **1,000-10,000:** Single EPYC server (12 workers) +- **10,000-100,000:** Both EPYC servers (24 workers) +- **100,000+:** Consider cloud scaling (AWS Batch, etc.) + +--- + +## ⚠️ Important Notes + +**Data Transfer:** +- Always compress packages: `tar -czf` (1.9MB → 1.1MB) +- Verify checksums after transfer +- Use rsync for incremental updates + +**Process Management:** +- Always use `nohup` or `screen` for long-running coordinators +- Workers auto-terminate when chunks complete +- Coordinator sends Telegram notification on completion + +**Database Safety:** +- SQLite WAL mode prevents most lock errors +- Backup exploration.db before major sweeps +- Never edit chunks table manually while coordinator running + +**SSH Reliability:** +- ServerAliveInterval prevents silent disconnects +- StrictHostKeyChecking=no avoids interactive prompts +- ProxyJump handles nested hops automatically + +--- + +See `../README.md` for overall documentation structure. diff --git a/docs/deployments/README.md b/docs/deployments/README.md new file mode 100644 index 0000000..0236a02 --- /dev/null +++ b/docs/deployments/README.md @@ -0,0 +1,314 @@ +# Deployment Status & Verification + +**Verification reports for major feature deployments.** + +This directory contains *_COMPLETE.md and DEPLOYMENT_*.md status files documenting deployment verification for significant system changes. + +--- + +## ✅ Recent Deployments + +### **December 2025** +- `DEPLOYMENT_SUCCESS_DEC3_2025.md` - **Bug fixes deployment** + - Bug #1: Revenge system external closure integration + - Bug #5: Smart Validation Queue bypass in execute endpoint + - 2 critical fixes deployed and verified + - **Status:** ✅ VERIFIED (Dec 3, 09:02 UTC) + +### **November 2025** +- `V9_IMPLEMENTATION_COMPLETE.md` - **v9 Money Line indicator** + - MA Gap analysis (±5 to +15 quality points) + - Momentum-Based SHORT Filter (ADX ≥23, Position-based entry) + - Quality thresholds: LONG ≥90, SHORT ≥80 + - **Status:** ✅ PRODUCTION (Nov 26, 2025) + +- `PHASE_7.3_ADAPTIVE_TRAILING_DEPLOYED.md` - **Adaptive trailing stop** + - Real-time 1-minute ADX monitoring + - Dynamic multipliers: ADX acceleration/deceleration + - Combined max: 3.16× for strong trending moves + - **Status:** ✅ DEPLOYED (Nov 27, 2025) + +- `ONE_YEAR_RETENTION_DEPLOYMENT.md` - **Data retention policy** + - Database: 1 year retention for Trade/PriceUpdate tables + - BlockedSignal: 90 days retention + - Automated cleanup job + - **Status:** ✅ ACTIVE + +- `RUNNER_SYSTEM_FIX_COMPLETE.md` - **Runner stop loss protection** + - Fixed: No SL protection between TP1 and TP2 + - Added: Explicit runner SL check in monitoring loop + - ADX-based runner SL positioning + - **Status:** ✅ VERIFIED (Nov 15, 2025) + +- `REENTRY_SYSTEM_COMPLETE.md` - **Manual trade validation** + - Fresh TradingView data validation (<5min old) + - Recent performance modifiers (±20 points) + - Minimum score 55 (vs 60 for new signals) + - **Status:** ✅ OPERATIONAL + +- `MA_CROSSOVER_DETECTION_COMPLETE.md` - **MA crossover tracking** + - n8n Parse Signal Enhanced detects "crossing" keyword + - Flags: isMACrossover, isDeathCross, isGoldenCross + - Purpose: Data collection for ADX weak→strong validation + - **Status:** ✅ IMPLEMENTED (Nov 27, 2025) + +- `CLUSTER_STOP_BUTTON_FIX_COMPLETE.md` - **Cluster control UI** + - Database-first status detection (SSH supplementary) + - Stop button properly shown when cluster active + - Conditional rendering based on cluster status + - **Status:** ✅ DEPLOYED (Nov 30, 2025) + +- `SMART_ENTRY_DEPLOYMENT_STATUS.md` - **Smart Entry Validation Queue** + - Block & Watch system for quality 50-89 signals + - ±0.3% confirmation thresholds (optimized from backtest) + - 30-second monitoring with 10-minute expiry + - **Status:** ✅ ACTIVE (Nov 30, 2025) + +- `SMART_ENTRY_TIMING_STATUS.md` - **Smart Entry pullback detection** + - Waits for 0.5-2% pullback before entry + - Adaptive based on ATR and market conditions + - Improves entry prices on quality 85+ signals + - **Status:** ✅ ENABLED + +- `TP1_FIX_DEPLOYMENT_SUMMARY.md` - **TP1 detection accuracy** + - Fixed: TP1 detection when on-chain orders fill fast + - Percentage-based exit reason inference (0.3-1.2% = TP1) + - Database records now accurate + - **Status:** ✅ VERIFIED (Nov 19, 2025) + +- `PHASE_4_VERIFICATION_COMPLETE.md` - **Historical verification** + - Phase 4 position scaling optimizations + - TP2-as-runner system validation + - 40% runner with ATR trailing stop + - **Status:** ✅ COMPLETE + +--- + +## 📋 Deployment Checklist Template + +**For Every Major Deployment:** + +### **1. Pre-Deployment** +- [ ] Code changes committed and pushed +- [ ] Unit tests pass (if applicable) +- [ ] Integration tests pass +- [ ] Documentation updated (copilot-instructions.md) +- [ ] Deployment doc created in this directory + +### **2. Build & Deploy** +```bash +# Build Docker image +cd /home/icke/traderv4 +docker compose build trading-bot + +# Restart container +docker compose up -d --force-recreate trading-bot + +# Verify container running +docker ps | grep trading-bot +``` + +### **3. Verification** +```bash +# 1. Check container start time > commit time +echo "=== Container Start Time ===" && \ +docker inspect trading-bot-v4 --format='{{.State.StartedAt}}' + +echo -e "\n=== Recent Commits ===" && \ +git log --oneline --since="YYYY-MM-DD HH:MM" --format='%h %ai %s' + +# 2. Verify logs show expected behavior +docker logs -f trading-bot-v4 | grep -E "expected_log_pattern" + +# 3. Test API endpoints +curl http://localhost:3001/api/endpoint/path + +# 4. Database verification (if schema changes) +docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c "\\dt" +``` + +### **4. Post-Deployment** +- [ ] Container start time > all commit timestamps ✅ +- [ ] Expected log messages appear ✅ +- [ ] API endpoints respond correctly ✅ +- [ ] Database schema correct (if applicable) ✅ +- [ ] Test trade executed successfully (if trading logic changed) ✅ +- [ ] User notified of deployment ✅ + +### **5. Documentation** +- [ ] Deployment doc updated with verification results +- [ ] Common Pitfalls added (if bug fix) +- [ ] Roadmap updated (if completing phase) +- [ ] Git commit for deployment verification + +--- + +## 🚨 Critical Deployment Rules + +**NEVER declare "deployed" or "working" without:** +1. ✅ Verifying container start time > commit timestamp +2. ✅ Observing expected logs in production +3. ✅ Testing API endpoint or feature behavior +4. ✅ SQL query confirming database state (if applicable) +5. ✅ Test trade validation (if Position Manager changes) + +**Why This Matters:** +- This is a **real money trading system** +- Premature "working" declarations cause financial losses +- Container restart required for code changes to take effect +- Code committed ≠ Code deployed ≠ Code verified + +--- + +## 📝 Creating Deployment Docs + +**Naming Convention:** +- `[FEATURE]_COMPLETE.md` - Major feature fully implemented +- `DEPLOYMENT_SUCCESS_[DATE].md` - Multiple fixes deployed together +- `PHASE_[N]_VERIFICATION_COMPLETE.md` - Roadmap phase completion + +**Required Sections:** +```markdown +# [Feature Name] - DEPLOYED ([Date]) + +**Purpose:** [What this deployment does] + +**Status:** ✅ VERIFIED | ⏳ PENDING | ❌ FAILED + +--- + +## Changes Deployed + +### **Files Modified** +- `path/to/file1.ts` - [Description] +- `path/to/file2.ts` - [Description] + +### **Key Improvements** +1. [Change 1] +2. [Change 2] + +--- + +## Deployment Process + +**Build Time:** [Time] +**Container Restart:** [Time] +**Git Commit:** [Hash] - [Message] + +### **Verification Steps** +```bash +# Commands used to verify deployment +``` + +### **Verification Results** +- ✅ Container start time: [Time] > Commit time: [Time] +- ✅ Expected logs observed: "[Log message]" +- ✅ API test successful: [Response] +- ✅ Database state correct: [Query results] + +--- + +## Expected Behavior + +**Before Deployment:** +- [Old behavior description] + +**After Deployment:** +- [New behavior description] + +--- + +## Monitoring + +**What to Watch:** +- [Log patterns to monitor] +- [Database queries to check] +- [API endpoints to test] + +**Success Indicators:** +- [How to know it's working correctly] + +**Failure Indicators:** +- [How to detect problems] + +--- + +## Rollback Procedure + +**If Problems Occur:** +```bash +# Revert to previous container +docker tag traderv4-trading-bot:backup traderv4-trading-bot:latest +docker compose up -d --force-recreate trading-bot + +# Or git revert +git revert [commit_hash] +git push +# Then rebuild/restart +``` + +--- + +## Impact + +**Expected:** +- [Benefit 1] +- [Benefit 2] + +**Actual:** (Update after 24-48 hours) +- [Measured impact] + +--- + +## Lessons Learned + +1. [Key insight #1] +2. [Key insight #2] +``` + +--- + +## 🔍 Finding Deployment Info + +**By Date:** +- Most recent → Check filename dates (YYYY-MM-DD or Month YYYY) +- Historical → Older files in this directory + +**By Feature:** +- v9 indicator → `V9_IMPLEMENTATION_COMPLETE.md` +- Trailing stops → `PHASE_7.3_ADAPTIVE_TRAILING_DEPLOYED.md` +- Runner system → `RUNNER_SYSTEM_FIX_COMPLETE.md` +- Smart Entry → `SMART_ENTRY_*_STATUS.md` +- Cluster UI → `CLUSTER_STOP_BUTTON_FIX_COMPLETE.md` + +**By Status:** +- ✅ VERIFIED → Deployment confirmed working +- ⏳ PENDING → Awaiting verification +- 🔄 IN PROGRESS → Deployment ongoing + +--- + +## 📊 Deployment Metrics + +**Success Rate:** +- Track deployments requiring rollback +- Track verification time (commit → confirmed working) +- Track time to detect issues + +**Typical Timeline:** +- Code change → Commit: 5-30 minutes +- Build Docker image: 60-90 seconds +- Container restart: 5-10 seconds +- Verification: 1-5 minutes +- **Total:** 10-45 minutes from code to verified + +**Best Practices:** +- Always use `--force-recreate` flag for container restart +- Wait 10 seconds after restart before testing +- Test in order: Logs → API → Database → Feature +- Document verification steps in deployment doc +- Update roadmaps after phase completions + +--- + +See `../README.md` for overall documentation structure. diff --git a/docs/roadmaps/README.md b/docs/roadmaps/README.md new file mode 100644 index 0000000..79ea828 --- /dev/null +++ b/docs/roadmaps/README.md @@ -0,0 +1,254 @@ +# Strategic Planning & Roadmaps + +**Long-term strategic planning for Trading Bot v4 system enhancements.** + +This directory contains all *ROADMAP*.md files documenting planned improvements across multiple optimization dimensions. + +--- + +## 🗺️ Active Roadmaps + +### **Master Roadmap** +- `OPTIMIZATION_MASTER_ROADMAP.md` - **Consolidated strategic view** + - 3 major initiatives: Signal Quality, Position Scaling, ATR-based TP + - Combined impact: 35-40% P&L improvement when complete + - Data collection phases with 50-100 trade validation requirements + - **Status:** Phase 1 (data collection) across all three initiatives + +### **Signal Quality** +- `SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md` - **Data-driven quality threshold optimization** + - Phase 1: Collect 10-20 blocked signals (🔄 IN PROGRESS) + - Phase 2: Analyze patterns, adjust thresholds (🔜 NEXT) + - Phase 3: Dual-threshold system or quality-specific sizing + - Phase 4: Automated price analysis for blocked signals + - Phase 5: ML-based scoring weight optimization + - **Goal:** Recover profitable setups currently blocked, filter true losers + +### **Position Scaling** +- `POSITION_SCALING_ROADMAP.md` - **TP/SL and runner optimization** + - Phase 1: ✅ COMPLETE - Data collection with quality scores + - Phase 2: ATR-based dynamic targets (adapt to volatility) + - Phase 3: Signal quality-based scaling (high quality = larger runners) + - Phase 4: Direction-based optimization (shorts vs longs) + - Phase 5: ✅ COMPLETE - TP2-as-runner system (40% runner with ATR trailing) + - Phase 6: ML-based exit prediction + - **Goal:** Capture more of winning moves, reduce stop-outs + +### **ATR-Based Risk Management** +- `ATR_BASED_TP_ROADMAP.md` - **Volatility-adaptive TP/SL system** + - Phase 1: ATR data collection from TradingView + - Phase 2: ✅ COMPLETE - ATR-based TP/SL calculation (Nov 17, 2025) + - Phase 3: Validation with 50+ trades + - Phase 4: Per-symbol ATR calibration + - **Goal:** Regime-agnostic targets (bull/bear markets adapt automatically) + +### **1-Minute Data Enhancements** +- `1MIN_DATA_ENHANCEMENTS_ROADMAP.md` - **High-frequency data utilization** + - Phase 7.1: Smart Entry Timing (pullback detection) + - Phase 7.2: Revenge Trading System (90s confirmation, ADX-based) + - Phase 7.3: ✅ DEPLOYED - Adaptive Trailing Stop with real-time ADX (Nov 27, 2025) + - Phase 7.4: Multi-timeframe signal validation + - **Goal:** Use 1-minute data for entry timing, trailing stop optimization + +### **Indicator Evolution** +- `INDICATOR_V9_MA_GAP_ROADMAP.md` - **TradingView strategy development** + - v9 Money Line: MA Gap analysis + Momentum SHORT Filter + - Parameter optimization: 65,536 exhaustive sweep on EPYC cluster + - Quality threshold tuning: LONG ≥90, SHORT ≥95 + - **Goal:** Maximum profitability while maintaining quality standards + +### **High Availability** +- `HA_SETUP_ROADMAP.md` - **99.9%+ uptime infrastructure** + - Phase 1-8: ✅ ALL COMPLETE (Nov 25, 2025) + - Primary: srvdocker02 (95.216.52.28:3001) + - Secondary: Hostinger (72.62.39.24:3001) + - Database replication, DNS failover, auto-recovery + - **Status:** Zero-downtime failover validated in production + +--- + +## 📊 Roadmap Methodology + +### **Phase Structure** +All roadmaps follow consistent phase naming: +- **Phase 1:** Data collection (50-100 trades typical) +- **Phase 2:** Analysis and pattern identification +- **Phase 3:** Implementation and testing +- **Phase 4:** Validation and optimization +- **Phase 5:** Production deployment +- **Phase 6+:** Advanced features (ML, automation) + +### **Status Indicators** +- ✅ **COMPLETE:** Implemented, deployed, validated with production data +- 🔄 **IN PROGRESS:** Active development or data collection +- 🔜 **NEXT:** Planned for immediate future (within 1-2 weeks) +- 🎯 **FUTURE:** Planned but not yet started +- 🤖 **DISTANT:** Long-term goal (ML, advanced features) + +### **Gating Criteria** +**Before moving to next phase:** +1. **Data requirement met:** Minimum sample size achieved (50-100 trades) +2. **SQL analysis complete:** Patterns identified, validated statistically +3. **User approval:** For parameter changes affecting risk/profitability +4. **Deployment verified:** Code deployed, container restarted, logs confirmed +5. **Documentation updated:** Roadmap status, copilot-instructions.md, deployment docs + +--- + +## 🎯 Current Focus (Dec 2025) + +### **Active Data Collection** +1. **Blocked Signals:** 0/20 target (need 10-20 for threshold analysis) +2. **v9 Indicator:** 10/100 trades (parameter optimization ongoing) +3. **Adaptive Leverage:** 5/50 trades (10x vs 5x tier validation) +4. **Smart Entry Queue:** 0/30 validations (pullback confirmation system) + +### **Recently Completed** +- ✅ TP2-as-runner system (40% runner, ATR trailing) +- ✅ ATR-based TP/SL (regime-agnostic targets) +- ✅ Adaptive leverage (10x high quality, 5x borderline) +- ✅ Phase 7.3 adaptive trailing (real-time ADX monitoring) +- ✅ HA setup (zero-downtime failover) + +### **Next Up** +1. **Quality threshold optimization** (LONG 90→?, SHORT 95→?) +2. **v9 parameter tuning** (65,536 combo EPYC sweep results) +3. **Multi-timeframe comparison** (5min vs 15min vs 1H) +4. **Smart Entry effectiveness** (±0.3% thresholds validation) + +--- + +## 📝 Adding New Roadmaps + +**When to create roadmap:** +- Multi-phase feature (>3 phases, >1 month duration) +- Strategic initiative affecting core system behavior +- Data collection requirement (need validation before production) +- User financial goals alignment needed + +**Template Structure:** +```markdown +# [Feature Name] Roadmap + +**Objective:** [One-line goal] +**Expected Impact:** [Dollar amount or % improvement] +**Duration:** [Estimated timeline] +**Status:** [Current phase] + +--- + +## Phase 1: [Name] (Status Emoji) +**Objective:** [What to achieve] +**Duration:** [Time estimate] +**Data Required:** [Sample size] + +### Tasks +- [ ] Task 1 +- [ ] Task 2 + +### Success Criteria +- [Measurable outcome] + +### Gating Decision +[What determines if we proceed to Phase 2] + +--- + +## Phase 2-N: [Continue pattern] + +--- + +## Progress Tracking +| Phase | Status | Start Date | End Date | Key Milestone | +|-------|--------|-----------|----------|---------------| +| Phase 1 | 🔄 IN PROGRESS | YYYY-MM-DD | - | 50 trades collected | + +--- + +## Dependencies +- [Other roadmaps or features] + +## Risks +- [Technical challenges] +- [Financial risks] + +## Decision Points +- [Key decisions requiring user approval] +``` + +--- + +## 🔍 Finding Roadmap Info + +**By Status:** +- Active → `OPTIMIZATION_MASTER_ROADMAP.md` (consolidated view) +- Complete → Check ✅ markers in individual roadmaps +- Next up → Look for 🔜 NEXT indicators +- Long-term → 🤖 DISTANT phases + +**By Feature:** +- Quality → `SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md` +- Position sizing → `POSITION_SCALING_ROADMAP.md` +- ATR system → `ATR_BASED_TP_ROADMAP.md` +- 1-min data → `1MIN_DATA_ENHANCEMENTS_ROADMAP.md` +- Indicator → `INDICATOR_V9_MA_GAP_ROADMAP.md` +- Infrastructure → `HA_SETUP_ROADMAP.md` + +**By Priority:** +1. **Critical path:** OPTIMIZATION_MASTER_ROADMAP.md +2. **Active development:** Any with 🔄 IN PROGRESS +3. **User-requested:** Check git commits for recent roadmap updates + +--- + +## 📈 Progress Tracking + +**Weekly Review:** +1. Update trade counts in data collection phases +2. Check SQL for analysis readiness (sample size thresholds) +3. Move phases from 🔄 to ✅ when complete +4. Update OPTIMIZATION_MASTER_ROADMAP.md combined impact estimates + +**Monthly Review:** +1. Validate expected vs actual impact (did optimization deliver promised %) +2. Adjust future phase estimates based on actual results +3. Reprioritize roadmaps based on performance data +4. Archive completed roadmaps to docs/archived/ + +**Milestone Commits:** +```bash +git commit -m "roadmap: Phase X complete - [Feature Name] + +- Data collection: X/Y trades +- Analysis results: [Key findings] +- Deployment: [Date] +- Impact: [Actual vs expected] + +Next: Phase X+1 starting [Date] +" +``` + +--- + +## ⚠️ Important Notes + +**Data-Driven Only:** +- Never implement Phase 3 without Phase 2 analysis complete +- Minimum 50 trades for statistical validity +- SQL queries required to prove assumptions + +**User Approval Required:** +- Quality threshold changes +- Leverage adjustments +- Risk parameter modifications +- Position sizing changes + +**Real Money System:** +- Every phase affects financial outcomes +- Conservative estimates (under-promise, over-deliver) +- Document both upside and downside scenarios +- Rollback procedures for every deployment + +--- + +See `../README.md` for overall documentation structure. diff --git a/docs/setup/README.md b/docs/setup/README.md new file mode 100644 index 0000000..63f3426 --- /dev/null +++ b/docs/setup/README.md @@ -0,0 +1,396 @@ +# Setup Guides & Configuration + +**Step-by-step guides for setting up trading bot components.** + +This directory contains setup instructions for TradingView alerts, n8n workflows, external services, and infrastructure configuration. + +--- + +## 🚀 Quick Setup Guides + +### **Essential Setup (Required for Trading)** +- `SIGNAL_QUALITY_SETUP_GUIDE.md` - **Signal quality system** + - TradingView alert configuration + - Webhook payload format (ATR, ADX, RSI, volume, price position) + - n8n Parse Signal Enhanced setup + - Quality scoring thresholds + - **Status:** ✅ PRODUCTION + +- `N8N_MARKET_DATA_SETUP.md` - **1-minute data collection** + - TradingView 1-minute alerts for market data cache + - Webhook endpoint: `/api/trading/market-data` + - Purpose: Smart Entry Validation Queue price updates + - Alert frequency: Every 1-5 minutes + - **Status:** ✅ ACTIVE + +### **Infrastructure Setup** +- `EPYC_SETUP_COMPREHENSIVE.md` - **EPYC cluster configuration** + - SSH configuration (nested hop through worker1) + - Python environment setup (3.11.2 + pandas + numpy) + - Package deployment (tar.gz transfer) + - Worker initialization scripts + - **Location:** Also in `../cluster/` directory + - **Status:** ✅ OPERATIONAL + +### **Development Setup** +- `QUICK_SETUP_CARD.md` - **Development environment** + - Docker Compose configuration + - Environment variables (.env file) + - Database initialization (PostgreSQL + Prisma) + - Local testing procedures + - **Status:** ✅ CURRENT + +--- + +## 📋 TradingView Alert Setup + +### **Production Trading Signals (5-minute chart)** + +**Alert Message Format:** +```json +{ + "action": "{{strategy.order.action}}", + "symbol": "{{ticker}}", + "timeframe": "{{interval}}", + "currentPrice": {{close}}, + "atr": {{ta.atr(14)}}, + "adx": {{ta.dmi(14, 14)}}, + "rsi": {{ta.rsi(14)}}, + "volumeRatio": {{volume / ta.sma(volume, 20)}}, + "pricePosition": {{(close - ta.lowest(low, 100)) / (ta.highest(high, 100) - ta.lowest(low, 100)) * 100}}, + "indicatorVersion": "v9" +} +``` + +**Alert Settings:** +- **Condition:** Strategy entry/exit +- **Trigger:** Once Per Bar Close +- **Expiration:** None (always active) +- **Webhook URL:** `https://your-domain.com/api/trading/execute` + +**Required Fields:** +- ✅ `atr` - ATR-based TP/SL system (CRITICAL) +- ✅ `adx` - Trend strength filtering +- ✅ `rsi` - Momentum confirmation +- ✅ `volumeRatio` - Volume validation +- ✅ `pricePosition` - Range position (0-100%) +- ✅ `timeframe` - Distinguishes 5min (trade) vs 15min/1H (data collection) +- ✅ `indicatorVersion` - Track strategy performance (v9 current) + +### **Market Data Alerts (1-minute chart)** + +**Alert Message Format:** +```json +{ + "action": "market_data", + "symbol": "{{ticker}}", + "timeframe": "{{interval}}", + "currentPrice": {{close}}, + "atr": {{ta.atr(14)}}, + "adx": {{ta.dmi(14, 14)}}, + "rsi": {{ta.rsi(14)}}, + "volumeRatio": {{volume / ta.sma(volume, 20)}}, + "pricePosition": {{(close - ta.lowest(low, 100)) / (ta.highest(high, 100) - ta.lowest(low, 100)) * 100}} +} +``` + +**Alert Settings:** +- **Condition:** Time trigger (every bar close) +- **Trigger:** Once Per Bar Close +- **Expiration:** None +- **Webhook URL:** `https://your-domain.com/api/trading/market-data` + +**Purpose:** +- Updates market data cache for Smart Entry Validation Queue +- Provides fresh prices for re-entry validation +- Enables sub-5-minute confirmation thresholds + +--- + +## 🔧 n8n Workflow Setup + +### **Parse Signal Enhanced** + +**Location:** `workflows/trading/parse_signal_enhanced.json` + +**Functions:** +1. **Extract Metrics:** ADX, ATR, RSI, volume, price position +2. **Normalize Timeframe:** "5", "15", "60", "240", "D" +3. **Extract Indicator Version:** "IND:v9" → "v9" +4. **Detect MA Crossovers:** "crossing" keyword → flags +5. **Route to Execute:** Calls `/api/trading/execute` + +**Key Nodes:** +- HTTP Request node → Receives TradingView webhook +- Code node → Parses JSON payload +- Set node → Normalizes field names +- HTTP Request node → Sends to execute endpoint + +### **Market Data Collector** + +**Location:** `workflows/trading/market_data_collector.json` + +**Functions:** +1. Receives 1-minute TradingView alerts +2. Validates symbol format (SOLUSDT → SOL-PERP) +3. Updates market data cache via `/api/trading/market-data` +4. Logs cache update timestamp + +--- + +## 🗄️ Database Setup + +### **Initial Setup (First Time)** +```bash +# 1. Start PostgreSQL container +cd /home/icke/traderv4 +docker compose up -d trading-bot-postgres + +# 2. Generate Prisma client +npx prisma generate + +# 3. Run migrations +DATABASE_URL="postgresql://postgres:your_password@localhost:5432/trading_bot_v4" \ +npx prisma migrate dev + +# 4. Verify tables created +docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c "\\dt" +``` + +### **Schema Updates (After Code Changes)** +```bash +# 1. Generate migration +DATABASE_URL="postgresql://...@localhost:5432/..." \ +npx prisma migrate dev --name describe_changes_here + +# 2. Regenerate client +npx prisma generate + +# 3. Rebuild Docker image (includes new Prisma client) +docker compose build trading-bot + +# 4. Restart container +docker compose up -d --force-recreate trading-bot +``` + +### **Database Backup** +```bash +# Backup to SQL file +docker exec trading-bot-postgres pg_dump -U postgres trading_bot_v4 > backup_$(date +%Y%m%d_%H%M%S).sql + +# Restore from backup +cat backup_file.sql | docker exec -i trading-bot-postgres psql -U postgres -d trading_bot_v4 +``` + +--- + +## 🔐 Environment Variables Setup + +### **Required Variables (.env file)** + +**Trading Configuration:** +```bash +# Drift Protocol (REQUIRED) +DRIFT_WALLET_PRIVATE_KEY="[91,24,...]" # JSON array or base58 +API_SECRET_KEY="your-secret-key-here" + +# Solana RPC (REQUIRED - use Helius!) +SOLANA_RPC_URL="https://mainnet.helius-rpc.com/?api-key=YOUR_KEY" +BACKUP_RPC_URL="https://api.mainnet-beta.solana.com" + +# Database (REQUIRED) +DATABASE_URL="postgresql://postgres:password@trading-bot-postgres:5432/trading_bot_v4" +POSTGRES_PASSWORD="your-secure-password" + +# Telegram (REQUIRED for notifications) +TELEGRAM_BOT_TOKEN="123456789:ABCdefGHIjklMNOpqrsTUVwxyz" +TELEGRAM_CHAT_ID="123456789" +``` + +**Position Sizing:** +```bash +# Per-Symbol Configuration +SOLANA_ENABLED=true +SOLANA_POSITION_SIZE=100 # Percentage (1-100) +SOLANA_LEVERAGE=15 # Leverage multiplier + +ETHEREUM_ENABLED=false +ETHEREUM_POSITION_SIZE=100 +ETHEREUM_LEVERAGE=1 + +# Global Fallback (BTC, etc.) +MAX_POSITION_SIZE_USD=100 +LEVERAGE=1 +``` + +**Quality Thresholds:** +```bash +MIN_SIGNAL_QUALITY_SCORE_LONG=90 # LONG signals +MIN_SIGNAL_QUALITY_SCORE_SHORT=80 # SHORT signals (more permissive) +``` + +**Adaptive Leverage:** +```bash +USE_ADAPTIVE_LEVERAGE=true +HIGH_QUALITY_LEVERAGE=10 # Quality ≥ threshold +LOW_QUALITY_LEVERAGE=5 # Quality < threshold +QUALITY_LEVERAGE_THRESHOLD_LONG=95 # LONG threshold +QUALITY_LEVERAGE_THRESHOLD_SHORT=90 # SHORT threshold +``` + +**See:** `../.github/copilot-instructions.md` Environment Variable Reference for complete list (100+ variables) + +--- + +## 🐳 Docker Setup + +### **Development (docker-compose.dev.yml)** +```bash +# Start all services +docker compose -f docker-compose.dev.yml up -d + +# View logs +docker compose -f docker-compose.dev.yml logs -f trading-bot + +# Stop all services +docker compose -f docker-compose.dev.yml down +``` + +### **Production (docker-compose.yml)** +```bash +# Build and start +docker compose up -d --build trading-bot + +# Restart with force recreate +docker compose up -d --force-recreate trading-bot + +# Check status +docker compose ps +docker logs -f trading-bot-v4 +``` + +### **Telegram Bot (docker-compose.telegram-bot.yml)** +```bash +# Start Telegram bot +docker compose -f docker-compose.telegram-bot.yml up -d + +# View logs +docker logs -f trading-bot-telegram + +# Test command +# Send "long sol" to Telegram bot +``` + +--- + +## 🧪 Testing Setup + +### **Test Trade Execution** +```bash +# Via Settings UI +# 1. Open http://localhost:3001/settings +# 2. Scroll to "Test Trading" section +# 3. Click "Test SOL LONG" or "Test SOL SHORT" +# 4. Monitor docker logs for execution + +# Via Telegram Bot +# Send message: "long sol --force" +# Watch for confirmation message +``` + +### **Verify Position Manager** +```bash +# 1. Execute test trade +# 2. Watch logs for monitoring cycle +docker logs -f trading-bot-v4 | grep -E "Position Manager|TP1|TP2|SL" + +# 3. Check database +docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c \ + "SELECT symbol, direction, entryPrice, currentSize, tp1Hit FROM \"Trade\" WHERE exitReason IS NULL;" +``` + +### **API Health Checks** +```bash +# Drift account health +curl http://localhost:3001/api/drift/account-health + +# Market data cache +curl http://localhost:3001/api/trading/market-data + +# Cluster status +curl http://localhost:3001/api/cluster/status + +# Analytics +curl http://localhost:3001/api/analytics/last-trade +``` + +--- + +## 🛠️ Troubleshooting + +### **Container Won't Start** +```bash +# Check logs for errors +docker logs trading-bot-v4 + +# Common issues: +# - DATABASE_URL incorrect (check connection string) +# - DRIFT_WALLET_PRIVATE_KEY format (JSON array or base58) +# - Port 3000 already in use (check with: lsof -i :3000) +``` + +### **TradingView Alerts Not Received** +```bash +# 1. Check n8n workflow execution log +# 2. Test webhook manually: +curl -X POST http://localhost:5678/webhook/parse-signal \ + -H "Content-Type: application/json" \ + -d '{"action":"buy","symbol":"SOLUSDT","timeframe":"5",...}' + +# 3. Verify webhook URL in TradingView alert settings +# 4. Check firewall allows incoming webhooks +``` + +### **Database Connection Failed** +```bash +# 1. Verify PostgreSQL running +docker ps | grep postgres + +# 2. Test connection +docker exec trading-bot-postgres psql -U postgres -c "SELECT 1" + +# 3. Check DATABASE_URL matches container name +# Runtime: postgresql://postgres:pass@trading-bot-postgres:5432/... +# Prisma CLI: postgresql://postgres:pass@localhost:5432/... +``` + +### **RPC Rate Limiting (429 Errors)** +```bash +# Symptom: Logs show "429 Too Many Requests" +# Fix: Switch to Helius RPC (higher rate limits) +# Update .env: +SOLANA_RPC_URL="https://mainnet.helius-rpc.com/?api-key=YOUR_KEY" + +# Restart container +docker compose restart trading-bot +``` + +--- + +## 📚 Additional Resources + +**External Documentation:** +- Drift Protocol: https://docs.drift.trade +- Pyth Network: https://docs.pyth.network +- Helius RPC: https://docs.helius.dev +- TradingView Alerts: https://www.tradingview.com/support/solutions/43000529348 + +**Internal Documentation:** +- **Main README:** `../README.md` +- **Architecture:** `../architecture/` directory +- **Roadmaps:** `../roadmaps/` directory +- **Common Pitfalls:** `../.github/copilot-instructions.md` + +--- + +See `../README.md` for overall documentation structure. diff --git a/workflows/trading/moneyline_1min_price_feed.pinescript b/workflows/trading/moneyline_1min_price_feed.pinescript new file mode 100644 index 0000000..03698ec --- /dev/null +++ b/workflows/trading/moneyline_1min_price_feed.pinescript @@ -0,0 +1,27 @@ +//@version=6 +indicator("Money Line - 1min Price Feed (Simplified)", overlay=false) + +// ========================================== +// PURPOSE: Lightweight 1-minute price updates for market data cache +// CHANGE (Dec 4, 2025): Removed all indicators to reduce TradingView alert queue pressure +// +// WHY: 60 alerts/hour with full metrics can cause 5-minute trading signals to be skipped +// SOLUTION: Send ONLY price + symbol + timeframe (70% smaller payload) +// +// USAGE: Create alert on indicator with "alert() function calls" +// WEBHOOK: https://flow.egonetix.de/webhook/tradingview-bot-v4 (SAME as trading signals) +// FORMAT: Minimal format - bot recognizes timeframe="1" and routes to data collection +// ========================================== + +// Display current price (visual confirmation alert is working) +plot(close, "Price", color=color.white, linewidth=2) + +// Build MINIMAL JSON message - ONLY essential data +// Format: "SYMBOL buy 1 @ PRICE" +// Bot uses timeframe="1" to route to BlockedSignal (no trade execution) +// Indicators (ATR/ADX/RSI/etc) removed - calculated from DB when needed +jsonMessage = 'SOLUSDT buy 1 @ ' + str.tostring(close) + +// Send alert every bar close (every 1 minute on 1min chart) +if barstate.isconfirmed + alert(jsonMessage, alert.freq_once_per_bar)