docs: Major documentation reorganization + ENV variable reference
**Documentation Structure:** - Created docs/ subdirectory organization (analysis/, architecture/, bugs/, cluster/, deployments/, roadmaps/, setup/, archived/) - Moved 68 root markdown files to appropriate categories - Root directory now clean (only README.md remains) - Total: 83 markdown files now organized by purpose **New Content:** - Added comprehensive Environment Variable Reference to copilot-instructions.md - 100+ ENV variables documented with types, defaults, purpose, notes - Organized by category: Required (Drift/RPC/Pyth), Trading Config (quality/ leverage/sizing), ATR System, Runner System, Risk Limits, Notifications, etc. - Includes usage examples (correct vs wrong patterns) **File Distribution:** - docs/analysis/ - Performance analyses, blocked signals, profit projections - docs/architecture/ - Adaptive leverage, ATR trailing, indicator tracking - docs/bugs/ - CRITICAL_*.md, FIXES_*.md bug reports (7 files) - docs/cluster/ - EPYC setup, distributed computing docs (3 files) - docs/deployments/ - *_COMPLETE.md, DEPLOYMENT_*.md status (12 files) - docs/roadmaps/ - All *ROADMAP*.md strategic planning files (7 files) - docs/setup/ - TradingView guides, signal quality, n8n setup (8 files) - docs/archived/2025_pre_nov/ - Obsolete verification checklist (1 file) **Key Improvements:** - ENV variable reference: Single source of truth for all configuration - Common Pitfalls #68-71: Already complete, verified during audit - Better findability: Category-based navigation vs 68 files in root - Preserves history: All files git mv (rename), not copy/delete - Zero broken functionality: Only documentation moved, no code changes **Verification:** - 83 markdown files now in docs/ subdirectories - Root directory cleaned: 68 files → 0 files (except README.md) - Git history preserved for all moved files - Container running: trading-bot-v4 (no restart needed) **Next Steps:** - Create README.md files in each docs subdirectory - Add navigation index - Update main README.md with new structure - Consolidate duplicate deployment docs - Archive truly obsolete files (old SQL backups) See: docs/analysis/CLEANUP_PLAN.md for complete reorganization strategy
This commit is contained in:
mindesbunister
389
docs/setup/SIGNAL_QUALITY_SETUP_GUIDE.md
Normal file
389
docs/setup/SIGNAL_QUALITY_SETUP_GUIDE.md
Normal file
@@ -0,0 +1,389 @@
|
||||
# Signal Quality Scoring System - Setup Guide
|
||||
|
||||
## Overview
|
||||
|
||||
The signal quality scoring system evaluates every trade signal based on 5 market context metrics before execution. Signals scoring below 91/100 are automatically blocked. This prevents overtrading and filters out low-quality setups.
|
||||
|
||||
**Threshold History:**
|
||||
- Nov 21 morning: Raised to 81 after v8 initial success (94.2 avg quality, 66.7% WR)
|
||||
- Nov 21 evening: Raised to 91 after trade #7 loss (ADX 19.0, quality 90, -$387)
|
||||
|
||||
**Data Validation (7 v8 trades):**
|
||||
Perfect quality score separation validates 91 threshold:
|
||||
- **ALL 4 winners:** Quality ≥95 (scores: 95, 95, 100, 105) ✅
|
||||
- **ALL 3 losers:** Quality ≤90 (scores: 80, 90, 90) ❌
|
||||
- **Conclusion:** 91 threshold would have prevented 100% of losses (-$624.90 total)
|
||||
|
||||
## ✅ Completed Components
|
||||
|
||||
### 1. TradingView Indicator ✅
|
||||
- **File:** `workflows/trading/moneyline_v5_final.pinescript`
|
||||
- **Status:** Complete and tested
|
||||
- **Metrics sent:** ATR%, ADX, RSI, Volume Ratio, Price Position
|
||||
- **Alert format:** `SOL buy .P 15 | ATR:1.85 | ADX:28.3 | RSI:62.5 | VOL:1.45 | POS:75.3`
|
||||
|
||||
### 2. n8n Parse Signal Enhanced ✅
|
||||
- **File:** `workflows/trading/parse_signal_enhanced.json`
|
||||
- **Status:** Complete and tested
|
||||
- **Function:** Extracts 5 context metrics from alert messages
|
||||
- **Backward compatible:** Works with old format (metrics default to 0)
|
||||
|
||||
### 3. Trading Bot API ✅
|
||||
- **check-risk endpoint:** Scores signals 0-100, blocks if <60
|
||||
- **execute endpoint:** Stores context metrics in database
|
||||
- **Database schema:** Updated with 5 new fields
|
||||
- **Status:** Built, deployed, running
|
||||
|
||||
## 📋 n8n Workflow Update Instructions
|
||||
|
||||
### Step 1: Import Parse Signal Enhanced Node
|
||||
|
||||
1. Open n8n workflow editor
|
||||
2. Go to "Money Machine" workflow
|
||||
3. Click the "+" icon to add a new node
|
||||
4. Select "Code" → "Import from file"
|
||||
5. Import: `/home/icke/traderv4/workflows/trading/parse_signal_enhanced.json`
|
||||
|
||||
### Step 2: Replace Old Parse Signal Node
|
||||
|
||||
**Old Node (lines 23-52 in Money_Machine.json):**
|
||||
```json
|
||||
{
|
||||
"parameters": {
|
||||
"fields": {
|
||||
"values": [
|
||||
{
|
||||
"name": "rawMessage",
|
||||
"stringValue": "={{ $json.body }}"
|
||||
},
|
||||
{
|
||||
"name": "symbol",
|
||||
"stringValue": "={{ $json.body.match(/\\bSOL\\b/i) ? 'SOL-PERP' : ... }}"
|
||||
},
|
||||
{
|
||||
"name": "direction",
|
||||
"stringValue": "={{ $json.body.match(/\\b(sell|short)\\b/i) ? 'short' : 'long' }}"
|
||||
},
|
||||
{
|
||||
"name": "timeframe",
|
||||
"stringValue": "={{ $json.body.match(/\\.P\\s+(\\d+)/)?.[1] || '15' }}"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"name": "Parse Signal",
|
||||
"type": "n8n-nodes-base.set"
|
||||
}
|
||||
```
|
||||
|
||||
**New Node (Parse Signal Enhanced):**
|
||||
- Extracts: symbol, direction, timeframe (same as before)
|
||||
- NEW: Also extracts ATR, ADX, RSI, volumeRatio, pricePosition
|
||||
- Place after the "Webhook" node
|
||||
- Connect: Webhook → Parse Signal Enhanced → 15min Chart Only?
|
||||
|
||||
### Step 3: Update Check Risk Node
|
||||
|
||||
**Current jsonBody (line 103):**
|
||||
```json
|
||||
{
|
||||
"symbol": "{{ $json.symbol }}",
|
||||
"direction": "{{ $json.direction }}"
|
||||
}
|
||||
```
|
||||
|
||||
**Updated jsonBody (add 5 context metrics):**
|
||||
```json
|
||||
{
|
||||
"symbol": "{{ $json.symbol }}",
|
||||
"direction": "{{ $json.direction }}",
|
||||
"atr": {{ $json.atr || 0 }},
|
||||
"adx": {{ $json.adx || 0 }},
|
||||
"rsi": {{ $json.rsi || 0 }},
|
||||
"volumeRatio": {{ $json.volumeRatio || 0 }},
|
||||
"pricePosition": {{ $json.pricePosition || 0 }}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 4: Update Execute Trade Node
|
||||
|
||||
**Current jsonBody (line 157):**
|
||||
```json
|
||||
{
|
||||
"symbol": "{{ $('Parse Signal').item.json.symbol }}",
|
||||
"direction": "{{ $('Parse Signal').item.json.direction }}",
|
||||
"timeframe": "{{ $('Parse Signal').item.json.timeframe }}",
|
||||
"signalStrength": "strong"
|
||||
}
|
||||
```
|
||||
|
||||
**Updated jsonBody (add 5 context metrics):**
|
||||
```json
|
||||
{
|
||||
"symbol": "{{ $('Parse Signal Enhanced').item.json.symbol }}",
|
||||
"direction": "{{ $('Parse Signal Enhanced').item.json.direction }}",
|
||||
"timeframe": "{{ $('Parse Signal Enhanced').item.json.timeframe }}",
|
||||
"signalStrength": "strong",
|
||||
"atr": {{ $('Parse Signal Enhanced').item.json.atr || 0 }},
|
||||
"adx": {{ $('Parse Signal Enhanced').item.json.adx || 0 }},
|
||||
"rsi": {{ $('Parse Signal Enhanced').item.json.rsi || 0 }},
|
||||
"volumeRatio": {{ $('Parse Signal Enhanced').item.json.volumeRatio || 0 }},
|
||||
"pricePosition": {{ $('Parse Signal Enhanced').item.json.pricePosition || 0 }}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 5: Update Telegram Notification (Optional)
|
||||
|
||||
You can add quality score to Telegram messages:
|
||||
|
||||
**Current message template (line 200):**
|
||||
```
|
||||
🟢 TRADE OPENED
|
||||
|
||||
📊 Symbol: ${symbol}
|
||||
📈 Direction: ${direction}
|
||||
...
|
||||
```
|
||||
|
||||
**Enhanced message template:**
|
||||
```
|
||||
🟢 TRADE OPENED
|
||||
|
||||
📊 Symbol: ${symbol}
|
||||
📈 Direction: ${direction}
|
||||
🎯 Quality Score: ${$('Check Risk').item.json.qualityScore || 'N/A'}/100
|
||||
...
|
||||
```
|
||||
|
||||
## 🧪 Testing Instructions
|
||||
|
||||
### Test 1: High-Quality Signal (Should Execute)
|
||||
|
||||
Send webhook:
|
||||
```bash
|
||||
curl -X POST http://localhost:5678/webhook/tradingview-bot-v4 \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"body": "SOL buy .P 15 | ATR:1.85 | ADX:32.3 | RSI:58.5 | VOL:1.65 | POS:45.3"}'
|
||||
```
|
||||
|
||||
**Expected:**
|
||||
- Parse Signal Enhanced extracts all 5 metrics
|
||||
- Check Risk calculates quality score ~80/100
|
||||
- Check Risk returns `passed: true`
|
||||
- Execute Trade runs and stores metrics in database
|
||||
- Telegram notification sent
|
||||
|
||||
### Test 2: Low-Quality Signal (Should Block)
|
||||
|
||||
Send webhook:
|
||||
```bash
|
||||
curl -X POST http://localhost:5678/webhook/tradingview-bot-v4 \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"body": "SOL buy .P 15 | ATR:0.35 | ADX:12.8 | RSI:78.5 | VOL:0.45 | POS:92.1"}'
|
||||
```
|
||||
|
||||
**Expected:**
|
||||
- Parse Signal Enhanced extracts all 5 metrics
|
||||
- Check Risk calculates quality score ~20/100
|
||||
- Check Risk returns `passed: false, reason: "Signal quality too low (20/100). Issues: ATR too low (chop/low volatility), Weak/no trend (ADX), RSI extreme vs direction, Volume too low, Chasing (long near range top)"`
|
||||
- Execute Trade does NOT run
|
||||
- Telegram error notification sent
|
||||
|
||||
### Test 3: Backward Compatibility (Should Execute)
|
||||
|
||||
Send old format without metrics:
|
||||
```bash
|
||||
curl -X POST http://localhost:5678/webhook/tradingview-bot-v4 \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"body": "SOL buy .P 15"}'
|
||||
```
|
||||
|
||||
**Expected:**
|
||||
- Parse Signal Enhanced extracts symbol/direction/timeframe, metrics default to 0
|
||||
- Check Risk skips quality scoring (ATR=0 means no metrics)
|
||||
- Check Risk returns `passed: true` (only checks risk limits)
|
||||
- Execute Trade runs with null metrics
|
||||
- Backward compatible
|
||||
|
||||
## 📊 Scoring Logic
|
||||
|
||||
### Scoring Breakdown (Base: 50 points)
|
||||
|
||||
1. **ATR Check** (-15 to +10 points)
|
||||
- ATR < 0.6%: -15 (choppy/low volatility)
|
||||
- ATR > 2.5%: -20 (extreme volatility)
|
||||
- 0.6-2.5%: +10 (healthy)
|
||||
|
||||
2. **ADX Check** (-15 to +15 points)
|
||||
- ADX > 25: +15 (strong trend)
|
||||
- ADX 18-25: +5 (moderate trend)
|
||||
- ADX < 18: -15 (weak/no trend)
|
||||
|
||||
3. **RSI Check** (-10 to +10 points)
|
||||
- Long + RSI > 50: +10 (momentum supports)
|
||||
- Long + RSI < 30: -10 (extreme oversold)
|
||||
- Short + RSI < 50: +10 (momentum supports)
|
||||
- Short + RSI > 70: -10 (extreme overbought)
|
||||
|
||||
4. **Volume Check** (-10 to +10 points)
|
||||
- Volume > 1.2x avg: +10 (strong participation)
|
||||
- Volume < 0.8x avg: -10 (low participation)
|
||||
- 0.8-1.2x avg: 0 (neutral)
|
||||
|
||||
5. **Price Position Check** (-15 to +5 points)
|
||||
- Long at range top (>80%): -15 (chasing)
|
||||
- Short at range bottom (<20%): -15 (chasing)
|
||||
- Otherwise: +5 (good position)
|
||||
|
||||
**Minimum Passing Score:** 60/100
|
||||
|
||||
### Example Scores
|
||||
|
||||
**Perfect Setup (Score: 90):**
|
||||
- ATR: 1.5% (+10)
|
||||
- ADX: 32 (+15)
|
||||
- RSI: 58 (long) (+10)
|
||||
- Volume: 1.8x (+10)
|
||||
- Price: 45% (+5)
|
||||
- **Total:** 50 + 10 + 15 + 10 + 10 + 5 = 90
|
||||
|
||||
**Terrible Setup (Score: 20):**
|
||||
- ATR: 0.3% (-15)
|
||||
- ADX: 12 (-15)
|
||||
- RSI: 78 (long) (-10)
|
||||
- Volume: 0.5x (-10)
|
||||
- Price: 92% (-15)
|
||||
- **Total:** 50 - 15 - 15 - 10 - 10 - 15 = -5 → Clamped to 0
|
||||
|
||||
## 🔍 Monitoring
|
||||
|
||||
### Check Logs
|
||||
|
||||
Watch check-risk decisions:
|
||||
```bash
|
||||
docker logs trading-bot-v4 --tail 100 -f | grep "Signal quality"
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
✅ Signal quality: 75/100 - HIGH QUALITY
|
||||
🎯 Quality reasons: Strong trend (ADX: 32.3), Healthy volatility (ATR: 1.85%), Good volume (1.65x avg), RSI supports direction (58.5), Good entry position (45.3%)
|
||||
```
|
||||
|
||||
```
|
||||
❌ Signal quality: 35/100 - TOO LOW (minimum: 60)
|
||||
⚠️ Quality reasons: Weak/no trend (ADX: 12.8), ATR too low (chop/low volatility), RSI extreme vs direction, Volume too low, Chasing (long near range top)
|
||||
```
|
||||
|
||||
### Database Query
|
||||
|
||||
Check stored metrics:
|
||||
```sql
|
||||
SELECT
|
||||
symbol,
|
||||
direction,
|
||||
entryPrice,
|
||||
atrAtEntry,
|
||||
adxAtEntry,
|
||||
rsiAtEntry,
|
||||
volumeAtEntry,
|
||||
pricePositionAtEntry,
|
||||
realizedPnL
|
||||
FROM "Trade"
|
||||
WHERE createdAt > NOW() - INTERVAL '7 days'
|
||||
ORDER BY createdAt DESC;
|
||||
```
|
||||
|
||||
## 🎛️ Tuning Parameters
|
||||
|
||||
All scoring thresholds are in `app/api/trading/check-risk/route.ts` (lines 210-320):
|
||||
|
||||
```typescript
|
||||
// ATR thresholds
|
||||
if (atr < 0.6) points -= 15 // Too low
|
||||
if (atr > 2.5) points -= 20 // Too high
|
||||
|
||||
// ADX thresholds
|
||||
if (adx > 25) points += 15 // Strong trend
|
||||
if (adx < 18) points -= 15 // Weak trend
|
||||
|
||||
// Minimum passing score
|
||||
if (score < 60) {
|
||||
return { passed: false, ... }
|
||||
}
|
||||
```
|
||||
|
||||
Adjust these based on backtesting results. For example:
|
||||
- If too many good trades blocked: Lower minimum score to 50
|
||||
- If still overtrading: Increase ADX threshold to 30
|
||||
- For different assets: Adjust ATR ranges (crypto vs stocks)
|
||||
|
||||
## 📈 Next Steps
|
||||
|
||||
1. **Deploy to Production:**
|
||||
- Update n8n workflow (Steps 1-5 above)
|
||||
- Test with both formats
|
||||
- Monitor logs for quality decisions
|
||||
|
||||
2. **Collect Data:**
|
||||
- Run for 2 weeks to gather quality scores
|
||||
- Analyze correlation: quality score vs P&L
|
||||
- Identify which metrics matter most
|
||||
|
||||
3. **Optimize:**
|
||||
- Query database: `SELECT AVG(realizedPnL) FROM Trade WHERE adxAtEntry > 25`
|
||||
- Fine-tune thresholds based on results
|
||||
- Consider dynamic scoring (different weights per symbol/timeframe)
|
||||
|
||||
4. **Future Enhancements:**
|
||||
- Add more metrics (spread, funding rate, correlation)
|
||||
- Machine learning: Train on historical trades
|
||||
- Per-asset scoring models
|
||||
- Signal source scoring (TradingView vs manual)
|
||||
|
||||
## 🚨 Troubleshooting
|
||||
|
||||
**Problem:** All signals blocked
|
||||
- Check logs: `docker logs trading-bot-v4 | grep "quality"`
|
||||
- Likely: TradingView not sending metrics (verify alert format)
|
||||
- Workaround: Temporarily lower minimum score to 40
|
||||
|
||||
**Problem:** No metrics in database
|
||||
- Check Parse Signal Enhanced extracted metrics: View n8n execution
|
||||
- Verify Check Risk received metrics: `curl localhost:3001/api/trading/check-risk` with test data
|
||||
- Check execute endpoint logs: Should show "Context metrics: ATR:..."
|
||||
|
||||
**Problem:** Metrics always 0
|
||||
- TradingView alert not using enhanced indicator
|
||||
- Parse Signal Enhanced regex not matching
|
||||
- Test parsing: `node -e "console.log('SOL buy .P 15 | ATR:1.85'.match(/ATR:([\d.]+)/))"`
|
||||
|
||||
## 📝 Files Modified
|
||||
|
||||
- ✅ `workflows/trading/moneyline_v5_final.pinescript` - Enhanced indicator
|
||||
- ✅ `workflows/trading/parse_signal_enhanced.json` - n8n parser
|
||||
- ✅ `app/api/trading/check-risk/route.ts` - Quality scoring
|
||||
- ✅ `app/api/trading/execute/route.ts` - Store metrics
|
||||
- ✅ `lib/database/trades.ts` - Updated interface
|
||||
- ✅ `prisma/schema.prisma` - Added 5 fields
|
||||
- ✅ `prisma/migrations/...add_rsi_and_price_position_metrics/` - Migration
|
||||
- ⏳ `workflows/trading/Money_Machine.json` - Manual update needed
|
||||
|
||||
## 🎯 Success Criteria
|
||||
|
||||
Signal quality scoring is working correctly when:
|
||||
|
||||
1. ✅ TradingView sends alerts with metrics
|
||||
2. ✅ n8n Parse Signal Enhanced extracts all 5 metrics
|
||||
3. ✅ Check Risk calculates quality score 0-100
|
||||
4. ✅ Low-quality signals (<60) are blocked with reasons
|
||||
5. ✅ High-quality signals (>60) execute normally
|
||||
6. ✅ Context metrics stored in database for every trade
|
||||
7. ✅ Backward compatible with old alerts (metrics=0, scoring skipped)
|
||||
8. ✅ Logs show quality score and reasons for every signal
|
||||
|
||||
---
|
||||
|
||||
**Status:** Ready for production testing
|
||||
**Last Updated:** 2024-10-30
|
||||
**Author:** Trading Bot v4 Signal Quality System
|
||||
Reference in New Issue
Block a user