trading_bot_v4/docs/1MIN_SIGNAL_OPTIMIZATION.md

# 1-Minute Signal Optimization & TradingView Webhook Flooding Fix

**Date:** December 4, 2025  
**Problem:** 1-minute signals flooding TradingView webhook, causing 5-minute trading signals to be skipped  
**Impact:** CRITICAL - Missed real trading opportunities (2 incidents confirmed)

---

## Root Cause Analysis

### Current System Behavior

**1-Minute Data Collection:**
- Frequency: 60 signals/hour per symbol = 180 signals/hour total (SOL/ETH/BTC)
- Payload size: Full indicator data (ATR, ADX, RSI, VOL, POS, MAGAP, IND, price)
- Purpose: Market data cache updates + BlockedSignal table analysis
- Database: ~1,408 records per 24 hours (confirmed via SQL query)

**5-Minute Trading Signals:**
- Frequency: 12 signals/hour per symbol max = 36 signals/hour total
- Purpose: Execute real trades (primary revenue source)
- Problem: **Getting skipped when 1-minute flood overwhelms TradingView webhook**

### TradingView Webhook Rate Limiting

TradingView alerts have undocumented rate limiting on webhook delivery:
- **Symptom:** When multiple alerts fire rapidly, some webhooks get dropped
- **Pattern:** High-frequency alerts (1-minute) cause lower-frequency alerts (5-minute) to be skipped
- **Impact:** Trading signals lost = missed trades = missed profits

### Why 1-Minute Signals Are Excessive

**Current usage vs actual needs:**

| Data Point | Current Frequency | Actually Needed For | Optimal Frequency |
|------------|-------------------|---------------------|-------------------|
| ATR | Every 1 minute | Smart Entry validation, trailing stops | Every 5 minutes |
| ADX | Every 1 minute | Runner SL, adaptive trailing | Every 5 minutes |
| RSI | Every 1 minute | Quality scoring (5min signals only) | Every 5 minutes |
| Volume | Every 1 minute | Quality scoring (5min signals only) | Every 5 minutes |
| Price Position | Every 1 minute | Quality scoring (5min signals only) | Every 5 minutes |
| MA Gap | Every 1 minute | V9 indicator scoring | Every 5 minutes |

**Key insight:** ALL market data is used for 5-minute signal validation. 1-minute granularity doesn't provide value since we only trade on 5-minute signals.

---

## Proposed Solution: Reduce 1-Minute Frequency OR Eliminate

### Option 1: Reduce to Every 5 Minutes (RECOMMENDED)

**Change 1-minute alerts to fire every 5 minutes instead:**
- Frequency: 12 signals/hour → Same as trading signals
- Data: Keep all indicators (ATR, ADX, RSI, etc.)
- Benefit: 83% reduction in webhook load (180/hour → 36/hour)
- Trade-off: None - market data still fresh for 5-minute trading signals

**Implementation:**
```pinescript
// In moneyline_1min_data_feed.pinescript:
// OLD:
if barstate.isconfirmed  // Fires every 1-minute candle

// NEW:
if barstate.isconfirmed and bar_index % 5 == 0  // Fires every 5 minutes
```

**Alert setup:**
- Keep chart on 1-minute timeframe
- Alert fires every 5th candle close = every 5 minutes
- Webhook URL: Same (tradingview-bot-v4)
- Format: Same trading signal format

### Option 2: Switch to Price-Only WebSocket (ALTERNATIVE)

**Replace TradingView alerts with Pyth WebSocket price feed:**
- Frequency: Real-time price updates (millisecond granularity)
- Data: Price only (no indicators)
- Benefit: 100% reduction in TradingView webhook load
- Trade-off: Lose indicator data (ADX/ATR/RSI) unless calculated bot-side

**Current system already has Pyth:**
- `lib/pyth/price-monitor.ts` - Real-time price WebSocket
- Used by Position Manager for TP/SL monitoring
- Provides price data every 400-800ms typical

**Indicators would need bot-side calculation:**
- ATR: Requires 14 candles of high/low/close data
- ADX: Requires 14 candles of directional movement
- RSI: Requires 14 candles of close prices
- Complexity: Moderate (need candle history management)

### Option 3: Eliminate 1-Minute Data Collection (SIMPLEST)

**Remove 1-minute alerts entirely:**
- Frequency: 0 signals/hour → 100% reduction
- Data: Use 5-minute signal data for cache
- Benefit: Zero webhook flood, zero complexity
- Trade-off: Market data cache only updates when 5-minute trading signals fire

**Impact analysis:**
```typescript
// Current cache update sources:
1. Execute endpoint auto-caches on every signal (line 104-118)
2. 1-minute alerts refresh cache 12× per hour
3. 5-minute trading signals refresh cache ~1-3× per hour

// After removing 1-minute alerts:
- Cache still updates from 5-minute trading signals
- Worst case: 5 minutes stale data between signals
- Acceptable: Smart Entry validation uses signal data, not cache
```

---

## Recommended Approach

**Phase 1: Immediate (Today)**
1. **Pause all 1-minute alerts in TradingView** (SOL/ETH/BTC)
2. Verify 5-minute signals start working consistently
3. Monitor for 24 hours - confirm no webhook skipping

**Phase 2: Evaluate Need (This Week)**
1. Review where 1-minute data is actually used:
   - Smart Entry Timer: Uses signal data at entry time (not cache)
   - Position Manager: Uses Pyth WebSocket for price (not cache)
   - Adaptive trailing: Uses signal ADX (not cache)
   - Quality scoring: Uses signal data (not cache)
2. Conclusion: **1-minute cache updates provide zero value**

**Phase 3: Long-Term (Optional)**
If bull/bear 1-minute trading proves viable:
1. Create separate lightweight webhook ONLY for 1-minute trades
2. Minimal payload: Symbol, direction, timeframe
3. No indicator data (calculate bot-side or accept lower quality)
4. Trade execution on 1-minute = different strategy than 5-minute

---

## Implementation Steps

### Step 1: Pause 1-Minute Alerts (Immediate)

**In TradingView:**
1. Go to Alert panel
2. Find: "SOL-PERP 1min Data Feed"
3. Click pause icon (do NOT delete - may need later)
4. Repeat for ETH-PERP and BTC-PERP

**Expected result:** 
- 180 webhooks/hour → 36 webhooks/hour (83% reduction)
- 5-minute signals start arriving consistently

### Step 2: Monitor 5-Minute Signal Delivery (24 hours)

**SQL verification:**
```sql
-- Check 5-minute signals received in last 24 hours
SELECT 
  symbol, 
  COUNT(*) as signals_received,
  COUNT(CASE WHEN "blockReason" = 'QUALITY_SCORE_TOO_LOW' THEN 1 END) as blocked_low_quality,
  COUNT(CASE WHEN "blockReason" = 'DATA_COLLECTION_ONLY' THEN 1 END) as data_collection
FROM "BlockedSignal"
WHERE timeframe = '5'
  AND "createdAt" > NOW() - INTERVAL '24 hours'
GROUP BY symbol;

-- Check actual trades executed
SELECT 
  symbol,
  COUNT(*) as trades_executed
FROM "Trade"
WHERE "createdAt" > NOW() - INTERVAL '24 hours'
  AND timeframe = '5'
GROUP BY symbol;
```

**Expected healthy baseline:**
- 5-minute signals: 36-288 per 24 hours (3 symbols × 12-96 per symbol)
- Some blocked for quality, some executed as trades
- Zero data collection signals (timeframe=5 always attempts execution)

### Step 3: Clean Up Database (Optional)

**If 1-minute data collection proven unnecessary:**
```sql
-- Archive old 1-minute data to separate table
CREATE TABLE "ArchivedBlockedSignal" AS 
SELECT * FROM "BlockedSignal" WHERE timeframe = '1';

-- Delete from main table to free space
DELETE FROM "BlockedSignal" WHERE timeframe = '1';

-- Result: ~1,408 records removed per day = ~10 MB/day savings
```

### Step 4: Update Documentation

**Files to update:**
- `.github/copilot-instructions.md` - Remove 1-minute data collection section
- `docs/1MIN_DATA_COLLECTION_SIMPLE.md` - Mark as deprecated
- `docs/1MIN_MARKET_DATA_IMPLEMENTATION.md` - Add note about TradingView flooding issue
- `workflows/trading/moneyline_1min_data_feed.pinescript` - Add warning comment

---

## Alternative: Minimal 1-Minute Bull/Bear Signals

**If you want to trade on 1-minute timeframe:**

### Separate Lightweight Webhook

**Create dedicated 1-minute trading alerts (NOT data collection):**
```pinescript
//@version=6
indicator("Money Line - 1min TRADING", overlay=true)

// Minimal logic - only when strong signal
atr = ta.atr(14)
[diPlus, diMinus, adx] = ta.dmi(14, 14)
rsi = ta.rsi(close, 14)

// Only alert on strong conditions (reduce frequency)
strongBuy = rsi < 30 and adx > 25
strongSell = rsi > 70 and adx > 25

if barstate.isconfirmed and strongBuy
    alert('SOLUSDT buy 1', alert.freq_once_per_bar)
    
if barstate.isconfirmed and strongSell
    alert('SOLUSDT sell 1', alert.freq_once_per_bar)
```

**Key differences:**
- Fires only on strong conditions (not every candle)
- Minimal payload (symbol, direction, timeframe)
- Separate from data collection
- Uses same webhook but distinguishable by timeframe=1 + strongCondition

### Bot-Side Changes for 1-Minute Trading

**In execute endpoint (route.ts):**
```typescript
// Allow 1-minute trading signals through
if (timeframe === '1' && qualityResult.score >= 90) {
  console.log('✅ 1-minute TRADING signal (high quality) - executing')
  // Proceed to trade execution
} else if (timeframe !== '5') {
  console.log('📊 DATA COLLECTION or low quality - skip execution')
  // Save to BlockedSignal
}
```

**Risk management for 1-minute trades:**
- Higher quality threshold (95+ instead of 90+)
- Smaller position size (50% of 5-minute size)
- Tighter stops (ATR × 1.5 instead of × 3.0)
- Different runner system (faster TP1 at 0.5%)

---

## Decision Matrix

| Option | Webhook Load | Data Quality | Complexity | Best For |
|--------|-------------|--------------|------------|----------|
| **Pause 1-min alerts** | ✅ 83% reduction | ✅ Same (uses 5-min data) | ✅ Zero changes | Current 5-min strategy |
| **5-min intervals** | ⚠️ 83% reduction | ✅ Same | ⚠️ Alert modification | Future flexibility |
| **Eliminate entirely** | ✅ 100% reduction | ✅ Same | ✅ Delete alerts | Simplification |
| **1-min trading** | ❌ Same load | ⚠️ Lower timeframe risk | ❌ High (new strategy) | Different strategy |

**Recommendation: Pause 1-min alerts** (Option 1)
- Solves immediate problem (missed 5-min signals)
- Zero code changes needed
- Reversible if needed later
- Can always enable 1-min trading as separate strategy

---

## Expected Outcomes

### Immediate Benefits
- ✅ 5-minute trading signals arrive consistently
- ✅ No more missed trade opportunities
- ✅ Reduced n8n workflow load (180 → 36 executions/hour)
- ✅ Cleaner logs (no 1-minute spam)
- ✅ Faster BlockedSignal table queries

### Long-Term Benefits
- 📊 Simpler system (fewer moving parts)
- 💾 Reduced database growth (10 MB/day savings)
- 🎯 Focus on proven 5-minute strategy
- 🔬 Can add 1-minute trading later as separate strategy

### No Downsides
- ✅ Smart Entry validation uses signal data (not cache)
- ✅ Position Manager uses Pyth WebSocket (not cache)
- ✅ Quality scoring uses signal data (not cache)
- ✅ Adaptive trailing uses signal ADX (not cache)

**Conclusion: 1-minute data collection provides zero value and actively harms system by flooding webhooks.**

---

## Implementation Timeline

### Today (December 4, 2025)
- [ ] Pause all 3 1-minute TradingView alerts
- [ ] Monitor next 5-minute signal (should arrive on time)
- [ ] Document in commit: "fix: Pause 1-minute alerts to prevent webhook flooding"

### This Week
- [ ] Verify 24 hours of consistent 5-minute signals
- [ ] SQL analysis: Compare signal delivery before/after
- [ ] Update documentation files
- [ ] Archive 1-minute BlockedSignal data (optional)

### Next Month (If Needed)
- [ ] Evaluate 1-minute bull/bear trading viability
- [ ] Design separate lightweight 1-minute strategy
- [ ] Create dedicated webhook for 1-minute trades
- [ ] Test with small position sizes

---

## Rollback Plan

**If pausing 1-minute alerts causes unexpected issues:**
1. Re-enable 1-minute alerts in TradingView
2. Check logs for what broke (cache dependencies?)
3. Fix actual dependency instead of flooding webhooks
4. Document the real reason 1-minute data was needed

**Unlikely scenarios that would require rollback:**
- Smart Entry Timer depends on cache refresh (no - uses signal data)
- Position Manager depends on cache (no - uses Pyth WebSocket)
- Quality scoring depends on cache (no - uses signal data)
- Adaptive trailing depends on cache (no - uses signal ADX)

**Reality check:** System was working perfectly before November 27, 2025 (when 1-minute alerts added). Removing them returns to proven stable state.

---

## Files to Update

### Immediate Changes (Code)
- None required - just pause TradingView alerts

### Documentation Updates
1. `.github/copilot-instructions.md`:
   - Remove/update "1-Minute Data Collection System" section
   - Add this incident to Common Pitfalls
   - Document webhook flooding issue

2. `docs/1MIN_DATA_COLLECTION_SIMPLE.md`:
   - Add warning about TradingView webhook flooding
   - Mark as deprecated/paused
   - Link to this document

3. `docs/1MIN_MARKET_DATA_IMPLEMENTATION.md`:
   - Add post-mortem section
   - Explain why 1-minute data collection was removed

4. `workflows/trading/moneyline_1min_data_feed.pinescript`:
   - Add comment about webhook flooding issue
   - Note: "DEPRECATED - Causes TradingView to skip 5-minute signals"

### Git Commits
```bash
# Commit 1: Documentation
git add docs/1MIN_SIGNAL_OPTIMIZATION.md
git commit -m "docs: Add 1-minute signal optimization analysis"

# Commit 2: Update copilot instructions
git add .github/copilot-instructions.md
git commit -m "docs: Document 1-minute webhook flooding issue (Common Pitfall)"

# Commit 3: Mark indicator deprecated
git add workflows/trading/moneyline_1min_data_feed.pinescript
git commit -m "docs: Mark 1-minute data feed as deprecated (webhook flooding)"

git push
```

---

## Key Takeaway

**TradingView webhook flooding is a REAL problem:**
- High-frequency alerts (1-minute) cause lower-frequency alerts (5-minute) to be dropped
- This is undocumented behavior but confirmed by 2 incidents
- Solution: Reduce webhook frequency OR use alternative data source (Pyth WebSocket)

**1-minute market data cache updates provide ZERO value:**
- All critical systems use signal data or Pyth WebSocket (not cache)
- Cache updates are a vestigial feature from early development
- Removing 1-minute alerts simplifies system without losing functionality

**If you want to trade 1-minute signals later:**
- Design as separate strategy (not data collection)
- Use dedicated webhook with strong filters
- Accept trade-offs (higher risk, smaller positions, faster stops)