# 1-Minute Market Data Collection - Implementation Plan

**Status:** READY TO IMPLEMENT - Zero Cost! (Nov 27, 2025)  
**Purpose:** Enable real-time ADX validation for revenge system (Enhancement #1)  
**Cost:** $0/month - No TradingView upgrade needed ✅

**GOOD NEWS:** Alert limits are for ACTIVE alerts (slots), not trigger frequency!
- Current: 4/20 alert slots used
- Needed: 3 slots (SOL/ETH/BTC)
- Result: 13 slots still free after implementation

---

## Current State

**TradingView Alerts:**
- Send data ONLY when trend changes (bullish/bearish transitions)
- Timeframes: 5min, 15min, 1H, Daily
- Problem: No fresh data between trend changes = stale cache for revenge ADX checks

**Market Data Cache:**
- 5-minute expiry
- Fields: ADX, ATR, RSI, volumeRatio, pricePosition, currentPrice
- Used by: Re-entry analytics, soon revenge system

---

## Proposed Solution: 1-Minute Market Data Alerts

### Database Impact Analysis

**Current schema:**
```sql
-- No separate table, data flows through cache only
-- Cache: In-memory Map, expires after 5 minutes
-- Zero database storage currently
```

**Proposed: Add MarketDataSnapshot table** (OPTIONAL)
```sql
model MarketDataSnapshot {
  id               String   @id @default(cuid())
  timestamp        DateTime @default(now())
  symbol           String
  timeframe        String   // "1", "5", "15", "60", "D"
  
  -- Metrics
  atr              Float
  adx              Float
  rsi              Float
  volumeRatio      Float
  pricePosition    Float
  currentPrice     Float
  
  -- Metadata
  indicatorVersion String?  // e.g., "v9"
  
  @@index([symbol, timestamp])
  @@index([symbol, timeframe, timestamp])
}
```

**Storage Calculation:**
```
Per record: ~150 bytes (8 fields × 8-20 bytes each)
Per minute: 150 bytes × 1 symbol = 150 bytes
Per hour: 150 × 60 = 9 KB
Per day: 9 KB × 24 = 216 KB
Per month: 216 KB × 30 = 6.48 MB
Per year: 6.48 MB × 12 = 77.76 MB

With 3 symbols (SOL, ETH, BTC):
Per month: 6.48 MB × 3 = 19.44 MB
Per year: 77.76 MB × 3 = 233.28 MB

CONCLUSION: Negligible storage impact (<1 GB/year for 3 symbols)
```

---

## Implementation Options

### Option A: Cache-Only (RECOMMENDED for MVP)

**Pros:**
- ✅ Zero database changes required
- ✅ Existing infrastructure (market-data-cache.ts)
- ✅ Fast implementation (just TradingView alert)
- ✅ No storage overhead

**Cons:**
- ❌ No historical analysis (can't backtest ADX patterns)
- ❌ Lost on container restart (cache clears)
- ❌ 5-minute window only (recent data)

**Use Case:**
- Revenge system ADX validation (real-time only)
- Re-entry analytics (already working)
- Good for Phase 1 validation

**Implementation Steps:**
1. Create TradingView 1-minute alert (15 lines of Pine Script)
2. Point to existing `/api/trading/market-data` endpoint
3. Cache handles the rest automatically
4. Test with revenge system

---

### Option B: Cache + Database Persistence

**Pros:**
- ✅ Historical analysis (backtest ADX-based filters)
- ✅ Survives container restarts
- ✅ Enables future ML models (train on historical patterns)
- ✅ Audit trail for debugging

**Cons:**
- ❌ Requires schema migration
- ❌ Need cleanup policy (auto-delete old data)
- ❌ Slightly more complex

**Use Case:**
- Long-term data science projects
- Pattern recognition (what ADX patterns precede stop hunts?)
- System optimization with historical validation

**Implementation Steps:**
1. Add MarketDataSnapshot model to schema
2. Update `/api/trading/market-data` to save to DB
3. Add cleanup job (delete data >30 days old)
4. Create TradingView 1-minute alert
5. Build analytics queries

---

## Recommended Approach: Hybrid (Start with A, Add B Later)

### Phase 1: Cache-Only (This Week)
```
1. Create 1-min TradingView alert
2. Point to /api/trading/market-data
3. Test revenge ADX validation
4. Monitor cache hit rate
5. Validate revenge outcomes improve
```

### Phase 2: Add Persistence (After 10+ Revenge Trades)
```
1. Add MarketDataSnapshot table
2. Save historical data
3. Backtest: "Would ADX filter have helped?"
4. Optimize thresholds based on data
```

---

## TradingView Alert Setup (1-Minute Data)

### Pine Script Code
```pinescript
//@version=5
indicator("Market Data Feed (1min)", overlay=true)

// Calculate metrics
atr = ta.atr(14)
adx = ta.dmi(14, 14)
rsi = ta.rsi(close, 14)
volumeRatio = volume / ta.sma(volume, 20)
pricePosition = (close - ta.lowest(low, 100)) / (ta.highest(high, 100) - ta.lowest(low, 100)) * 100

// Alert condition: Every bar close (1 minute)
alertcondition(true, title="1min Data Feed", message='
{
  "action": "market_data",
  "symbol": "{{ticker}}",
  "timeframe": "1",
  "atr": ' + str.tostring(atr) + ',
  "adx": ' + str.tostring(adx) + ',
  "rsi": ' + str.tostring(rsi) + ',
  "volumeRatio": ' + str.tostring(volumeRatio) + ',
  "pricePosition": ' + str.tostring(pricePosition) + ',
  "currentPrice": ' + str.tostring(close) + ',
  "indicatorVersion": "v9"
}
')
```

### Alert Configuration
- **Condition:** "Once Per Bar Close"
- **Timeframe:** 1 minute chart
- **Frequency:** Every 1 minute (24/7)
- **Webhook URL:** `https://your-domain.com/api/trading/market-data`
- **Symbol:** SOL-PERP (start with one, add more later)

### Expected Alert Volume
```
Per symbol: 60 alerts/hour = 1,440 alerts/day = 43,200 alerts/month
With 3 symbols: 4,320 alerts/day = 129,600 alerts/month

TradingView Alert Limits:
- Limit is on ACTIVE alerts (slots), not trigger count ✅
- Essential plan: 20 alert slots
- User's current usage: 4/20 slots used
- Needed for this feature: 3 slots (SOL/ETH/BTC 1-minute)
- Remaining after: 13/20 slots free

Result: NO UPGRADE NEEDED - sufficient alert slots available ✅
```

---

## Benefits Beyond Revenge System

### 1. Improved Re-Entry Analytics
- Current: Uses stale data or historical fallback
- With 1-min: Always fresh data (<1 minute old)
- Effect: Better manual trade validation

### 2. Pattern Recognition
- Track ADX behavior before/after stop hunts
- Identify "fake-out" patterns (ADX spikes then drops)
- Optimize entry timing (ADX crossing 20 upward)

### 3. Market Regime Detection
- Real-time: Is market trending or chopping?
- Use case: Disable trading during low-ADX periods
- Implementation: `if (cache.get('SOL-PERP')?.adx < 15) return 'Market too choppy, skip trade'`

### 4. Signal Quality Evolution
- Compare 1-min vs 5-min ADX at signal time
- Question: Does fresher data improve quality scores?
- A/B test: 5-min alerts vs 1-min alerts performance

### 5. Future ML Models
- Features: ADX_1min_ago, ADX_5min_ago, ADX_15min_ago
- Predict: Will this signal hit TP1 or SL?
- Training data: Historical 1-min snapshots + trade outcomes

---

## API Endpoint Impact

### Current `/api/trading/market-data` Endpoint
```typescript
// app/api/trading/market-data/route.ts
export async function POST(request: Request) {
  const body = await request.json()
  
  // Update cache (already handles this)
  const cache = getMarketDataCache()
  cache.set(body.symbol, {
    atr: body.atr,
    adx: body.adx,
    rsi: body.rsi,
    volumeRatio: body.volumeRatio,
    pricePosition: body.pricePosition,
    currentPrice: body.currentPrice,
    timestamp: Date.now()
  })
  
  // NEW (Phase 2): Save to database
  if (process.env.STORE_MARKET_DATA === 'true') {
    await prisma.marketDataSnapshot.create({
      data: {
        symbol: body.symbol,
        timeframe: body.timeframe,
        atr: body.atr,
        adx: body.adx,
        rsi: body.rsi,
        volumeRatio: body.volumeRatio,
        pricePosition: body.pricePosition,
        currentPrice: body.currentPrice,
        indicatorVersion: body.indicatorVersion
      }
    })
  }
  
  return NextResponse.json({ success: true })
}
```

**Rate Limit Considerations:**
- 1 alert/minute = 1,440 requests/day per symbol
- With 3 symbols = 4,320 requests/day
- Current bot handles 10,000+ Position Manager checks/day
- **Impact: Negligible** (0.04% increase in total requests)

---

## Implementation Checklist

### Phase 1: Cache-Only (Immediate)
- [ ] Create 1-min TradingView alert on SOL-PERP
- [ ] Configure webhook to `/api/trading/market-data`
- [ ] Verify cache updates every minute: `curl http://localhost:3001/api/trading/market-data`
- [ ] Test revenge system ADX validation with fresh data
- [ ] Monitor for 24 hours, check cache staleness
- [ ] Add ETH-PERP and BTC-PERP alerts if successful

### Phase 2: Database Persistence (After Validation)
- [ ] Add MarketDataSnapshot model to schema
- [ ] Update API to save snapshots (feature flag controlled)
- [ ] Add cleanup job (delete data >30 days)
- [ ] Create analytics queries (ADX patterns before stop hunts)
- [ ] Build historical backtest: "Would ADX filter help?"

### Phase 3: Revenge System Integration
- [ ] Implement Enhancement #1 Option A (fetch fresh ADX)
- [ ] Add logging: ADX at stop-out vs ADX at revenge entry
- [ ] Track: revenge_with_adx_confirmation vs revenge_without
- [ ] After 20 trades: Compare win rates

---

## Cost Analysis

### TradingView Subscription
- **Current Plan:** Essential ($14.95/month)
  - **GOOD NEWS:** Alert limits are for ACTIVE alerts (slots), not trigger count ✅
  - Current usage: 4/20 alert slots used
  - **Available:** 16 unused alert slots
  - **NO UPGRADE NEEDED** for 1-minute data collection

- **Alert Slot Requirements:**
  - SOL-PERP 1-minute: 1 alert slot
  - ETH-PERP 1-minute: 1 alert slot  
  - BTC-PERP 1-minute: 1 alert slot
  - **Total needed:** 3 alert slots (13 slots still free after implementation)

### Database Storage (Phase 2 Only)
- **Monthly:** ~20 MB with 3 symbols
- **Annual:** ~240 MB
- **Cost:** Free (within PostgreSQL disk allocation)

### Server Resources
- **CPU:** Negligible (cache write = microseconds)
- **Memory:** +60 KB per symbol in cache (180 KB total for 3 symbols)
- **Network:** 150 bytes × 4,320 alerts/day = 648 KB/day = 19.4 MB/month

**Total Additional Cost:** $0/month (no TradingView upgrade needed!) ✅

---

## Risk Mitigation

### What If Alerts Fail?
**Problem:** TradingView alert service down, network issue, rate limiting

**Solution:**
```typescript
// In revenge system, check data freshness
const cache = getMarketDataCache()
const freshData = cache.get(stopHunt.symbol)

if (!freshData) {
  console.log('⚠️ No market data in cache, using fallback')
  // Option 1: Use originalADX as proxy
  // Option 2: Skip ADX validation (fail-open)
  // Option 3: Block revenge (fail-closed)
}

const dataAge = Date.now() - freshData.timestamp
if (dataAge > 300000) { // >5 minutes old
  console.log(`⚠️ Stale data (${(dataAge/60000).toFixed(1)}min old)`)
  // Apply same fallback logic
}
```

### What If Cache Overflows?
**Not an issue:** Map with 3 symbols = 180 KB memory (negligible)

### What If Database Grows Too Large?
**Solution (Phase 2):**
```typescript
// Daily cleanup job
async function cleanupOldMarketData() {
  const thirtyDaysAgo = new Date(Date.now() - 30 * 24 * 60 * 60 * 1000)
  
  await prisma.marketDataSnapshot.deleteMany({
    where: { timestamp: { lt: thirtyDaysAgo } }
  })
  
  console.log('🗑️ Cleaned up market data older than 30 days')
}
```

---

## Next Steps

1. **User Decision:** Start with Phase 1 (cache-only) or implement both phases?
2. **TradingView Upgrade:** Confirm Pro/Premium subscription for unlimited alerts
3. **Symbol Priority:** Start with SOL-PERP only or all 3 symbols?
4. **Create Alert:** I'll provide exact Pine Script + webhook config
5. **Deploy:** Test for 24 hours before revenge system integration

**Recommendation:** Start with Phase 1 (cache-only) on SOL-PERP, validate for 1 week, then expand.