trading_bot_v4/docs/1MIN_MARKET_DATA_IMPLEMENTATION.md

1-Minute Market Data Collection - Implementation Plan

Status: READY TO IMPLEMENT - Zero Cost! (Nov 27, 2025)
Purpose: Enable real-time ADX validation for revenge system (Enhancement #1)
Cost: $0/month - No TradingView upgrade needed ✅

GOOD NEWS: Alert limits are for ACTIVE alerts (slots), not trigger frequency!

• Current: 4/20 alert slots used
• Needed: 3 slots (SOL/ETH/BTC)
• Result: 13 slots still free after implementation

---

Current State

TradingView Alerts:

• Send data ONLY when trend changes (bullish/bearish transitions)
• Timeframes: 5min, 15min, 1H, Daily
• Problem: No fresh data between trend changes = stale cache for revenge ADX checks

Market Data Cache:

• 5-minute expiry
• Fields: ADX, ATR, RSI, volumeRatio, pricePosition, currentPrice
• Used by: Re-entry analytics, soon revenge system

---

Proposed Solution: 1-Minute Market Data Alerts

Database Impact Analysis

Current schema:

-- No separate table, data flows through cache only
-- Cache: In-memory Map, expires after 5 minutes
-- Zero database storage currently

Proposed: Add MarketDataSnapshot table (OPTIONAL)

model MarketDataSnapshot {
  id               String   @id @default(cuid())
  timestamp        DateTime @default(now())
  symbol           String
  timeframe        String   // "1", "5", "15", "60", "D"
  
  -- Metrics
  atr              Float
  adx              Float
  rsi              Float
  volumeRatio      Float
  pricePosition    Float
  currentPrice     Float
  
  -- Metadata
  indicatorVersion String?  // e.g., "v9"
  
  @@index([symbol, timestamp])
  @@index([symbol, timeframe, timestamp])
}

Storage Calculation:

Per record: ~150 bytes (8 fields × 8-20 bytes each)
Per minute: 150 bytes × 1 symbol = 150 bytes
Per hour: 150 × 60 = 9 KB
Per day: 9 KB × 24 = 216 KB
Per month: 216 KB × 30 = 6.48 MB
Per year: 6.48 MB × 12 = 77.76 MB

With 3 symbols (SOL, ETH, BTC):
Per month: 6.48 MB × 3 = 19.44 MB
Per year: 77.76 MB × 3 = 233.28 MB

CONCLUSION: Negligible storage impact (<1 GB/year for 3 symbols)

---

Implementation Options

Option A: Cache-Only (RECOMMENDED for MVP)

Pros:

• ✅ Zero database changes required
• ✅ Existing infrastructure (market-data-cache.ts)
• ✅ Fast implementation (just TradingView alert)
• ✅ No storage overhead

Cons:

• ❌ No historical analysis (can't backtest ADX patterns)
• ❌ Lost on container restart (cache clears)
• ❌ 5-minute window only (recent data)

Use Case:

• Revenge system ADX validation (real-time only)
• Re-entry analytics (already working)
• Good for Phase 1 validation

Implementation Steps:

1. Create TradingView 1-minute alert (15 lines of Pine Script)
2. Point to existing /api/trading/market-data endpoint
3. Cache handles the rest automatically
4. Test with revenge system

---

Option B: Cache + Database Persistence

Pros:

• ✅ Historical analysis (backtest ADX-based filters)
• ✅ Survives container restarts
• ✅ Enables future ML models (train on historical patterns)
• ✅ Audit trail for debugging

Cons:

• ❌ Requires schema migration
• ❌ Need cleanup policy (auto-delete old data)
• ❌ Slightly more complex

Use Case:

• Long-term data science projects
• Pattern recognition (what ADX patterns precede stop hunts?)
• System optimization with historical validation

Implementation Steps:

1. Add MarketDataSnapshot model to schema
2. Update /api/trading/market-data to save to DB
3. Add cleanup job (delete data >30 days old)
4. Create TradingView 1-minute alert
5. Build analytics queries

---

Recommended Approach: Hybrid (Start with A, Add B Later)

Phase 1: Cache-Only (This Week)

1. Create 1-min TradingView alert
2. Point to /api/trading/market-data
3. Test revenge ADX validation
4. Monitor cache hit rate
5. Validate revenge outcomes improve

Phase 2: Add Persistence (After 10+ Revenge Trades)

1. Add MarketDataSnapshot table
2. Save historical data
3. Backtest: "Would ADX filter have helped?"
4. Optimize thresholds based on data

---

TradingView Alert Setup (1-Minute Data)

Pine Script Code

//@version=5
indicator("Market Data Feed (1min)", overlay=true)

// Calculate metrics
atr = ta.atr(14)
adx = ta.dmi(14, 14)
rsi = ta.rsi(close, 14)
volumeRatio = volume / ta.sma(volume, 20)
pricePosition = (close - ta.lowest(low, 100)) / (ta.highest(high, 100) - ta.lowest(low, 100)) * 100

// Alert condition: Every bar close (1 minute)
alertcondition(true, title="1min Data Feed", message='
{
  "action": "market_data",
  "symbol": "{{ticker}}",
  "timeframe": "1",
  "atr": ' + str.tostring(atr) + ',
  "adx": ' + str.tostring(adx) + ',
  "rsi": ' + str.tostring(rsi) + ',
  "volumeRatio": ' + str.tostring(volumeRatio) + ',
  "pricePosition": ' + str.tostring(pricePosition) + ',
  "currentPrice": ' + str.tostring(close) + ',
  "indicatorVersion": "v9"
}
')

Alert Configuration

• Condition: "Once Per Bar Close"
• Timeframe: 1 minute chart
• Frequency: Every 1 minute (24/7)
• Webhook URL: https://your-domain.com/api/trading/market-data
• Symbol: SOL-PERP (start with one, add more later)

Expected Alert Volume

Per symbol: 60 alerts/hour = 1,440 alerts/day = 43,200 alerts/month
With 3 symbols: 4,320 alerts/day = 129,600 alerts/month

TradingView Alert Limits:
- Limit is on ACTIVE alerts (slots), not trigger count ✅
- Essential plan: 20 alert slots
- User's current usage: 4/20 slots used
- Needed for this feature: 3 slots (SOL/ETH/BTC 1-minute)
- Remaining after: 13/20 slots free

Result: NO UPGRADE NEEDED - sufficient alert slots available ✅

---

Benefits Beyond Revenge System

1. Improved Re-Entry Analytics

• Current: Uses stale data or historical fallback
• With 1-min: Always fresh data (<1 minute old)
• Effect: Better manual trade validation

2. Pattern Recognition

• Track ADX behavior before/after stop hunts
• Identify "fake-out" patterns (ADX spikes then drops)
• Optimize entry timing (ADX crossing 20 upward)

3. Market Regime Detection

• Real-time: Is market trending or chopping?
• Use case: Disable trading during low-ADX periods
• Implementation: if (cache.get('SOL-PERP')?.adx < 15) return 'Market too choppy, skip trade'

4. Signal Quality Evolution

• Compare 1-min vs 5-min ADX at signal time
• Question: Does fresher data improve quality scores?
• A/B test: 5-min alerts vs 1-min alerts performance

5. Future ML Models

• Features: ADX_1min_ago, ADX_5min_ago, ADX_15min_ago
• Predict: Will this signal hit TP1 or SL?
• Training data: Historical 1-min snapshots + trade outcomes

---

API Endpoint Impact

Current /api/trading/market-data Endpoint

// app/api/trading/market-data/route.ts
export async function POST(request: Request) {
  const body = await request.json()
  
  // Update cache (already handles this)
  const cache = getMarketDataCache()
  cache.set(body.symbol, {
    atr: body.atr,
    adx: body.adx,
    rsi: body.rsi,
    volumeRatio: body.volumeRatio,
    pricePosition: body.pricePosition,
    currentPrice: body.currentPrice,
    timestamp: Date.now()
  })
  
  // NEW (Phase 2): Save to database
  if (process.env.STORE_MARKET_DATA === 'true') {
    await prisma.marketDataSnapshot.create({
      data: {
        symbol: body.symbol,
        timeframe: body.timeframe,
        atr: body.atr,
        adx: body.adx,
        rsi: body.rsi,
        volumeRatio: body.volumeRatio,
        pricePosition: body.pricePosition,
        currentPrice: body.currentPrice,
        indicatorVersion: body.indicatorVersion
      }
    })
  }
  
  return NextResponse.json({ success: true })
}

Rate Limit Considerations:

• 1 alert/minute = 1,440 requests/day per symbol
• With 3 symbols = 4,320 requests/day
• Current bot handles 10,000+ Position Manager checks/day
• Impact: Negligible (0.04% increase in total requests)

---

Implementation Checklist

Phase 1: Cache-Only (Immediate)

• Create 1-min TradingView alert on SOL-PERP
• Configure webhook to /api/trading/market-data
• Verify cache updates every minute: curl http://localhost:3001/api/trading/market-data
• Test revenge system ADX validation with fresh data
• Monitor for 24 hours, check cache staleness
• Add ETH-PERP and BTC-PERP alerts if successful

Phase 2: Database Persistence (After Validation)

• Add MarketDataSnapshot model to schema
• Update API to save snapshots (feature flag controlled)
• Add cleanup job (delete data >30 days)
• Create analytics queries (ADX patterns before stop hunts)
• Build historical backtest: "Would ADX filter help?"

Phase 3: Revenge System Integration

• Implement Enhancement #1 Option A (fetch fresh ADX)
• Add logging: ADX at stop-out vs ADX at revenge entry
• Track: revenge_with_adx_confirmation vs revenge_without
• After 20 trades: Compare win rates

---

Cost Analysis

TradingView Subscription

• Current Plan: Essential ($14.95/month)

  • GOOD NEWS: Alert limits are for ACTIVE alerts (slots), not trigger count ✅
  • Current usage: 4/20 alert slots used
  • Available: 16 unused alert slots
  • NO UPGRADE NEEDED for 1-minute data collection

• Alert Slot Requirements:

  • SOL-PERP 1-minute: 1 alert slot
  • ETH-PERP 1-minute: 1 alert slot
  • BTC-PERP 1-minute: 1 alert slot
  • Total needed: 3 alert slots (13 slots still free after implementation)

Database Storage (Phase 2 Only)

• Monthly: ~20 MB with 3 symbols
• Annual: ~240 MB
• Cost: Free (within PostgreSQL disk allocation)

Server Resources

• CPU: Negligible (cache write = microseconds)
• Memory: +60 KB per symbol in cache (180 KB total for 3 symbols)
• Network: 150 bytes × 4,320 alerts/day = 648 KB/day = 19.4 MB/month

Total Additional Cost: $0/month (no TradingView upgrade needed!) ✅

---

Risk Mitigation

What If Alerts Fail?

Problem: TradingView alert service down, network issue, rate limiting

Solution:

// In revenge system, check data freshness
const cache = getMarketDataCache()
const freshData = cache.get(stopHunt.symbol)

if (!freshData) {
  console.log('⚠️ No market data in cache, using fallback')
  // Option 1: Use originalADX as proxy
  // Option 2: Skip ADX validation (fail-open)
  // Option 3: Block revenge (fail-closed)
}

const dataAge = Date.now() - freshData.timestamp
if (dataAge > 300000) { // >5 minutes old
  console.log(`⚠️ Stale data (${(dataAge/60000).toFixed(1)}min old)`)
  // Apply same fallback logic
}

What If Cache Overflows?

Not an issue: Map with 3 symbols = 180 KB memory (negligible)

What If Database Grows Too Large?

Solution (Phase 2):

// Daily cleanup job
async function cleanupOldMarketData() {
  const thirtyDaysAgo = new Date(Date.now() - 30 * 24 * 60 * 60 * 1000)
  
  await prisma.marketDataSnapshot.deleteMany({
    where: { timestamp: { lt: thirtyDaysAgo } }
  })
  
  console.log('🗑️ Cleaned up market data older than 30 days')
}

---

Next Steps

1. User Decision: Start with Phase 1 (cache-only) or implement both phases?
2. TradingView Upgrade: Confirm Pro/Premium subscription for unlimited alerts
3. Symbol Priority: Start with SOL-PERP only or all 3 symbols?
4. Create Alert: I'll provide exact Pine Script + webhook config
5. Deploy: Test for 24 hours before revenge system integration

Recommendation: Start with Phase 1 (cache-only) on SOL-PERP, validate for 1 week, then expand.