Enhancement #6 - SL Distance Validation (Data Collection Phase): - Added slDistanceAtEntry field to StopHunt schema - Calculates distance from revenge entry to stop zone (LONG vs SHORT logic) - Logs distance in dollars + × ATR multiplier - Purpose: Collect 20+ revenge trade samples for optimal multiplier analysis - Created comprehensive analysis guide with SQL queries - Decision deferred until empirical data collected Enhancement #1 - ADX Confirmation (Implementation Plan): - Documented complete 1-minute TradingView alert strategy - Storage analysis: 19.44 MB/month for 3 symbols (negligible) - Two-phase approach: Cache-only MVP → Optional DB persistence - Provided TradingView Pine Script (ready to use) - Cost breakdown: Pro subscription $49.95/month required - Benefits: Real-time ADX, pattern recognition, ML features - Implementation checklist with validation phases Files Changed: - prisma/schema.prisma: +1 field (slDistanceAtEntry) - lib/trading/stop-hunt-tracker.ts: +10 lines (distance calculation + logging) - docs/1MIN_MARKET_DATA_IMPLEMENTATION.md: NEW (comprehensive plan) - docs/ENHANCEMENT_6_ANALYSIS_GUIDE.md: NEW (SQL queries + decision matrix) Status: Enhancement #4 and #10 deployed (previous commit) Enhancement #6 data collection enabled (this commit) Awaiting 20+ revenge trades for Enhancement #6 decision
This commit is contained in:
mindesbunister
379
docs/1MIN_MARKET_DATA_IMPLEMENTATION.md
Normal file
379
docs/1MIN_MARKET_DATA_IMPLEMENTATION.md
Normal file
@@ -0,0 +1,379 @@
|
||||
# 1-Minute Market Data Collection - Implementation Plan
|
||||
|
||||
**Status:** READY TO IMPLEMENT (Nov 27, 2025)
|
||||
**Purpose:** Enable real-time ADX validation for revenge system (Enhancement #1)
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
**TradingView Alerts:**
|
||||
- Send data ONLY when trend changes (bullish/bearish transitions)
|
||||
- Timeframes: 5min, 15min, 1H, Daily
|
||||
- Problem: No fresh data between trend changes = stale cache for revenge ADX checks
|
||||
|
||||
**Market Data Cache:**
|
||||
- 5-minute expiry
|
||||
- Fields: ADX, ATR, RSI, volumeRatio, pricePosition, currentPrice
|
||||
- Used by: Re-entry analytics, soon revenge system
|
||||
|
||||
---
|
||||
|
||||
## Proposed Solution: 1-Minute Market Data Alerts
|
||||
|
||||
### Database Impact Analysis
|
||||
|
||||
**Current schema:**
|
||||
```sql
|
||||
-- No separate table, data flows through cache only
|
||||
-- Cache: In-memory Map, expires after 5 minutes
|
||||
-- Zero database storage currently
|
||||
```
|
||||
|
||||
**Proposed: Add MarketDataSnapshot table** (OPTIONAL)
|
||||
```sql
|
||||
model MarketDataSnapshot {
|
||||
id String @id @default(cuid())
|
||||
timestamp DateTime @default(now())
|
||||
symbol String
|
||||
timeframe String // "1", "5", "15", "60", "D"
|
||||
|
||||
-- Metrics
|
||||
atr Float
|
||||
adx Float
|
||||
rsi Float
|
||||
volumeRatio Float
|
||||
pricePosition Float
|
||||
currentPrice Float
|
||||
|
||||
-- Metadata
|
||||
indicatorVersion String? // e.g., "v9"
|
||||
|
||||
@@index([symbol, timestamp])
|
||||
@@index([symbol, timeframe, timestamp])
|
||||
}
|
||||
```
|
||||
|
||||
**Storage Calculation:**
|
||||
```
|
||||
Per record: ~150 bytes (8 fields × 8-20 bytes each)
|
||||
Per minute: 150 bytes × 1 symbol = 150 bytes
|
||||
Per hour: 150 × 60 = 9 KB
|
||||
Per day: 9 KB × 24 = 216 KB
|
||||
Per month: 216 KB × 30 = 6.48 MB
|
||||
Per year: 6.48 MB × 12 = 77.76 MB
|
||||
|
||||
With 3 symbols (SOL, ETH, BTC):
|
||||
Per month: 6.48 MB × 3 = 19.44 MB
|
||||
Per year: 77.76 MB × 3 = 233.28 MB
|
||||
|
||||
CONCLUSION: Negligible storage impact (<1 GB/year for 3 symbols)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Options
|
||||
|
||||
### Option A: Cache-Only (RECOMMENDED for MVP)
|
||||
|
||||
**Pros:**
|
||||
- ✅ Zero database changes required
|
||||
- ✅ Existing infrastructure (market-data-cache.ts)
|
||||
- ✅ Fast implementation (just TradingView alert)
|
||||
- ✅ No storage overhead
|
||||
|
||||
**Cons:**
|
||||
- ❌ No historical analysis (can't backtest ADX patterns)
|
||||
- ❌ Lost on container restart (cache clears)
|
||||
- ❌ 5-minute window only (recent data)
|
||||
|
||||
**Use Case:**
|
||||
- Revenge system ADX validation (real-time only)
|
||||
- Re-entry analytics (already working)
|
||||
- Good for Phase 1 validation
|
||||
|
||||
**Implementation Steps:**
|
||||
1. Create TradingView 1-minute alert (15 lines of Pine Script)
|
||||
2. Point to existing `/api/trading/market-data` endpoint
|
||||
3. Cache handles the rest automatically
|
||||
4. Test with revenge system
|
||||
|
||||
---
|
||||
|
||||
### Option B: Cache + Database Persistence
|
||||
|
||||
**Pros:**
|
||||
- ✅ Historical analysis (backtest ADX-based filters)
|
||||
- ✅ Survives container restarts
|
||||
- ✅ Enables future ML models (train on historical patterns)
|
||||
- ✅ Audit trail for debugging
|
||||
|
||||
**Cons:**
|
||||
- ❌ Requires schema migration
|
||||
- ❌ Need cleanup policy (auto-delete old data)
|
||||
- ❌ Slightly more complex
|
||||
|
||||
**Use Case:**
|
||||
- Long-term data science projects
|
||||
- Pattern recognition (what ADX patterns precede stop hunts?)
|
||||
- System optimization with historical validation
|
||||
|
||||
**Implementation Steps:**
|
||||
1. Add MarketDataSnapshot model to schema
|
||||
2. Update `/api/trading/market-data` to save to DB
|
||||
3. Add cleanup job (delete data >30 days old)
|
||||
4. Create TradingView 1-minute alert
|
||||
5. Build analytics queries
|
||||
|
||||
---
|
||||
|
||||
## Recommended Approach: Hybrid (Start with A, Add B Later)
|
||||
|
||||
### Phase 1: Cache-Only (This Week)
|
||||
```
|
||||
1. Create 1-min TradingView alert
|
||||
2. Point to /api/trading/market-data
|
||||
3. Test revenge ADX validation
|
||||
4. Monitor cache hit rate
|
||||
5. Validate revenge outcomes improve
|
||||
```
|
||||
|
||||
### Phase 2: Add Persistence (After 10+ Revenge Trades)
|
||||
```
|
||||
1. Add MarketDataSnapshot table
|
||||
2. Save historical data
|
||||
3. Backtest: "Would ADX filter have helped?"
|
||||
4. Optimize thresholds based on data
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## TradingView Alert Setup (1-Minute Data)
|
||||
|
||||
### Pine Script Code
|
||||
```pinescript
|
||||
//@version=5
|
||||
indicator("Market Data Feed (1min)", overlay=true)
|
||||
|
||||
// Calculate metrics
|
||||
atr = ta.atr(14)
|
||||
adx = ta.dmi(14, 14)
|
||||
rsi = ta.rsi(close, 14)
|
||||
volumeRatio = volume / ta.sma(volume, 20)
|
||||
pricePosition = (close - ta.lowest(low, 100)) / (ta.highest(high, 100) - ta.lowest(low, 100)) * 100
|
||||
|
||||
// Alert condition: Every bar close (1 minute)
|
||||
alertcondition(true, title="1min Data Feed", message='
|
||||
{
|
||||
"action": "market_data",
|
||||
"symbol": "{{ticker}}",
|
||||
"timeframe": "1",
|
||||
"atr": ' + str.tostring(atr) + ',
|
||||
"adx": ' + str.tostring(adx) + ',
|
||||
"rsi": ' + str.tostring(rsi) + ',
|
||||
"volumeRatio": ' + str.tostring(volumeRatio) + ',
|
||||
"pricePosition": ' + str.tostring(pricePosition) + ',
|
||||
"currentPrice": ' + str.tostring(close) + ',
|
||||
"indicatorVersion": "v9"
|
||||
}
|
||||
')
|
||||
```
|
||||
|
||||
### Alert Configuration
|
||||
- **Condition:** "Once Per Bar Close"
|
||||
- **Timeframe:** 1 minute chart
|
||||
- **Frequency:** Every 1 minute (24/7)
|
||||
- **Webhook URL:** `https://your-domain.com/api/trading/market-data`
|
||||
- **Symbol:** SOL-PERP (start with one, add more later)
|
||||
|
||||
### Expected Alert Volume
|
||||
```
|
||||
Per symbol: 60 alerts/hour = 1,440 alerts/day
|
||||
With 3 symbols: 4,320 alerts/day
|
||||
TradingView free tier: 400 alerts/month (NOT ENOUGH)
|
||||
TradingView Pro tier: Unlimited alerts ✅
|
||||
|
||||
User needs: Pro or Premium plan ($14.95-$59.95/month)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Benefits Beyond Revenge System
|
||||
|
||||
### 1. Improved Re-Entry Analytics
|
||||
- Current: Uses stale data or historical fallback
|
||||
- With 1-min: Always fresh data (<1 minute old)
|
||||
- Effect: Better manual trade validation
|
||||
|
||||
### 2. Pattern Recognition
|
||||
- Track ADX behavior before/after stop hunts
|
||||
- Identify "fake-out" patterns (ADX spikes then drops)
|
||||
- Optimize entry timing (ADX crossing 20 upward)
|
||||
|
||||
### 3. Market Regime Detection
|
||||
- Real-time: Is market trending or chopping?
|
||||
- Use case: Disable trading during low-ADX periods
|
||||
- Implementation: `if (cache.get('SOL-PERP')?.adx < 15) return 'Market too choppy, skip trade'`
|
||||
|
||||
### 4. Signal Quality Evolution
|
||||
- Compare 1-min vs 5-min ADX at signal time
|
||||
- Question: Does fresher data improve quality scores?
|
||||
- A/B test: 5-min alerts vs 1-min alerts performance
|
||||
|
||||
### 5. Future ML Models
|
||||
- Features: ADX_1min_ago, ADX_5min_ago, ADX_15min_ago
|
||||
- Predict: Will this signal hit TP1 or SL?
|
||||
- Training data: Historical 1-min snapshots + trade outcomes
|
||||
|
||||
---
|
||||
|
||||
## API Endpoint Impact
|
||||
|
||||
### Current `/api/trading/market-data` Endpoint
|
||||
```typescript
|
||||
// app/api/trading/market-data/route.ts
|
||||
export async function POST(request: Request) {
|
||||
const body = await request.json()
|
||||
|
||||
// Update cache (already handles this)
|
||||
const cache = getMarketDataCache()
|
||||
cache.set(body.symbol, {
|
||||
atr: body.atr,
|
||||
adx: body.adx,
|
||||
rsi: body.rsi,
|
||||
volumeRatio: body.volumeRatio,
|
||||
pricePosition: body.pricePosition,
|
||||
currentPrice: body.currentPrice,
|
||||
timestamp: Date.now()
|
||||
})
|
||||
|
||||
// NEW (Phase 2): Save to database
|
||||
if (process.env.STORE_MARKET_DATA === 'true') {
|
||||
await prisma.marketDataSnapshot.create({
|
||||
data: {
|
||||
symbol: body.symbol,
|
||||
timeframe: body.timeframe,
|
||||
atr: body.atr,
|
||||
adx: body.adx,
|
||||
rsi: body.rsi,
|
||||
volumeRatio: body.volumeRatio,
|
||||
pricePosition: body.pricePosition,
|
||||
currentPrice: body.currentPrice,
|
||||
indicatorVersion: body.indicatorVersion
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
return NextResponse.json({ success: true })
|
||||
}
|
||||
```
|
||||
|
||||
**Rate Limit Considerations:**
|
||||
- 1 alert/minute = 1,440 requests/day per symbol
|
||||
- With 3 symbols = 4,320 requests/day
|
||||
- Current bot handles 10,000+ Position Manager checks/day
|
||||
- **Impact: Negligible** (0.04% increase in total requests)
|
||||
|
||||
---
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
### Phase 1: Cache-Only (Immediate)
|
||||
- [ ] Create 1-min TradingView alert on SOL-PERP
|
||||
- [ ] Configure webhook to `/api/trading/market-data`
|
||||
- [ ] Verify cache updates every minute: `curl http://localhost:3001/api/trading/market-data`
|
||||
- [ ] Test revenge system ADX validation with fresh data
|
||||
- [ ] Monitor for 24 hours, check cache staleness
|
||||
- [ ] Add ETH-PERP and BTC-PERP alerts if successful
|
||||
|
||||
### Phase 2: Database Persistence (After Validation)
|
||||
- [ ] Add MarketDataSnapshot model to schema
|
||||
- [ ] Update API to save snapshots (feature flag controlled)
|
||||
- [ ] Add cleanup job (delete data >30 days)
|
||||
- [ ] Create analytics queries (ADX patterns before stop hunts)
|
||||
- [ ] Build historical backtest: "Would ADX filter help?"
|
||||
|
||||
### Phase 3: Revenge System Integration
|
||||
- [ ] Implement Enhancement #1 Option A (fetch fresh ADX)
|
||||
- [ ] Add logging: ADX at stop-out vs ADX at revenge entry
|
||||
- [ ] Track: revenge_with_adx_confirmation vs revenge_without
|
||||
- [ ] After 20 trades: Compare win rates
|
||||
|
||||
---
|
||||
|
||||
## Cost Analysis
|
||||
|
||||
### TradingView Subscription
|
||||
- **Current:** Essential ($14.95/month) - 400 alerts/month (NOT ENOUGH)
|
||||
- **Required:** Pro ($49.95/month) - Unlimited alerts ✅
|
||||
- **Alternative:** Premium ($59.95/month) - More indicators + features
|
||||
|
||||
### Database Storage (Phase 2 Only)
|
||||
- **Monthly:** ~20 MB with 3 symbols
|
||||
- **Annual:** ~240 MB
|
||||
- **Cost:** Free (within PostgreSQL disk allocation)
|
||||
|
||||
### Server Resources
|
||||
- **CPU:** Negligible (cache write = microseconds)
|
||||
- **Memory:** +60 KB per symbol in cache (180 KB total for 3 symbols)
|
||||
- **Network:** 150 bytes × 4,320 alerts/day = 648 KB/day = 19.4 MB/month
|
||||
|
||||
**Total Additional Cost:** ~$35/month (TradingView Pro upgrade)
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
### What If Alerts Fail?
|
||||
**Problem:** TradingView alert service down, network issue, rate limiting
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// In revenge system, check data freshness
|
||||
const cache = getMarketDataCache()
|
||||
const freshData = cache.get(stopHunt.symbol)
|
||||
|
||||
if (!freshData) {
|
||||
console.log('⚠️ No market data in cache, using fallback')
|
||||
// Option 1: Use originalADX as proxy
|
||||
// Option 2: Skip ADX validation (fail-open)
|
||||
// Option 3: Block revenge (fail-closed)
|
||||
}
|
||||
|
||||
const dataAge = Date.now() - freshData.timestamp
|
||||
if (dataAge > 300000) { // >5 minutes old
|
||||
console.log(`⚠️ Stale data (${(dataAge/60000).toFixed(1)}min old)`)
|
||||
// Apply same fallback logic
|
||||
}
|
||||
```
|
||||
|
||||
### What If Cache Overflows?
|
||||
**Not an issue:** Map with 3 symbols = 180 KB memory (negligible)
|
||||
|
||||
### What If Database Grows Too Large?
|
||||
**Solution (Phase 2):**
|
||||
```typescript
|
||||
// Daily cleanup job
|
||||
async function cleanupOldMarketData() {
|
||||
const thirtyDaysAgo = new Date(Date.now() - 30 * 24 * 60 * 60 * 1000)
|
||||
|
||||
await prisma.marketDataSnapshot.deleteMany({
|
||||
where: { timestamp: { lt: thirtyDaysAgo } }
|
||||
})
|
||||
|
||||
console.log('🗑️ Cleaned up market data older than 30 days')
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **User Decision:** Start with Phase 1 (cache-only) or implement both phases?
|
||||
2. **TradingView Upgrade:** Confirm Pro/Premium subscription for unlimited alerts
|
||||
3. **Symbol Priority:** Start with SOL-PERP only or all 3 symbols?
|
||||
4. **Create Alert:** I'll provide exact Pine Script + webhook config
|
||||
5. **Deploy:** Test for 24 hours before revenge system integration
|
||||
|
||||
**Recommendation:** Start with Phase 1 (cache-only) on SOL-PERP, validate for 1 week, then expand.
|
||||
|
||||
Reference in New Issue
Block a user