trading_bot_v4/.github/copilot-instructions.md

AI Agent Instructions for Trading Bot v4

Mission & Financial Goals

Primary Objective: Build wealth systematically from $106 → $100,000+ through algorithmic trading

Current Phase: Phase 1 - Survival & Proof (Nov 2025 - Jan 2026)

• Starting Capital: $106 (+ $1,000 deposit in 2 weeks)
• Target: $2,500 by end of Phase 1 (Month 2.5)
• Strategy: Aggressive compounding, 0 withdrawals
• Position Sizing: 100% of account ($106 at 20x leverage = $2,120 notional)
• Risk Tolerance: EXTREME - This is recovery/proof-of-concept mode
• Win Target: 20-30% monthly returns to reach $2,500

Why This Matters for AI Agents:

• Every dollar counts at this stage - optimize for profitability, not just safety
• User needs this system to work for long-term financial goals ($300-500/month withdrawals starting Month 3)
• No changes that reduce win rate unless they improve profit factor
• System must prove itself before scaling (see TRADING_GOALS.md for full 8-phase roadmap)

Key Constraints:

• Can't afford extended drawdowns (limited capital)
• Must maintain 60%+ win rate to compound effectively
• Quality over quantity - only trade 70+ signal quality scores
• After 3 consecutive losses, STOP and review system

Architecture Overview

Type: Autonomous cryptocurrency trading bot with Next.js 15 frontend + Solana/Drift Protocol backend

Data Flow: TradingView → n8n webhook → Next.js API → Drift Protocol (Solana DEX) → Real-time monitoring → Auto-exit

Key Design Principle: Dual-layer redundancy - every trade has both on-chain orders (Drift) AND software monitoring (Position Manager) as backup.

Exit Strategy: TP2-as-Runner system (CURRENT):

• TP1 at +0.4%: Close configurable % (default 75%, adjustable via TAKE_PROFIT_1_SIZE_PERCENT)
• TP2 at +0.7%: Activates trailing stop on full remaining % (no position close)
• Runner: Remaining % after TP1 with ATR-based trailing stop (default 25%, configurable)
• Note: All UI displays dynamically calculate runner% as 100 - TAKE_PROFIT_1_SIZE_PERCENT

Per-Symbol Configuration: SOL and ETH have independent enable/disable toggles and position sizing:

• SOLANA_ENABLED, SOLANA_POSITION_SIZE, SOLANA_LEVERAGE (defaults: true, $210, 10x)
• ETHEREUM_ENABLED, ETHEREUM_POSITION_SIZE, ETHEREUM_LEVERAGE (defaults: true, $4, 1x)
• BTC and other symbols fall back to global settings (MAX_POSITION_SIZE_USD, LEVERAGE)
• Priority: Per-symbol ENV → Market config → Global ENV → Defaults

Signal Quality System: Filters trades based on 5 metrics (ATR, ADX, RSI, volumeRatio, pricePosition) scored 0-100. Only trades scoring 60+ are executed. Scores stored in database for future optimization.

Timeframe-Aware Scoring: Signal quality thresholds adjust based on timeframe (5min vs daily):

• 5min: ADX 12+ trending (vs 18+ for daily), ATR 0.2-0.7% healthy (vs 0.4%+ for daily)
• Anti-chop filter: -20 points for extreme sideways (ADX <10, ATR <0.25%, Vol <0.9x)
• Pass timeframe param to scoreSignalQuality() from TradingView alerts (e.g., timeframe: "5")

MAE/MFE Tracking: Every trade tracks Maximum Favorable Excursion (best profit %) and Maximum Adverse Excursion (worst loss %) updated every 2s. Used for data-driven optimization of TP/SL levels.

Manual Trading via Telegram: Send plain-text messages like long sol, short eth, long btc to open positions instantly (bypasses n8n, calls /api/trading/execute directly with preset healthy metrics).

Re-Entry Analytics System: Manual trades are validated before execution using fresh TradingView data:

• Market data cached from TradingView signals (5min expiry)
• /api/analytics/reentry-check scores re-entry based on fresh metrics + recent performance
• Telegram bot blocks low-quality re-entries unless --force flag used
• Uses real TradingView ADX/ATR/RSI when available, falls back to historical data
• Penalty for recent losing trades, bonus for winning streaks

Critical Components

1. Signal Quality Scoring (lib/trading/signal-quality.ts)

Purpose: Unified quality validation system that scores trading signals 0-100 based on 5 market metrics

Timeframe-aware thresholds:

scoreSignalQuality({ 
  atr, adx, rsi, volumeRatio, pricePosition, 
  timeframe?: string // "5" for 5min, undefined for higher timeframes
})

5min chart adjustments:

• ADX healthy range: 12-22 (vs 18-30 for daily)
• ATR healthy range: 0.2-0.7% (vs 0.4%+ for daily)
• Anti-chop filter: -20 points for extreme sideways (ADX <10, ATR <0.25%, Vol <0.9x)

Price position penalties (all timeframes):

• Long at 90-95%+ range: -15 to -30 points (chasing highs)
• Short at <5-10% range: -15 to -30 points (chasing lows)
• Prevents flip-flop losses from entering range extremes

Key behaviors:

• Returns score 0-100 and detailed breakdown object
• Minimum score 60 required to execute trade
• Called by both /api/trading/check-risk and /api/trading/execute
• Scores saved to database for post-trade analysis

2. Position Manager (lib/trading/position-manager.ts)

Purpose: Software-based monitoring loop that checks prices every 2 seconds and closes positions via market orders

Singleton pattern: Always use getInitializedPositionManager() - never instantiate directly

const positionManager = await getInitializedPositionManager()
await positionManager.addTrade(activeTrade)

Key behaviors:

• Tracks ActiveTrade objects in a Map
• TP2-as-Runner system: TP1 (configurable %, default 75%) → TP2 trigger (no close, activate trailing) → Runner (remaining %) with ATR-based trailing stop
• Dynamic SL adjustments: Moves to breakeven after TP1, locks profit at +1.2%
• On-chain order synchronization: After TP1 hits, calls cancelAllOrders() then placeExitOrders() with updated SL price at breakeven (uses retryWithBackoff() for rate limit handling)
• ATR-based trailing stop: Calculates trail distance as (atrAtEntry / currentPrice × 100) × trailingStopAtrMultiplier, clamped between min/max %
• Trailing stop: Activates when TP2 price hit, tracks peakPrice and trails dynamically
• Closes positions via closePosition() market orders when targets hit
• Acts as backup if on-chain orders don't fill
• State persistence: Saves to database, restores on restart via configSnapshot.positionManagerState
• Grace period for new trades: Skips "external closure" detection for positions <30 seconds old (Drift positions take 5-10s to propagate)
• Exit reason detection: Uses trade state flags (tp1Hit, tp2Hit) and realized P&L to determine exit reason, NOT current price (avoids misclassification when price moves after order fills)
• Real P&L calculation: Calculates actual profit based on entry vs exit price, not SDK's potentially incorrect values

3. Telegram Bot (telegram_command_bot.py)

Purpose: Python-based Telegram bot for manual trading commands and position status monitoring

Manual trade commands via plain text:

# User sends plain text message (not slash commands)
"long sol"          → Validates via analytics, then opens SOL-PERP long
"short eth"         → Validates via analytics, then opens ETH-PERP short
"long btc --force"  → Skips analytics validation, opens BTC-PERP long immediately

Key behaviors:

• MessageHandler processes all text messages (not just commands)
• Maps user-friendly symbols (sol, eth, btc) to Drift format (SOL-PERP, etc.)
• Analytics validation: Calls /api/analytics/reentry-check before execution
  • Blocks trades with score <55 unless --force flag used
  • Uses fresh TradingView data (<5min old) when available
  • Falls back to historical metrics with penalty
  • Considers recent trade performance (last 3 trades)
• Calls /api/trading/execute directly with preset healthy metrics (ATR=0.45, ADX=32, RSI=58/42)
• Bypasses n8n workflow and TradingView requirements
• 60-second timeout for API calls
• Responds with trade confirmation or analytics rejection message

Status command:

/status → Returns JSON of open positions from Drift

Implementation details:

• Uses python-telegram-bot library
• Deployed via docker-compose.telegram-bot.yml
• Requires TELEGRAM_BOT_TOKEN and TELEGRAM_CHANNEL_ID in .env
• API calls to http://trading-bot:3000/api/trading/execute

Drift client integration:

• Singleton pattern: Use initializeDriftService() and getDriftService() - maintains single connection

const driftService = await initializeDriftService()
const health = await driftService.getAccountHealth()

• Wallet handling: Supports both JSON array [91,24,...] and base58 string formats from Phantom wallet

4. Rate Limit Monitoring (lib/drift/orders.ts + app/api/analytics/rate-limits)

Purpose: Track and analyze Solana RPC rate limiting (429 errors) to prevent silent failures

Retry mechanism with exponential backoff:

await retryWithBackoff(async () => {
  return await driftClient.cancelOrders(...)
}, maxRetries = 3, baseDelay = 2000)

Database logging: Three event types in SystemEvent table:

• rate_limit_hit: Each 429 error (logged with attempt #, delay, error snippet)
• rate_limit_recovered: Successful retry (logged with total time, retry count)
• rate_limit_exhausted: Failed after max retries (CRITICAL - order operation failed)

Analytics endpoint:

curl http://localhost:3001/api/analytics/rate-limits

Returns: Total hits/recoveries/failures, hourly patterns, recovery times, success rate

Key behaviors:

• Only RPC calls wrapped: cancelAllOrders(), placeExitOrders(), closePosition()
• Position Manager 2s loop does NOT make RPC calls (only price checks via Pyth WebSocket)
• Exponential backoff: 2s → 4s → 8s delays on retry
• Logs to both console and database for post-trade analysis

Monitoring queries: See docs/RATE_LIMIT_MONITORING.md for SQL queries

5. Order Placement (lib/drift/orders.ts)

Critical functions:

• openPosition() - Opens market position with transaction confirmation
• closePosition() - Closes position with transaction confirmation
• placeExitOrders() - Places TP/SL orders on-chain
• cancelAllOrders() - Cancels all reduce-only orders for a market

CRITICAL: Transaction Confirmation Pattern Both openPosition() and closePosition() MUST confirm transactions on-chain:

const txSig = await driftClient.placePerpOrder(orderParams)
console.log('⏳ Confirming transaction on-chain...')
const connection = driftService.getConnection()
const confirmation = await connection.confirmTransaction(txSig, 'confirmed')

if (confirmation.value.err) {
  throw new Error(`Transaction failed: ${JSON.stringify(confirmation.value.err)}`)
}
console.log('✅ Transaction confirmed on-chain')

Without this, the SDK returns signatures for transactions that never execute, causing phantom trades/closes.

CRITICAL: Drift SDK position.size is USD, not tokens The Drift SDK returns position.size as USD notional value, NOT token quantity:

// WRONG: Multiply by price (inflates by 156x for SOL at $157)
const positionSizeUSD = position.size * currentPrice

// CORRECT: Use directly as USD value
const positionSizeUSD = Math.abs(position.size)

This affects Position Manager's TP1 detection - if calculated incorrectly, TP1 will never trigger because expected size won't match actual size.

Solana RPC Rate Limiting with Exponential Backoff Solana RPC endpoints return 429 errors under load. Always use retry logic for order operations:

export async function retryWithBackoff<T>(
  operation: () => Promise<T>,
  maxRetries: number = 3,
  initialDelay: number = 2000
): Promise<T> {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await operation()
    } catch (error: any) {
      if (error?.message?.includes('429') && attempt < maxRetries - 1) {
        const delay = initialDelay * Math.pow(2, attempt)
        console.log(`⏳ Rate limited, retrying in ${delay/1000}s... (attempt ${attempt + 1}/${maxRetries})`)
        await new Promise(resolve => setTimeout(resolve, delay))
        continue
      }
      throw error
    }
  }
  throw new Error('Max retries exceeded')
}

// Usage in cancelAllOrders
await retryWithBackoff(() => driftClient.cancelOrders(...))

Without this, order cancellations fail silently during TP1→breakeven order updates, leaving ghost orders that cause incorrect fills.

Dual Stop System (USE_DUAL_STOPS=true):

// Soft stop: TRIGGER_LIMIT at -1.5% (avoids wicks)
// Hard stop: TRIGGER_MARKET at -2.5% (guarantees exit)

Order types:

• Entry: MARKET (immediate execution)
• TP1/TP2: LIMIT reduce-only orders
• Soft SL: TRIGGER_LIMIT reduce-only
• Hard SL: TRIGGER_MARKET reduce-only

6. Database (lib/database/trades.ts + prisma/schema.prisma)

Purpose: PostgreSQL via Prisma ORM for trade history and analytics

Models: Trade, PriceUpdate, SystemEvent, DailyStats, BlockedSignal

Singleton pattern: Use getPrismaClient() - never instantiate PrismaClient directly

Key functions:

• createTrade() - Save trade after execution (includes dual stop TX signatures + signalQualityScore)
• updateTradeExit() - Record exit with P&L
• addPriceUpdate() - Track price movements (called by Position Manager)
• getTradeStats() - Win rate, profit factor, avg win/loss
• getLastTrade() - Fetch most recent trade for analytics dashboard
• createBlockedSignal() - Save blocked signals for data-driven optimization analysis
• getRecentBlockedSignals() - Query recent blocked signals
• getBlockedSignalsForAnalysis() - Fetch signals needing price analysis (future automation)

Important fields:

• signalQualityScore (Int?) - 0-100 score for data-driven optimization
• signalQualityVersion (String?) - Tracks which scoring logic was used ('v1', 'v2', 'v3', 'v4')
  • v1: Original logic (price position < 5% threshold)
  • v2: Added volume compensation for low ADX (2025-11-07)
  • v3: Stricter breakdown requirements: positions < 15% require (ADX > 18 AND volume > 1.2x) OR (RSI < 35 for shorts / RSI > 60 for longs)
  • v4: CURRENT - Blocked signals tracking enabled for data-driven threshold optimization (2025-11-11)
  • All new trades tagged with current version for comparative analysis
• maxFavorableExcursion / maxAdverseExcursion - Track best/worst P&L during trade lifetime
• maxFavorablePrice / maxAdversePrice - Track prices at MFE/MAE points
• configSnapshot (Json) - Stores Position Manager state for crash recovery
• atr, adx, rsi, volumeRatio, pricePosition - Context metrics from TradingView

BlockedSignal model fields (NEW):

• Signal metrics: atr, adx, rsi, volumeRatio, pricePosition, timeframe
• Quality scoring: signalQualityScore, signalQualityVersion, scoreBreakdown (JSON), minScoreRequired
• Block tracking: blockReason (QUALITY_SCORE_TOO_LOW, COOLDOWN_PERIOD, HOURLY_TRADE_LIMIT, etc.), blockDetails
• Future analysis: priceAfter1/5/15/30Min, wouldHitTP1/TP2/SL, analysisComplete
• Automatically saved by check-risk endpoint when signals are blocked
• Enables data-driven optimization: collect 10-20 blocked signals → analyze patterns → adjust thresholds

Per-symbol functions:

• getLastTradeTimeForSymbol(symbol) - Get last trade time for specific coin (enables per-symbol cooldown)
• Each coin (SOL/ETH/BTC) has independent cooldown timer to avoid missed opportunities

Configuration System

Three-layer merge:

1. DEFAULT_TRADING_CONFIG (config/trading.ts)
2. Environment variables (.env) via getConfigFromEnv()
3. Runtime overrides via getMergedConfig(overrides)

Always use: getMergedConfig() to get final config - never read env vars directly in business logic

Per-symbol position sizing: Use getPositionSizeForSymbol(symbol, config) which returns { size, leverage, enabled }

const { size, leverage, enabled } = getPositionSizeForSymbol('SOL-PERP', config)
if (!enabled) {
  return NextResponse.json({ success: false, error: 'Symbol trading disabled' }, { status: 400 })
}

Symbol normalization: TradingView sends "SOLUSDT" → must convert to "SOL-PERP" for Drift

const driftSymbol = normalizeTradingViewSymbol(body.symbol)

API Endpoints Architecture

Authentication: All /api/trading/* endpoints (except /test) require Authorization: Bearer API_SECRET_KEY

Pattern: Each endpoint follows same flow:

1. Auth check
2. Get config via getMergedConfig()
3. Initialize Drift service
4. Check account health
5. Execute operation
6. Save to database
7. Add to Position Manager if applicable

Key endpoints:

• /api/trading/execute - Main entry point from n8n (production, requires auth), auto-caches market data
• /api/trading/check-risk - Pre-execution validation (duplicate check, quality score, per-symbol cooldown, rate limits, symbol enabled check, saves blocked signals automatically)
• /api/trading/test - Test trades from settings UI (no auth required, respects symbol enable/disable)
• /api/trading/close - Manual position closing (requires symbol normalization)
• /api/trading/cancel-orders - Manual order cleanup (for stuck/ghost orders after rate limit failures)
• /api/trading/positions - Query open positions from Drift
• /api/trading/market-data - Webhook for TradingView market data updates (GET for debug, POST for data)
• /api/settings - Get/update config (writes to .env file, includes per-symbol settings)
• /api/analytics/last-trade - Fetch most recent trade details for dashboard (includes quality score)
• /api/analytics/reentry-check - Validate manual re-entry with fresh TradingView data + recent performance
• /api/analytics/version-comparison - Compare performance across signal quality logic versions (v1/v2/v3/v4)
• /api/restart - Create restart flag for watch-restart.sh script

Critical Workflows

Execute Trade (Production)

TradingView alert → n8n Parse Signal Enhanced (extracts metrics + timeframe)
  ↓ /api/trading/check-risk [validates quality score ≥60, checks duplicates, per-symbol cooldown]
  ↓ /api/trading/execute
  ↓ normalize symbol (SOLUSDT → SOL-PERP)
  ↓ getMergedConfig()
  ↓ getPositionSizeForSymbol() [check if symbol enabled + get sizing]
  ↓ openPosition() [MARKET order]
  ↓ calculate dual stop prices if enabled
  ↓ placeExitOrders() [on-chain TP1/TP2/SL orders]
  ↓ scoreSignalQuality({ ..., timeframe }) [compute 0-100 score with timeframe-aware thresholds]
  ↓ createTrade() [save to database with signalQualityScore]
  ↓ positionManager.addTrade() [start monitoring]

Position Monitoring Loop

Position Manager every 2s:
  ↓ Verify on-chain position still exists (detect external closures)
  ↓ getPythPriceMonitor().getLatestPrice()
  ↓ Calculate current P&L and update MAE/MFE metrics
  ↓ Check emergency stop (-2%) → closePosition(100%)
  ↓ Check SL hit → closePosition(100%)
  ↓ Check TP1 hit → closePosition(75%), cancelAllOrders(), placeExitOrders() with SL at breakeven
  ↓ Check profit lock trigger (+1.2%) → move SL to +configured%
  ↓ Check TP2 hit → closePosition(80% of remaining), activate runner
  ↓ Check trailing stop (if runner active) → adjust SL dynamically based on peakPrice
  ↓ addPriceUpdate() [save to database every N checks]
  ↓ saveTradeState() [persist Position Manager state + MAE/MFE for crash recovery]

Settings Update

Web UI → /api/settings POST
  ↓ Validate new settings
  ↓ Write to .env file using string replacement
  ↓ Return success
  ↓ User clicks "Restart Bot" → /api/restart
  ↓ Creates /tmp/trading-bot-restart.flag
  ↓ watch-restart.sh detects flag
  ↓ Executes: docker restart trading-bot-v4

Docker Context

Multi-stage build: deps → builder → runner (Node 20 Alpine)

Critical Dockerfile steps:

1. Install deps with npm install --production
2. Copy source and npx prisma generate (MUST happen before build)
3. npm run build (Next.js standalone output)
4. Runner stage copies standalone + static + node_modules + Prisma client

Container networking:

• External: trading-bot-v4 on port 3001
• Internal: Next.js on port 3000
• Database: trading-bot-postgres on 172.28.0.0/16 network

DATABASE_URL caveat: Use trading-bot-postgres (container name) in .env for runtime, but localhost:5432 for Prisma CLI migrations from host

Project-Specific Patterns

1. Singleton Services

Never create multiple instances - always use getter functions:

const driftService = await initializeDriftService() // NOT: new DriftService()
const positionManager = getPositionManager()        // NOT: new PositionManager()
const prisma = getPrismaClient()                     // NOT: new PrismaClient()

2. Price Calculations

Direction matters for long vs short:

function calculatePrice(entry: number, percent: number, direction: 'long' | 'short') {
  if (direction === 'long') {
    return entry * (1 + percent / 100)  // Long: +1% = higher price
  } else {
    return entry * (1 - percent / 100)  // Short: +1% = lower price
  }
}

3. Error Handling

Database failures should not fail trades - always wrap in try/catch:

try {
  await createTrade(params)
  console.log('💾 Trade saved to database')
} catch (dbError) {
  console.error('❌ Failed to save trade:', dbError)
  // Don't fail the trade if database save fails
}

4. Reduce-Only Orders

All exit orders MUST be reduce-only (can only close, not open positions):

const orderParams = {
  reduceOnly: true,  // CRITICAL for TP/SL orders
  // ... other params
}

Testing Commands

# Local development
npm run dev

# Build production
npm run build && npm start

# Docker build and restart
docker compose build trading-bot
docker compose up -d --force-recreate trading-bot
docker logs -f trading-bot-v4

# Database operations
npx prisma generate                                    # Generate client
DATABASE_URL="postgresql://...@localhost:5432/..." npx prisma migrate dev
docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c "\dt"

# Test trade from UI
# Go to http://localhost:3001/settings
# Click "Test LONG" or "Test SHORT"

SQL Analysis Queries

Essential queries for monitoring signal quality and blocked signals. Run via:

docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c "YOUR_QUERY"

Phase 1: Monitor Data Collection Progress

-- Check blocked signals count (target: 10-20 for Phase 2)
SELECT COUNT(*) as total_blocked FROM "BlockedSignal";

-- Score distribution of blocked signals
SELECT 
  CASE 
    WHEN signalQualityScore >= 60 THEN '60-64 (Close Call)'
    WHEN signalQualityScore >= 55 THEN '55-59 (Marginal)'
    WHEN signalQualityScore >= 50 THEN '50-54 (Weak)'
    ELSE '0-49 (Very Weak)'
  END as tier,
  COUNT(*) as count,
  ROUND(AVG(signalQualityScore)::numeric, 1) as avg_score
FROM "BlockedSignal"
WHERE blockReason = 'QUALITY_SCORE_TOO_LOW'
GROUP BY tier
ORDER BY MIN(signalQualityScore) DESC;

-- Recent blocked signals with full details
SELECT 
  symbol,
  direction,
  signalQualityScore as score,
  ROUND(adx::numeric, 1) as adx,
  ROUND(atr::numeric, 2) as atr,
  ROUND(pricePosition::numeric, 1) as pos,
  ROUND(volumeRatio::numeric, 2) as vol,
  blockReason,
  TO_CHAR(createdAt, 'MM-DD HH24:MI') as time
FROM "BlockedSignal"
ORDER BY createdAt DESC
LIMIT 10;

Phase 2: Compare Blocked vs Executed Trades

-- Compare executed trades in 60-69 score range
SELECT 
  signalQualityScore as score,
  COUNT(*) as trades,
  ROUND(AVG(realizedPnL)::numeric, 2) as avg_pnl,
  ROUND(SUM(realizedPnL)::numeric, 2) as total_pnl,
  ROUND(100.0 * SUM(CASE WHEN realizedPnL > 0 THEN 1 ELSE 0 END) / COUNT(*)::numeric, 1) as win_rate
FROM "Trade"
WHERE exitReason IS NOT NULL
  AND signalQualityScore BETWEEN 60 AND 69
GROUP BY signalQualityScore
ORDER BY signalQualityScore;

-- Block reason breakdown
SELECT 
  blockReason,
  COUNT(*) as count,
  ROUND(AVG(signalQualityScore)::numeric, 1) as avg_score
FROM "BlockedSignal"
GROUP BY blockReason
ORDER BY count DESC;

Analyze Specific Patterns

-- Blocked signals at range extremes (price position)
SELECT 
  direction,
  signalQualityScore as score,
  ROUND(pricePosition::numeric, 1) as pos,
  ROUND(adx::numeric, 1) as adx,
  ROUND(volumeRatio::numeric, 2) as vol,
  symbol,
  TO_CHAR(createdAt, 'MM-DD HH24:MI') as time
FROM "BlockedSignal"
WHERE blockReason = 'QUALITY_SCORE_TOO_LOW'
  AND (pricePosition < 10 OR pricePosition > 90)
ORDER BY signalQualityScore DESC;

-- ADX distribution in blocked signals
SELECT 
  CASE 
    WHEN adx >= 25 THEN 'Strong (25+)'
    WHEN adx >= 20 THEN 'Moderate (20-25)'
    WHEN adx >= 15 THEN 'Weak (15-20)'
    ELSE 'Very Weak (<15)'
  END as adx_tier,
  COUNT(*) as count,
  ROUND(AVG(signalQualityScore)::numeric, 1) as avg_score
FROM "BlockedSignal"
WHERE blockReason = 'QUALITY_SCORE_TOO_LOW'
  AND adx IS NOT NULL
GROUP BY adx_tier
ORDER BY MIN(adx) DESC;

Usage Pattern:

1. Run "Monitor Data Collection" queries weekly during Phase 1
2. Once 10+ blocked signals collected, run "Compare Blocked vs Executed" queries
3. Use "Analyze Specific Patterns" to identify optimization opportunities
4. Full query reference: BLOCKED_SIGNALS_TRACKING.md

Common Pitfalls

1. Prisma not generated in Docker: Must run npx prisma generate in Dockerfile BEFORE npm run build

2. Wrong DATABASE_URL: Container runtime needs trading-bot-postgres, Prisma CLI from host needs localhost:5432

3. Symbol format mismatch: Always normalize with normalizeTradingViewSymbol() before calling Drift (applies to ALL endpoints including /api/trading/close)

4. Missing reduce-only flag: Exit orders without reduceOnly: true can accidentally open new positions

5. Singleton violations: Creating multiple DriftClient or Position Manager instances causes connection/state issues

6. Type errors with Prisma: The Trade type from Prisma is only available AFTER npx prisma generate - use explicit types or // @ts-ignore carefully

7. Quality score duplication: Signal quality calculation exists in BOTH check-risk and execute endpoints - keep logic synchronized

8. TP2-as-Runner configuration:

   • takeProfit2SizePercent: 0 means "TP2 activates trailing stop, no position close"
   • This creates runner of remaining % after TP1 (default 25%, configurable via TAKE_PROFIT_1_SIZE_PERCENT)
   • TAKE_PROFIT_2_PERCENT=0.7 sets TP2 trigger price, TAKE_PROFIT_2_SIZE_PERCENT should be 0
   • Settings UI correctly shows "TP2 activates trailing stop" with dynamic runner % calculation

9. P&L calculation CRITICAL: Use actual entry vs exit price calculation, not SDK values:

const profitPercent = this.calculateProfitPercent(trade.entryPrice, exitPrice, trade.direction)
const actualRealizedPnL = (closedSizeUSD * profitPercent) / 100
trade.realizedPnL += actualRealizedPnL  // NOT: result.realizedPnL from SDK

10. Transaction confirmation CRITICAL: Both openPosition() AND closePosition() MUST call connection.confirmTransaction() after placePerpOrder(). Without this, the SDK returns transaction signatures that aren't confirmed on-chain, causing "phantom trades" or "phantom closes". Always check confirmation.value.err before proceeding.

11. Execution order matters: When creating trades via API endpoints, the order MUST be:

    1. Open position + place exit orders
    2. Save to database (createTrade())
    3. Add to Position Manager (positionManager.addTrade())

    If Position Manager is added before database save, race conditions occur where monitoring checks before the trade exists in DB.

12. New trade grace period: Position Manager skips "external closure" detection for trades <30 seconds old because Drift positions take 5-10 seconds to propagate after opening. Without this grace period, new positions are immediately detected as "closed externally" and cancelled.

13. Drift minimum position sizes: Actual minimums differ from documentation:

    • SOL-PERP: 0.1 SOL (~$5-15 depending on price)
    • ETH-PERP: 0.01 ETH (~$38-40 at $4000/ETH)
    • BTC-PERP: 0.0001 BTC (~$10-12 at $100k/BTC)

    Always calculate: minOrderSize × currentPrice must exceed Drift's $4 minimum. Add buffer for price movement.

14. Exit reason detection bug: Position Manager was using current price to determine exit reason, but on-chain orders filled at a DIFFERENT price in the past. Now uses trade.tp1Hit / trade.tp2Hit flags and realized P&L to correctly identify whether TP1, TP2, or SL triggered. Prevents profitable trades being mislabeled as "SL" exits.

15. Per-symbol cooldown: Cooldown period is per-symbol, NOT global. ETH trade at 10:00 does NOT block SOL trade at 10:01. Each coin (SOL/ETH/BTC) has independent cooldown timer to avoid missing opportunities on different assets.

16. Timeframe-aware scoring crucial: Signal quality thresholds MUST adjust for 5min vs higher timeframes:

    • 5min charts naturally have lower ADX (12-22 healthy) and ATR (0.2-0.7% healthy) than daily charts
    • Without timeframe awareness, valid 5min breakouts get blocked as "low quality"
    • Anti-chop filter applies -20 points for extreme sideways regardless of timeframe
    • Always pass timeframe parameter from TradingView alerts to scoreSignalQuality()

17. Price position chasing causes flip-flops: Opening longs at 90%+ range or shorts at <10% range reliably loses money:

    • Database analysis showed overnight flip-flop losses all had price position 9-94% (chasing extremes)
    • These trades had valid ADX (16-18) but entered at worst possible time
    • Quality scoring now penalizes -15 to -30 points for range extremes
    • Prevents rapid reversals when price is already overextended

18. TradingView ADX minimum for 5min: Set ADX filter to 15 (not 20+) in TradingView alerts for 5min charts:

    • Higher timeframes can use ADX 20+ for strong trends
    • 5min charts need lower threshold to catch valid breakouts
    • Bot's quality scoring provides second-layer filtering with context-aware metrics
    • Two-stage filtering (TradingView + bot) prevents both overtrading and missing valid signals

19. Prisma Decimal type handling: Raw SQL queries return Prisma Decimal objects, not plain numbers:

    • Use any type for numeric fields in $queryRaw results: total_pnl: any
    • Convert with Number() before returning to frontend: totalPnL: Number(stat.total_pnl) || 0
    • Frontend uses .toFixed() which doesn't exist on Decimal objects
    • Applies to all aggregations: SUM(), AVG(), ROUND() - all return Decimal types
    • Example: /api/analytics/version-comparison converts all numeric fields

20. ATR-based trailing stop implementation (Nov 11, 2025): Runner system was using FIXED 0.3% trailing, causing immediate stops:

    • Problem: At $168 SOL, 0.3% = $0.50 wiggle room. Trades with +7-9% MFE exited for losses.
    • Fix: trailingDistancePercent = (atrAtEntry / currentPrice * 100) × trailingStopAtrMultiplier
    • Config: TRAILING_STOP_ATR_MULTIPLIER=1.5, MIN=0.25%, MAX=0.9%, ACTIVATION=0.5%
    • Typical improvement: 0.45% ATR × 1.5 = 0.675% trail ($1.13 vs $0.50 = 2.26x more room)
    • Fallback: If atrAtEntry unavailable, uses clamped legacy trailingStopPercent
    • Log verification: Look for "📊 ATR-based trailing: 0.0045 (0.52%) × 1.5x = 0.78%" messages
    • ActiveTrade interface: Must include atrAtEntry?: number field for calculation
    • See ATR_TRAILING_STOP_FIX.md for full details and database analysis

File Conventions

• API routes: app/api/[feature]/[action]/route.ts (Next.js 15 App Router)
• Services: lib/[service]/[module].ts (drift, pyth, trading, database)
• Config: Single source in config/trading.ts with env merging
• Types: Define interfaces in same file as implementation (not separate types directory)
• Console logs: Use emojis for visual scanning: 🎯 🚀 ✅ ❌ 💰 📊 🛡️

Re-Entry Analytics System (Phase 1)

Purpose: Validate manual Telegram trades using fresh TradingView data + recent performance analysis

Components:

1. Market Data Cache (lib/trading/market-data-cache.ts)

   • Singleton service storing TradingView metrics
   • 5-minute expiry on cached data
   • Tracks: ATR, ADX, RSI, volume ratio, price position, timeframe

2. Market Data Webhook (app/api/trading/market-data/route.ts)

   • Receives TradingView alerts every 1-5 minutes
   • POST: Updates cache with fresh metrics
   • GET: View cached data (debugging)

3. Re-Entry Check Endpoint (app/api/analytics/reentry-check/route.ts)

   • Validates manual trade requests
   • Uses fresh TradingView data if available (<5min old)
   • Falls back to historical metrics from last trade
   • Scores signal quality + applies performance modifiers:
     • -20 points if last 3 trades lost money (avgPnL < -5%)
     • +10 points if last 3 trades won (avgPnL > +5%, WR >= 66%)
     • -5 points for stale data, -10 points for no data
   • Minimum score: 55 (vs 60 for new signals)

4. Auto-Caching (app/api/trading/execute/route.ts)

   • Every trade signal from TradingView auto-caches metrics
   • Ensures fresh data available for manual re-entries

5. Telegram Integration (telegram_command_bot.py)

   • Calls /api/analytics/reentry-check before executing manual trades
   • Shows data freshness ("✅ FRESH 23s old" vs "⚠️ Historical")
   • Blocks low-quality re-entries unless --force flag used
   • Fail-open: Proceeds if analytics check fails

User Flow:

User: "long sol"
  ↓ Check cache for SOL-PERP
  ↓ Fresh data? → Use real TradingView metrics
  ↓ Stale/missing? → Use historical + penalty
  ↓ Score quality + recent performance
  ↓ Score >= 55? → Execute
  ↓ Score < 55? → Block (unless --force)

TradingView Setup: Create alerts that fire every 1-5 minutes with this webhook message:

{
  "action": "market_data",
  "symbol": "{{ticker}}",
  "timeframe": "{{interval}}",
  "atr": {{ta.atr(14)}},
  "adx": {{ta.dmi(14, 14)}},
  "rsi": {{ta.rsi(14)}},
  "volumeRatio": {{volume / ta.sma(volume, 20)}},
  "pricePosition": {{(close - ta.lowest(low, 100)) / (ta.highest(high, 100) - ta.lowest(low, 100)) * 100}},
  "currentPrice": {{close}}
}

Webhook URL: https://your-domain.com/api/trading/market-data

Per-Symbol Trading Controls

Purpose: Independent enable/disable toggles and position sizing for SOL and ETH to support different trading strategies (e.g., ETH for data collection at minimal size, SOL for profit generation).

Configuration Priority:

1. Per-symbol ENV vars (highest priority)
   • SOLANA_ENABLED, SOLANA_POSITION_SIZE, SOLANA_LEVERAGE
   • ETHEREUM_ENABLED, ETHEREUM_POSITION_SIZE, ETHEREUM_LEVERAGE
2. Market-specific config (from MARKET_CONFIGS in config/trading.ts)
3. Global ENV vars (fallback for BTC and other symbols)
   • MAX_POSITION_SIZE_USD, LEVERAGE
4. Default config (lowest priority)

Settings UI: app/settings/page.tsx has dedicated sections:

• 💎 Solana section: Toggle + position size + leverage + risk calculator
• ⚡ Ethereum section: Toggle + position size + leverage + risk calculator
• 💰 Global fallback: For BTC-PERP and future symbols

Example usage:

// In execute/test endpoints
const { size, leverage, enabled } = getPositionSizeForSymbol(driftSymbol, config)
if (!enabled) {
  return NextResponse.json({
    success: false,
    error: 'Symbol trading disabled'
  }, { status: 400 })
}

Test buttons: Settings UI has symbol-specific test buttons:

• 💎 Test SOL LONG/SHORT (disabled when SOLANA_ENABLED=false)
• ⚡ Test ETH LONG/SHORT (disabled when ETHEREUM_ENABLED=false)

When Making Changes

1. Adding new config: Update DEFAULT_TRADING_CONFIG + getConfigFromEnv() + .env file
2. Adding database fields: Update prisma/schema.prisma → npx prisma migrate dev → npx prisma generate → rebuild Docker
3. Changing order logic: Test with DRY_RUN=true first, use small position sizes ($10)
4. API endpoint changes: Update both endpoint + corresponding n8n workflow JSON (Check Risk and Execute Trade nodes)
5. Docker changes: Rebuild with docker compose build trading-bot then restart container
6. Modifying quality score logic: Update BOTH /api/trading/check-risk and /api/trading/execute endpoints, ensure timeframe-aware thresholds are synchronized
7. Exit strategy changes: Modify Position Manager logic + update on-chain order placement in placeExitOrders()
8. TradingView alert changes: Ensure alerts pass timeframe field (e.g., "timeframe": "5") to enable proper signal quality scoring

Development Roadmap

See SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md for systematic signal quality improvements:

• Phase 1 (🔄 IN PROGRESS): Collect 10-20 blocked signals with quality scores (1-2 weeks)
• Phase 2 (🔜 NEXT): Analyze patterns and make data-driven threshold decisions
• Phase 3 (🎯 FUTURE): Implement dual-threshold system or other optimizations based on data
• Phase 4 (🤖 FUTURE): Automated price analysis for blocked signals
• Phase 5 (🧠 DISTANT): ML-based scoring weight optimization

See POSITION_SCALING_ROADMAP.md for planned position management optimizations:

• Phase 1 (✅ COMPLETE): Collect data with quality scores (20-50 trades needed)
• Phase 2: ATR-based dynamic targets (adapt to volatility)
• Phase 3: Signal quality-based scaling (high quality = larger runners)
• Phase 4: Direction-based optimization (shorts vs longs have different performance)
• Phase 5 (✅ COMPLETE): TP2-as-runner system implemented - configurable runner (default 25%, adjustable via TAKE_PROFIT_1_SIZE_PERCENT) with ATR-based trailing stop
• Phase 6: ML-based exit prediction (future)

Recent Implementation: TP2-as-runner system provides 5x larger runner (default 25% vs old 5%) for better profit capture on extended moves. When TP2 price is hit, trailing stop activates on full remaining position instead of closing partial amount. Runner size is configurable (100% - TP1 close %).

Blocked Signals Tracking (Nov 11, 2025): System now automatically saves all blocked signals to database for data-driven optimization. See BLOCKED_SIGNALS_TRACKING.md for SQL queries and analysis workflows.

Data-driven approach: Each phase requires validation through SQL analysis before implementation. No premature optimization.

Signal Quality Version Tracking: Database tracks signalQualityVersion field to compare algorithm performance:

• Analytics dashboard shows version comparison: trades, win rate, P&L, extreme position stats
• v4 (current) includes blocked signals tracking for data-driven optimization
• Focus on extreme positions (< 15% range) - v3 aimed to reduce losses from weak ADX entries
• SQL queries in docs/analysis/SIGNAL_QUALITY_VERSION_ANALYSIS.sql for deep-dive analysis
• Need 20+ trades per version before meaningful comparison

Financial Roadmap Integration: All technical improvements must align with current phase objectives (see top of document):

• Phase 1 (CURRENT): Prove system works, compound aggressively, 60%+ win rate mandatory
• Phase 2-3: Transition to sustainable growth while funding withdrawals
• Phase 4+: Scale capital while reducing risk progressively
• See TRADING_GOALS.md for complete 8-phase plan ($106 → $1M+)
• SQL queries in docs/analysis/SIGNAL_QUALITY_VERSION_ANALYSIS.sql for deep-dive analysis
• Need 20+ trades per version before meaningful comparison

Blocked Signals Analysis: See BLOCKED_SIGNALS_TRACKING.md for:

• SQL queries to analyze blocked signal patterns
• Score distribution and metric analysis
• Comparison with executed trades at similar quality levels
• Future automation of price tracking (would TP1/TP2/SL have hit?)

Integration Points

• n8n: Expects exact response format from /api/trading/execute (see n8n-complete-workflow.json)
• Drift Protocol: Uses SDK v2.75.0 - check docs at docs.drift.trade for API changes
• Pyth Network: WebSocket + HTTP fallback for price feeds (handles reconnection)
• PostgreSQL: Version 16-alpine, must be running before bot starts

---

Key Mental Model: Think of this as two parallel systems (on-chain orders + software monitoring) working together. The Position Manager is the "backup brain" that constantly watches and acts if on-chain orders fail. Both write to the same database for complete trade history.