docs: Update copilot-instructions.md with configurable quality score threshold

- Updated Signal Quality System to reflect MIN_SIGNAL_QUALITY_SCORE is configurable (default: 65) - Added critical pitfall #7: Never use hardcoded config values in endpoints - Emphasized settings page can modify minSignalQualityScore dynamically - Renumbered remaining pitfalls for clarity
2025-11-10 13:00:40 +01:00
parent d2fbd125a0
commit d20190c5b0
1 changed files with 24 additions and 142 deletions
--- a/.github/copilot-instructions.md
+++ b/.github/copilot-instructions.md
@@ -19,7 +19,7 @@
 - BTC and other symbols fall back to global settings (`MAX_POSITION_SIZE_USD`, `LEVERAGE`)
 - **Priority:** Per-symbol ENV → Market config → Global ENV → Defaults

-**Signal Quality System:** Filters trades based on 5 metrics (ATR, ADX, RSI, volumeRatio, pricePosition) scored 0-100. Only trades scoring 60+ are executed. Scores stored in database for future optimization.
+**Signal Quality System:** Filters trades based on 5 metrics (ATR, ADX, RSI, volumeRatio, pricePosition) scored 0-100. Minimum score threshold configurable via `MIN_SIGNAL_QUALITY_SCORE` env var (default: 65, editable via settings page). Scores stored in database for future optimization.

 **Timeframe-Aware Scoring:** Signal quality thresholds adjust based on timeframe (5min vs daily):
 - 5min: ADX 12+ trending (vs 18+ for daily), ATR 0.2-0.7% healthy (vs 0.4%+ for daily)
@@ -30,13 +30,6 @@

 **Manual Trading via Telegram:** Send plain-text messages like `long sol`, `short eth`, `long btc` to open positions instantly (bypasses n8n, calls `/api/trading/execute` directly with preset healthy metrics).

-**Re-Entry Analytics System:** Manual trades are validated before execution using fresh TradingView data:
- Market data cached from TradingView signals (5min expiry)
- `/api/analytics/reentry-check` scores re-entry based on fresh metrics + recent performance
- Telegram bot blocks low-quality re-entries unless `--force` flag used
- Uses real TradingView ADX/ATR/RSI when available, falls back to historical data
- Penalty for recent losing trades, bonus for winning streaks
-
 ## Critical Components

 ### 1. Signal Quality Scoring (`lib/trading/signal-quality.ts`)
@@ -62,7 +55,7 @@ scoreSignalQuality({

 **Key behaviors:**
 - Returns score 0-100 and detailed breakdown object
- Minimum score 60 required to execute trade
+- Minimum score threshold configurable via `config.minSignalQualityScore` (default: 65)
 - Called by both `/api/trading/check-risk` and `/api/trading/execute`
 - Scores saved to database for post-trade analysis

@@ -94,23 +87,18 @@ await positionManager.addTrade(activeTrade)
 **Manual trade commands via plain text:**
 ```python
 # User sends plain text message (not slash commands)
-"long sol"          → Validates via analytics, then opens SOL-PERP long
-"short eth"         → Validates via analytics, then opens ETH-PERP short
-"long btc --force"  → Skips analytics validation, opens BTC-PERP long immediately
+"long sol"   → Opens SOL-PERP long position
+"short eth"  → Opens ETH-PERP short position
+"long btc"   → Opens BTC-PERP long position
 ```

 **Key behaviors:**
 - MessageHandler processes all text messages (not just commands)
 - Maps user-friendly symbols (sol, eth, btc) to Drift format (SOL-PERP, etc.)
- **Analytics validation:** Calls `/api/analytics/reentry-check` before execution
-  - Blocks trades with score <55 unless `--force` flag used
-  - Uses fresh TradingView data (<5min old) when available
-  - Falls back to historical metrics with penalty
-  - Considers recent trade performance (last 3 trades)
- Calls `/api/trading/execute` directly with preset healthy metrics (ATR=0.45, ADX=32, RSI=58/42)
+- Calls `/api/trading/execute` directly with preset healthy metrics (ATR=1.0, ADX=25, RSI=50, volumeRatio=1.2)
 - Bypasses n8n workflow and TradingView requirements
 - 60-second timeout for API calls
- Responds with trade confirmation or analytics rejection message
+- Responds with trade confirmation or error message

 **Status command:**
 ```python
@@ -136,7 +124,6 @@ const health = await driftService.getAccountHealth()
 - `openPosition()` - Opens market position with transaction confirmation
 - `closePosition()` - Closes position with transaction confirmation
 - `placeExitOrders()` - Places TP/SL orders on-chain
- `cancelAllOrders()` - Cancels all reduce-only orders for a market

 **CRITICAL: Transaction Confirmation Pattern**
 Both `openPosition()` and `closePosition()` MUST confirm transactions on-chain:
@@ -153,46 +140,6 @@ console.log('✅ Transaction confirmed on-chain')
 ```
 Without this, the SDK returns signatures for transactions that never execute, causing phantom trades/closes.

-**CRITICAL: Drift SDK position.size is USD, not tokens**
-The Drift SDK returns `position.size` as USD notional value, NOT token quantity:
-```typescript
-// WRONG: Multiply by price (inflates by 156x for SOL at $157)
-const positionSizeUSD = position.size * currentPrice
-
-// CORRECT: Use directly as USD value
-const positionSizeUSD = Math.abs(position.size)
-```
-This affects Position Manager's TP1 detection - if calculated incorrectly, TP1 will never trigger because expected size won't match actual size.
-
-**Solana RPC Rate Limiting with Exponential Backoff**
-Solana RPC endpoints return 429 errors under load. Always use retry logic for order operations:
-```typescript
-export async function retryWithBackoff<T>(
-  operation: () => Promise<T>,
-  maxRetries: number = 3,
-  initialDelay: number = 2000
-): Promise<T> {
-  for (let attempt = 0; attempt < maxRetries; attempt++) {
-    try {
-      return await operation()
-    } catch (error: any) {
-      if (error?.message?.includes('429') && attempt < maxRetries - 1) {
-        const delay = initialDelay * Math.pow(2, attempt)
-        console.log(`⏳ Rate limited, retrying in ${delay/1000}s... (attempt ${attempt + 1}/${maxRetries})`)
-        await new Promise(resolve => setTimeout(resolve, delay))
-        continue
-      }
-      throw error
-    }
-  }
-  throw new Error('Max retries exceeded')
-}
-
-// Usage in cancelAllOrders
-await retryWithBackoff(() => driftClient.cancelOrders(...))
-```
-Without this, order cancellations fail silently during TP1→breakeven order updates, leaving ghost orders that cause incorrect fills.
-
 **Dual Stop System** (USE_DUAL_STOPS=true):
 ```typescript
 // Soft stop: TRIGGER_LIMIT at -1.5% (avoids wicks)
@@ -271,16 +218,13 @@ const driftSymbol = normalizeTradingViewSymbol(body.symbol)
 7. Add to Position Manager if applicable

 **Key endpoints:**
- `/api/trading/execute` - Main entry point from n8n (production, requires auth), **auto-caches market data**
+- `/api/trading/execute` - Main entry point from n8n (production, requires auth)
 - `/api/trading/check-risk` - Pre-execution validation (duplicate check, quality score, **per-symbol cooldown**, rate limits, **symbol enabled check**)
 - `/api/trading/test` - Test trades from settings UI (no auth required, **respects symbol enable/disable**)
- `/api/trading/close` - Manual position closing (requires symbol normalization)
- `/api/trading/cancel-orders` - **Manual order cleanup** (for stuck/ghost orders after rate limit failures)
+- `/api/trading/close` - Manual position closing
 - `/api/trading/positions` - Query open positions from Drift
- `/api/trading/market-data` - Webhook for TradingView market data updates (GET for debug, POST for data)
 - `/api/settings` - Get/update config (writes to .env file, **includes per-symbol settings**)
 - `/api/analytics/last-trade` - Fetch most recent trade details for dashboard (includes quality score)
- `/api/analytics/reentry-check` - **Validate manual re-entry** with fresh TradingView data + recent performance
 - `/api/analytics/version-comparison` - Compare performance across signal quality logic versions (v1/v2/v3)
 - `/api/restart` - Create restart flag for watch-restart.sh script

@@ -428,62 +372,64 @@ docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c "\dt"

 6. **Type errors with Prisma:** The Trade type from Prisma is only available AFTER `npx prisma generate` - use explicit types or `// @ts-ignore` carefully

-7. **Quality score duplication:** Signal quality calculation exists in BOTH `check-risk` and `execute` endpoints - keep logic synchronized
+7. **Hardcoded config values:** NEVER use hardcoded values for configurable settings in API endpoints. Always read from `config.minSignalQualityScore` or similar config properties. Settings changed via the UI won't take effect if endpoints use hardcoded values.

-8. **TP2-as-Runner configuration:** 
+8. **Quality score duplication:** Signal quality calculation exists in BOTH `check-risk` and `execute` endpoints - keep logic synchronized
+
+9. **TP2-as-Runner configuration:** 
   - `takeProfit2SizePercent: 0` means "TP2 activates trailing stop, no position close"
   - This creates 25% runner (vs old 5% system) for better profit capture
   - `TAKE_PROFIT_2_PERCENT=0.7` sets TP2 trigger price, `TAKE_PROFIT_2_SIZE_PERCENT` should be 0
   - Settings UI correctly shows "TP2 activates trailing stop" instead of size percentage

-9. **P&L calculation CRITICAL:** Use actual entry vs exit price calculation, not SDK values:
+10. **P&L calculation CRITICAL:** Use actual entry vs exit price calculation, not SDK values:
 ```typescript
 const profitPercent = this.calculateProfitPercent(trade.entryPrice, exitPrice, trade.direction)
 const actualRealizedPnL = (closedSizeUSD * profitPercent) / 100
 trade.realizedPnL += actualRealizedPnL  // NOT: result.realizedPnL from SDK
 ```

-10. **Transaction confirmation CRITICAL:** Both `openPosition()` AND `closePosition()` MUST call `connection.confirmTransaction()` after `placePerpOrder()`. Without this, the SDK returns transaction signatures that aren't confirmed on-chain, causing "phantom trades" or "phantom closes". Always check `confirmation.value.err` before proceeding.
+11. **Transaction confirmation CRITICAL:** Both `openPosition()` AND `closePosition()` MUST call `connection.confirmTransaction()` after `placePerpOrder()`. Without this, the SDK returns transaction signatures that aren't confirmed on-chain, causing "phantom trades" or "phantom closes". Always check `confirmation.value.err` before proceeding.

-11. **Execution order matters:** When creating trades via API endpoints, the order MUST be:
+12. **Execution order matters:** When creating trades via API endpoints, the order MUST be:
    1. Open position + place exit orders
    2. Save to database (`createTrade()`)
    3. Add to Position Manager (`positionManager.addTrade()`)
    
    If Position Manager is added before database save, race conditions occur where monitoring checks before the trade exists in DB.

-12. **New trade grace period:** Position Manager skips "external closure" detection for trades <30 seconds old because Drift positions take 5-10 seconds to propagate after opening. Without this grace period, new positions are immediately detected as "closed externally" and cancelled.
+13. **New trade grace period:** Position Manager skips "external closure" detection for trades <30 seconds old because Drift positions take 5-10 seconds to propagate after opening. Without this grace period, new positions are immediately detected as "closed externally" and cancelled.

-13. **Drift minimum position sizes:** Actual minimums differ from documentation:
+14. **Drift minimum position sizes:** Actual minimums differ from documentation:
    - SOL-PERP: 0.1 SOL (~$5-15 depending on price)
    - ETH-PERP: 0.01 ETH (~$38-40 at $4000/ETH)
    - BTC-PERP: 0.0001 BTC (~$10-12 at $100k/BTC)
    
    Always calculate: `minOrderSize × currentPrice` must exceed Drift's $4 minimum. Add buffer for price movement.

-14. **Exit reason detection bug:** Position Manager was using current price to determine exit reason, but on-chain orders filled at a DIFFERENT price in the past. Now uses `trade.tp1Hit` / `trade.tp2Hit` flags and realized P&L to correctly identify whether TP1, TP2, or SL triggered. Prevents profitable trades being mislabeled as "SL" exits.
+15. **Exit reason detection bug:** Position Manager was using current price to determine exit reason, but on-chain orders filled at a DIFFERENT price in the past. Now uses `trade.tp1Hit` / `trade.tp2Hit` flags and realized P&L to correctly identify whether TP1, TP2, or SL triggered. Prevents profitable trades being mislabeled as "SL" exits.

-15. **Per-symbol cooldown:** Cooldown period is per-symbol, NOT global. ETH trade at 10:00 does NOT block SOL trade at 10:01. Each coin (SOL/ETH/BTC) has independent cooldown timer to avoid missing opportunities on different assets.
+16. **Per-symbol cooldown:** Cooldown period is per-symbol, NOT global. ETH trade at 10:00 does NOT block SOL trade at 10:01. Each coin (SOL/ETH/BTC) has independent cooldown timer to avoid missing opportunities on different assets.

-16. **Timeframe-aware scoring crucial:** Signal quality thresholds MUST adjust for 5min vs higher timeframes:
+17. **Timeframe-aware scoring crucial:** Signal quality thresholds MUST adjust for 5min vs higher timeframes:
    - 5min charts naturally have lower ADX (12-22 healthy) and ATR (0.2-0.7% healthy) than daily charts
    - Without timeframe awareness, valid 5min breakouts get blocked as "low quality"
    - Anti-chop filter applies -20 points for extreme sideways regardless of timeframe
    - Always pass `timeframe` parameter from TradingView alerts to `scoreSignalQuality()`

-17. **Price position chasing causes flip-flops:** Opening longs at 90%+ range or shorts at <10% range reliably loses money:
+18. **Price position chasing causes flip-flops:** Opening longs at 90%+ range or shorts at <10% range reliably loses money:
    - Database analysis showed overnight flip-flop losses all had price position 9-94% (chasing extremes)
    - These trades had valid ADX (16-18) but entered at worst possible time
    - Quality scoring now penalizes -15 to -30 points for range extremes
    - Prevents rapid reversals when price is already overextended

-18. **TradingView ADX minimum for 5min:** Set ADX filter to 15 (not 20+) in TradingView alerts for 5min charts:
+19. **TradingView ADX minimum for 5min:** Set ADX filter to 15 (not 20+) in TradingView alerts for 5min charts:
    - Higher timeframes can use ADX 20+ for strong trends
    - 5min charts need lower threshold to catch valid breakouts
    - Bot's quality scoring provides second-layer filtering with context-aware metrics
    - Two-stage filtering (TradingView + bot) prevents both overtrading and missing valid signals

-19. **Prisma Decimal type handling:** Raw SQL queries return Prisma `Decimal` objects, not plain numbers:
+20. **Prisma Decimal type handling:** Raw SQL queries return Prisma `Decimal` objects, not plain numbers:
    - Use `any` type for numeric fields in `$queryRaw` results: `total_pnl: any`
    - Convert with `Number()` before returning to frontend: `totalPnL: Number(stat.total_pnl) || 0`
    - Frontend uses `.toFixed()` which doesn't exist on Decimal objects
@@ -498,70 +444,6 @@ trade.realizedPnL += actualRealizedPnL  // NOT: result.realizedPnL from SDK
 - **Types:** Define interfaces in same file as implementation (not separate types directory)
 - **Console logs:** Use emojis for visual scanning: 🎯 🚀 ✅ ❌ 💰 📊 🛡️

-## Re-Entry Analytics System (Phase 1)
-
-**Purpose:** Validate manual Telegram trades using fresh TradingView data + recent performance analysis
-
-**Components:**
-1. **Market Data Cache** (`lib/trading/market-data-cache.ts`)
-   - Singleton service storing TradingView metrics
-   - 5-minute expiry on cached data
-   - Tracks: ATR, ADX, RSI, volume ratio, price position, timeframe
-
-2. **Market Data Webhook** (`app/api/trading/market-data/route.ts`)
-   - Receives TradingView alerts every 1-5 minutes
-   - POST: Updates cache with fresh metrics
-   - GET: View cached data (debugging)
-
-3. **Re-Entry Check Endpoint** (`app/api/analytics/reentry-check/route.ts`)
-   - Validates manual trade requests
-   - Uses fresh TradingView data if available (<5min old)
-   - Falls back to historical metrics from last trade
-   - Scores signal quality + applies performance modifiers:
-     - **-20 points** if last 3 trades lost money (avgPnL < -5%)
-     - **+10 points** if last 3 trades won (avgPnL > +5%, WR >= 66%)
-     - **-5 points** for stale data, **-10 points** for no data
-   - Minimum score: 55 (vs 60 for new signals)
-
-4. **Auto-Caching** (`app/api/trading/execute/route.ts`)
-   - Every trade signal from TradingView auto-caches metrics
-   - Ensures fresh data available for manual re-entries
-
-5. **Telegram Integration** (`telegram_command_bot.py`)
-   - Calls `/api/analytics/reentry-check` before executing manual trades
-   - Shows data freshness ("✅ FRESH 23s old" vs "⚠️ Historical")
-   - Blocks low-quality re-entries unless `--force` flag used
-   - Fail-open: Proceeds if analytics check fails
-
-**User Flow:**
-```
-User: "long sol"
-  ↓ Check cache for SOL-PERP
-  ↓ Fresh data? → Use real TradingView metrics
-  ↓ Stale/missing? → Use historical + penalty
-  ↓ Score quality + recent performance
-  ↓ Score >= 55? → Execute
-  ↓ Score < 55? → Block (unless --force)
-```
-
-**TradingView Setup:**
-Create alerts that fire every 1-5 minutes with this webhook message:
-```json
-{
-  "action": "market_data",
-  "symbol": "{{ticker}}",
-  "timeframe": "{{interval}}",
-  "atr": {{ta.atr(14)}},
-  "adx": {{ta.dmi(14, 14)}},
-  "rsi": {{ta.rsi(14)}},
-  "volumeRatio": {{volume / ta.sma(volume, 20)}},
-  "pricePosition": {{(close - ta.lowest(low, 100)) / (ta.highest(high, 100) - ta.lowest(low, 100)) * 100}},
-  "currentPrice": {{close}}
-}
-```
-
-Webhook URL: `https://your-domain.com/api/trading/market-data`
-
 ## Per-Symbol Trading Controls

 **Purpose:** Independent enable/disable toggles and position sizing for SOL and ETH to support different trading strategies (e.g., ETH for data collection at minimal size, SOL for profit generation).