CRITICAL FIX: Use ?? instead of || for tp2SizePercent to allow 0 value

BUG FOUND:
Line 558: tp2SizePercent: config.takeProfit2SizePercent || 100

When config.takeProfit2SizePercent = 0 (TP2-as-runner system), JavaScript's ||
operator treats 0 as falsy and falls back to 100, causing TP2 to close 100%
of remaining position instead of activating trailing stop.

IMPACT:
- On-chain orders placed correctly (line 481 uses ?? correctly)
- Position Manager reads from DB and expects TP2 to close position
- Result: User sees TWO take-profit orders instead of runner system

FIX:
Changed both tp1SizePercent and tp2SizePercent to use ?? operator:
- tp1SizePercent: config.takeProfit1SizePercent ?? 75
- tp2SizePercent: config.takeProfit2SizePercent ?? 0

This allows 0 value to be saved correctly for TP2-as-runner system.

VERIFICATION NEEDED:
Current open SHORT position in database has tp2SizePercent=100 from before
this fix. Next trade will use correct runner system.
This commit is contained in:
mindesbunister
2025-11-10 19:46:03 +01:00
parent 089308a07e
commit c3a053df63
9 changed files with 411 additions and 529 deletions

View File

@@ -19,7 +19,7 @@
- BTC and other symbols fall back to global settings (`MAX_POSITION_SIZE_USD`, `LEVERAGE`)
- **Priority:** Per-symbol ENV → Market config → Global ENV → Defaults
**Signal Quality System:** Filters trades based on 5 metrics (ATR, ADX, RSI, volumeRatio, pricePosition) scored 0-100. Minimum score threshold configurable via `MIN_SIGNAL_QUALITY_SCORE` env var (default: 65, editable via settings page). Scores stored in database for future optimization.
**Signal Quality System:** Filters trades based on 5 metrics (ATR, ADX, RSI, volumeRatio, pricePosition) scored 0-100. Only trades scoring 60+ are executed. Scores stored in database for future optimization.
**Timeframe-Aware Scoring:** Signal quality thresholds adjust based on timeframe (5min vs daily):
- 5min: ADX 12+ trending (vs 18+ for daily), ATR 0.2-0.7% healthy (vs 0.4%+ for daily)
@@ -30,30 +30,12 @@
**Manual Trading via Telegram:** Send plain-text messages like `long sol`, `short eth`, `long btc` to open positions instantly (bypasses n8n, calls `/api/trading/execute` directly with preset healthy metrics).
## Recent Critical Fixes (2024-11-10)
### Runner System - Three Cascading Bugs Fixed
The TP2-as-runner feature was broken by three separate bugs:
1. **P&L Calculation Bug (65x inflation)** - `lib/drift/orders.ts`, `lib/trading/position-manager.ts`
- Calculated P&L on notional ($2,100) instead of collateral ($210)
- Database showed +$1,345, reality was -$806 loss
- Fix: `collateralUSD = notional / leverage`, calculate P&L on collateral
2. **Post-TP1 Logic Bug** - `lib/trading/position-manager.ts` lines 1010-1030
- Placed TP order at TP2 price after TP1 hit (closed position instead of trailing)
- Fix: Check `if (config.takeProfit2SizePercent === 0)` to skip TP orders
3. **JavaScript || Operator Bug** - `app/api/trading/execute/route.ts`, `test/route.ts`
- `config.takeProfit2SizePercent || 100` treated 0 as falsy → returned 100
- Fix: Use `??` (nullish coalescing) instead of `||` for numeric defaults
### Anti-Chop Filter V2 - Range-Bound Detection
- **Problem:** Flip-flop trades in sideways markets (stopped out in 8-24 seconds)
- **Fix:** -25 points when price position <40% AND ADX <25 (both conditions)
- **Location:** `lib/trading/signal-quality.ts` lines 145-165
- **Impact:** Win rate 43.8% → 55.6%, profit per trade +86%
- **Backtest:** Would have blocked all 3 flip-flop trades from today
**Re-Entry Analytics System:** Manual trades are validated before execution using fresh TradingView data:
- Market data cached from TradingView signals (5min expiry)
- `/api/analytics/reentry-check` scores re-entry based on fresh metrics + recent performance
- Telegram bot blocks low-quality re-entries unless `--force` flag used
- Uses real TradingView ADX/ATR/RSI when available, falls back to historical data
- Penalty for recent losing trades, bonus for winning streaks
## Critical Components
@@ -76,14 +58,11 @@ scoreSignalQuality({
**Price position penalties (all timeframes):**
- Long at 90-95%+ range: -15 to -30 points (chasing highs)
- Short at <5-10% range: -15 to -30 points (chasing lows)
- **ANTI-CHOP (v2024-11-10):** Price position <40% + ADX <25 = -25 points (RANGE-BOUND CHOP)
- Prevents flip-flop losses from entering range extremes
- Targets sideways markets where price is low in range but trend is weak
- Backtest: 43.8% → 55.6% win rate, 86% higher profit per trade
- Prevents flip-flop losses from entering range extremes
**Key behaviors:**
- Returns score 0-100 and detailed breakdown object
- Minimum score threshold configurable via `config.minSignalQualityScore` (default: 65)
- Minimum score 60 required to execute trade
- Called by both `/api/trading/check-risk` and `/api/trading/execute`
- Scores saved to database for post-trade analysis
@@ -115,18 +94,23 @@ await positionManager.addTrade(activeTrade)
**Manual trade commands via plain text:**
```python
# User sends plain text message (not slash commands)
"long sol" Opens SOL-PERP long position
"short eth" Opens ETH-PERP short position
"long btc" Opens BTC-PERP long position
"long sol" Validates via analytics, then opens SOL-PERP long
"short eth" Validates via analytics, then opens ETH-PERP short
"long btc --force" Skips analytics validation, opens BTC-PERP long immediately
```
**Key behaviors:**
- MessageHandler processes all text messages (not just commands)
- Maps user-friendly symbols (sol, eth, btc) to Drift format (SOL-PERP, etc.)
- Calls `/api/trading/execute` directly with preset healthy metrics (ATR=1.0, ADX=25, RSI=50, volumeRatio=1.2)
- **Analytics validation:** Calls `/api/analytics/reentry-check` before execution
- Blocks trades with score <55 unless `--force` flag used
- Uses fresh TradingView data (<5min old) when available
- Falls back to historical metrics with penalty
- Considers recent trade performance (last 3 trades)
- Calls `/api/trading/execute` directly with preset healthy metrics (ATR=0.45, ADX=32, RSI=58/42)
- Bypasses n8n workflow and TradingView requirements
- 60-second timeout for API calls
- Responds with trade confirmation or error message
- Responds with trade confirmation or analytics rejection message
**Status command:**
```python
@@ -152,6 +136,7 @@ const health = await driftService.getAccountHealth()
- `openPosition()` - Opens market position with transaction confirmation
- `closePosition()` - Closes position with transaction confirmation
- `placeExitOrders()` - Places TP/SL orders on-chain
- `cancelAllOrders()` - Cancels all reduce-only orders for a market
**CRITICAL: Transaction Confirmation Pattern**
Both `openPosition()` and `closePosition()` MUST confirm transactions on-chain:
@@ -168,6 +153,46 @@ console.log('✅ Transaction confirmed on-chain')
```
Without this, the SDK returns signatures for transactions that never execute, causing phantom trades/closes.
**CRITICAL: Drift SDK position.size is USD, not tokens**
The Drift SDK returns `position.size` as USD notional value, NOT token quantity:
```typescript
// WRONG: Multiply by price (inflates by 156x for SOL at $157)
const positionSizeUSD = position.size * currentPrice
// CORRECT: Use directly as USD value
const positionSizeUSD = Math.abs(position.size)
```
This affects Position Manager's TP1 detection - if calculated incorrectly, TP1 will never trigger because expected size won't match actual size.
**Solana RPC Rate Limiting with Exponential Backoff**
Solana RPC endpoints return 429 errors under load. Always use retry logic for order operations:
```typescript
export async function retryWithBackoff<T>(
operation: () => Promise<T>,
maxRetries: number = 3,
initialDelay: number = 2000
): Promise<T> {
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
return await operation()
} catch (error: any) {
if (error?.message?.includes('429') && attempt < maxRetries - 1) {
const delay = initialDelay * Math.pow(2, attempt)
console.log(`⏳ Rate limited, retrying in ${delay/1000}s... (attempt ${attempt + 1}/${maxRetries})`)
await new Promise(resolve => setTimeout(resolve, delay))
continue
}
throw error
}
}
throw new Error('Max retries exceeded')
}
// Usage in cancelAllOrders
await retryWithBackoff(() => driftClient.cancelOrders(...))
```
Without this, order cancellations fail silently during TP1→breakeven order updates, leaving ghost orders that cause incorrect fills.
**Dual Stop System** (USE_DUAL_STOPS=true):
```typescript
// Soft stop: TRIGGER_LIMIT at -1.5% (avoids wicks)
@@ -246,14 +271,16 @@ const driftSymbol = normalizeTradingViewSymbol(body.symbol)
7. Add to Position Manager if applicable
**Key endpoints:**
- `/api/trading/execute` - Main entry point from n8n (production, requires auth)
- `/api/trading/execute` - Main entry point from n8n (production, requires auth), **auto-caches market data**
- `/api/trading/check-risk` - Pre-execution validation (duplicate check, quality score, **per-symbol cooldown**, rate limits, **symbol enabled check**)
- `/api/trading/test` - Test trades from settings UI (no auth required, **respects symbol enable/disable**)
- `/api/trading/close` - Manual position closing
- `/api/trading/close` - Manual position closing (requires symbol normalization)
- `/api/trading/cancel-orders` - **Manual order cleanup** (for stuck/ghost orders after rate limit failures)
- `/api/trading/positions` - Query open positions from Drift
- `/api/trading/sync-positions` - **Re-sync Position Manager with actual Drift positions** (no auth, for recovery from partial fills/restarts)
- `/api/trading/market-data` - Webhook for TradingView market data updates (GET for debug, POST for data)
- `/api/settings` - Get/update config (writes to .env file, **includes per-symbol settings**)
- `/api/analytics/last-trade` - Fetch most recent trade details for dashboard (includes quality score)
- `/api/analytics/reentry-check` - **Validate manual re-entry** with fresh TradingView data + recent performance
- `/api/analytics/version-comparison` - Compare performance across signal quality logic versions (v1/v2/v3)
- `/api/restart` - Create restart flag for watch-restart.sh script
@@ -401,102 +428,68 @@ docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c "\dt"
6. **Type errors with Prisma:** The Trade type from Prisma is only available AFTER `npx prisma generate` - use explicit types or `// @ts-ignore` carefully
7. **Hardcoded config values:** NEVER use hardcoded values for configurable settings in API endpoints. Always read from `config.minSignalQualityScore` or similar config properties. Settings changed via the UI won't take effect if endpoints use hardcoded values.
7. **Quality score duplication:** Signal quality calculation exists in BOTH `check-risk` and `execute` endpoints - keep logic synchronized
8. **Quality score duplication:** Signal quality calculation exists in BOTH `check-risk` and `execute` endpoints - keep logic synchronized
9. **TP2-as-Runner configuration:**
8. **TP2-as-Runner configuration:**
- `takeProfit2SizePercent: 0` means "TP2 activates trailing stop, no position close"
- This creates 25% runner (vs old 5% system) for better profit capture
- `TAKE_PROFIT_2_PERCENT=0.7` sets TP2 trigger price, `TAKE_PROFIT_2_SIZE_PERCENT` should be 0
- Settings UI correctly shows "TP2 activates trailing stop" instead of size percentage
10. **P&L calculation CRITICAL:** Use actual entry vs exit price calculation, not SDK values:
9. **P&L calculation CRITICAL:** Use actual entry vs exit price calculation, not SDK values:
```typescript
const profitPercent = this.calculateProfitPercent(trade.entryPrice, exitPrice, trade.direction)
const actualRealizedPnL = (closedSizeUSD * profitPercent) / 100
trade.realizedPnL += actualRealizedPnL // NOT: result.realizedPnL from SDK
```
11. **Transaction confirmation CRITICAL:** Both `openPosition()` AND `closePosition()` MUST call `connection.confirmTransaction()` after `placePerpOrder()`. Without this, the SDK returns transaction signatures that aren't confirmed on-chain, causing "phantom trades" or "phantom closes". Always check `confirmation.value.err` before proceeding.
10. **Transaction confirmation CRITICAL:** Both `openPosition()` AND `closePosition()` MUST call `connection.confirmTransaction()` after `placePerpOrder()`. Without this, the SDK returns transaction signatures that aren't confirmed on-chain, causing "phantom trades" or "phantom closes". Always check `confirmation.value.err` before proceeding.
12. **Execution order matters:** When creating trades via API endpoints, the order MUST be:
11. **Execution order matters:** When creating trades via API endpoints, the order MUST be:
1. Open position + place exit orders
2. Save to database (`createTrade()`)
3. Add to Position Manager (`positionManager.addTrade()`)
If Position Manager is added before database save, race conditions occur where monitoring checks before the trade exists in DB.
13. **New trade grace period:** Position Manager skips "external closure" detection for trades <30 seconds old because Drift positions take 5-10 seconds to propagate after opening. Without this grace period, new positions are immediately detected as "closed externally" and cancelled.
12. **New trade grace period:** Position Manager skips "external closure" detection for trades <30 seconds old because Drift positions take 5-10 seconds to propagate after opening. Without this grace period, new positions are immediately detected as "closed externally" and cancelled.
14. **Drift minimum position sizes:** Actual minimums differ from documentation:
13. **Drift minimum position sizes:** Actual minimums differ from documentation:
- SOL-PERP: 0.1 SOL (~$5-15 depending on price)
- ETH-PERP: 0.01 ETH (~$38-40 at $4000/ETH)
- BTC-PERP: 0.0001 BTC (~$10-12 at $100k/BTC)
Always calculate: `minOrderSize × currentPrice` must exceed Drift's $4 minimum. Add buffer for price movement.
15. **Exit reason detection bug:** Position Manager was using current price to determine exit reason, but on-chain orders filled at a DIFFERENT price in the past. Now uses `trade.tp1Hit` / `trade.tp2Hit` flags and realized P&L to correctly identify whether TP1, TP2, or SL triggered. Prevents profitable trades being mislabeled as "SL" exits.
14. **Exit reason detection bug:** Position Manager was using current price to determine exit reason, but on-chain orders filled at a DIFFERENT price in the past. Now uses `trade.tp1Hit` / `trade.tp2Hit` flags and realized P&L to correctly identify whether TP1, TP2, or SL triggered. Prevents profitable trades being mislabeled as "SL" exits.
16. **Per-symbol cooldown:** Cooldown period is per-symbol, NOT global. ETH trade at 10:00 does NOT block SOL trade at 10:01. Each coin (SOL/ETH/BTC) has independent cooldown timer to avoid missing opportunities on different assets.
15. **Per-symbol cooldown:** Cooldown period is per-symbol, NOT global. ETH trade at 10:00 does NOT block SOL trade at 10:01. Each coin (SOL/ETH/BTC) has independent cooldown timer to avoid missing opportunities on different assets.
17. **Timeframe-aware scoring crucial:** Signal quality thresholds MUST adjust for 5min vs higher timeframes:
16. **Timeframe-aware scoring crucial:** Signal quality thresholds MUST adjust for 5min vs higher timeframes:
- 5min charts naturally have lower ADX (12-22 healthy) and ATR (0.2-0.7% healthy) than daily charts
- Without timeframe awareness, valid 5min breakouts get blocked as "low quality"
- Anti-chop filter applies -20 points for extreme sideways regardless of timeframe
- Always pass `timeframe` parameter from TradingView alerts to `scoreSignalQuality()`
18. **Price position chasing causes flip-flops:** Opening longs at 90%+ range or shorts at <10% range reliably loses money:
17. **Price position chasing causes flip-flops:** Opening longs at 90%+ range or shorts at <10% range reliably loses money:
- Database analysis showed overnight flip-flop losses all had price position 9-94% (chasing extremes)
- These trades had valid ADX (16-18) but entered at worst possible time
- Quality scoring now penalizes -15 to -30 points for range extremes
- Prevents rapid reversals when price is already overextended
19. **TradingView ADX minimum for 5min:** Set ADX filter to 15 (not 20+) in TradingView alerts for 5min charts:
18. **TradingView ADX minimum for 5min:** Set ADX filter to 15 (not 20+) in TradingView alerts for 5min charts:
- Higher timeframes can use ADX 20+ for strong trends
- 5min charts need lower threshold to catch valid breakouts
- Bot's quality scoring provides second-layer filtering with context-aware metrics
- Two-stage filtering (TradingView + bot) prevents both overtrading and missing valid signals
20. **Prisma Decimal type handling:** Raw SQL queries return Prisma `Decimal` objects, not plain numbers:
19. **Prisma Decimal type handling:** Raw SQL queries return Prisma `Decimal` objects, not plain numbers:
- Use `any` type for numeric fields in `$queryRaw` results: `total_pnl: any`
- Convert with `Number()` before returning to frontend: `totalPnL: Number(stat.total_pnl) || 0`
- Frontend uses `.toFixed()` which doesn't exist on Decimal objects
- Applies to all aggregations: SUM(), AVG(), ROUND() - all return Decimal types
- Example: `/api/analytics/version-comparison` converts all numeric fields
21. **JavaScript || vs ?? operators CRITICAL:** When setting default values for numeric config, ALWAYS use `??` (nullish coalescing):
```typescript
// WRONG - treats 0 as falsy:
tp2SizePercent: config.takeProfit2SizePercent || 100 // 0 becomes 100!
// CORRECT - only null/undefined are nullish:
tp2SizePercent: config.takeProfit2SizePercent ?? 100 // 0 stays 0
```
- `||` treats `0`, `false`, `""`, `null`, `undefined` as falsy
- `??` only treats `null` and `undefined` as nullish
- Critical for runner system: `TAKE_PROFIT_2_SIZE_PERCENT=0` must be respected
- This bug caused TP2 orders to be placed at 100% despite config setting 0
- Applies to ALL numeric config values where 0 is valid (TP sizes, leverage, thresholds)
22. **Range-bound chop detection:** The anti-chop filter V2 (implemented 2024-11-10) prevents flip-flop losses:
- Detection: Price position <40% of range + ADX <25 = weak range-bound market
- Penalty: -25 points to signal quality score
- Why: Trades entering early in range with weak trend get whipsawed in seconds
- Evidence: Backtest showed 5 flip-flop trades (8-24 second holds) all had this pattern
- Result: Win rate improved from 43.8% to 55.6%, profit per trade +86%
- Implementation: `lib/trading/signal-quality.ts` checks both conditions before price position scoring
23. **Position Manager sync issues:** Partial fills from on-chain orders can cause Position Manager to lose tracking:
- Symptom: Database shows position "closed", but Drift shows position still open without stop loss
- Cause: On-chain orders partially fill (0.29 SOL × 3 times), Position Manager closes database record, but remainder stays open
- Impact: Remaining position has NO software-based stop loss protection (only relies on on-chain orders)
- Solution: Use `/api/trading/sync-positions` endpoint to re-sync Position Manager with actual Drift positions
- Access: Settings UI "Sync Positions" button (orange), or CLI `scripts/sync-positions.sh`
- When: After manual Telegram trades, bot restarts, rate limiting issues, or suspected tracking loss
- Recovery: Endpoint fetches actual Drift positions, re-adds missing ones to Position Manager with calculated TP/SL
- Documentation: See `docs/guides/POSITION_SYNC_GUIDE.md` for details
## File Conventions
- **API routes:** `app/api/[feature]/[action]/route.ts` (Next.js 15 App Router)
@@ -505,6 +498,70 @@ trade.realizedPnL += actualRealizedPnL // NOT: result.realizedPnL from SDK
- **Types:** Define interfaces in same file as implementation (not separate types directory)
- **Console logs:** Use emojis for visual scanning: 🎯 🚀 ✅ ❌ 💰 📊 🛡️
## Re-Entry Analytics System (Phase 1)
**Purpose:** Validate manual Telegram trades using fresh TradingView data + recent performance analysis
**Components:**
1. **Market Data Cache** (`lib/trading/market-data-cache.ts`)
- Singleton service storing TradingView metrics
- 5-minute expiry on cached data
- Tracks: ATR, ADX, RSI, volume ratio, price position, timeframe
2. **Market Data Webhook** (`app/api/trading/market-data/route.ts`)
- Receives TradingView alerts every 1-5 minutes
- POST: Updates cache with fresh metrics
- GET: View cached data (debugging)
3. **Re-Entry Check Endpoint** (`app/api/analytics/reentry-check/route.ts`)
- Validates manual trade requests
- Uses fresh TradingView data if available (<5min old)
- Falls back to historical metrics from last trade
- Scores signal quality + applies performance modifiers:
- **-20 points** if last 3 trades lost money (avgPnL < -5%)
- **+10 points** if last 3 trades won (avgPnL > +5%, WR >= 66%)
- **-5 points** for stale data, **-10 points** for no data
- Minimum score: 55 (vs 60 for new signals)
4. **Auto-Caching** (`app/api/trading/execute/route.ts`)
- Every trade signal from TradingView auto-caches metrics
- Ensures fresh data available for manual re-entries
5. **Telegram Integration** (`telegram_command_bot.py`)
- Calls `/api/analytics/reentry-check` before executing manual trades
- Shows data freshness ("✅ FRESH 23s old" vs "⚠️ Historical")
- Blocks low-quality re-entries unless `--force` flag used
- Fail-open: Proceeds if analytics check fails
**User Flow:**
```
User: "long sol"
↓ Check cache for SOL-PERP
↓ Fresh data? → Use real TradingView metrics
↓ Stale/missing? → Use historical + penalty
↓ Score quality + recent performance
↓ Score >= 55? → Execute
↓ Score < 55? → Block (unless --force)
```
**TradingView Setup:**
Create alerts that fire every 1-5 minutes with this webhook message:
```json
{
"action": "market_data",
"symbol": "{{ticker}}",
"timeframe": "{{interval}}",
"atr": {{ta.atr(14)}},
"adx": {{ta.dmi(14, 14)}},
"rsi": {{ta.rsi(14)}},
"volumeRatio": {{volume / ta.sma(volume, 20)}},
"pricePosition": {{(close - ta.lowest(low, 100)) / (ta.highest(high, 100) - ta.lowest(low, 100)) * 100}},
"currentPrice": {{close}}
}
```
Webhook URL: `https://your-domain.com/api/trading/market-data`
## Per-Symbol Trading Controls
**Purpose:** Independent enable/disable toggles and position sizing for SOL and ETH to support different trading strategies (e.g., ETH for data collection at minimal size, SOL for profit generation).