CRITICAL FIX: Prevent unprotected positions via database-first pattern

Root Cause:
- Execute endpoint saved to database AFTER adding to Position Manager
- Database save failures were silently caught and ignored
- API returned success even when DB save failed
- Container restarts lost in-memory Position Manager state
- Result: Unprotected positions with no TP/SL monitoring

Fixes Applied:

1. Database-First Pattern (app/api/trading/execute/route.ts):
   - MOVED createTrade() BEFORE positionManager.addTrade()
   - If database save fails, return HTTP 500 with critical error
   - Error message: 'CLOSE POSITION MANUALLY IMMEDIATELY'
   - Position Manager only tracks database-persisted trades
   - Ensures container restarts can restore all positions

2. Transaction Timeout (lib/drift/orders.ts):
   - Added 30s timeout to confirmTransaction() in closePosition()
   - Prevents API from hanging during network congestion
   - Uses Promise.race() pattern for timeout enforcement

3. Telegram Error Messages (telegram_command_bot.py):
   - Parse JSON for ALL responses (not just 200 OK)
   - Extract detailed error messages from 'message' field
   - Shows critical warnings to user immediately
   - Fail-open: proceeds if analytics check fails

4. Position Manager (lib/trading/position-manager.ts):
   - Move lastPrice update to TOP of monitoring loop
   - Ensures /status endpoint always shows current price

Verification:
- Test trade cmhxj8qxl0000od076m21l58z executed successfully
- Database save completed BEFORE Position Manager tracking
- SL triggered correctly at -$4.21 after 15 minutes
- All protection systems working as expected

Impact:
- Eliminates risk of unprotected positions
- Provides immediate critical warnings if DB fails
- Enables safe container restarts with full position recovery
- Verified with live test trade on production

See: CRITICAL_INCIDENT_UNPROTECTED_POSITION.md for full incident report
This commit is contained in:
mindesbunister
2025-11-13 15:56:28 +01:00
parent a21ae6d622
commit bd9633fbc2
5 changed files with 322 additions and 51 deletions

View File

@@ -506,39 +506,11 @@ export async function POST(request: NextRequest): Promise<NextResponse<ExecuteTr
console.error('❌ Unexpected error placing exit orders:', err)
}
// Add to position manager for monitoring AFTER orders are placed
await positionManager.addTrade(activeTrade)
console.log('✅ Trade added to position manager for monitoring')
// Create response object
const response: ExecuteTradeResponse = {
success: true,
positionId: openResult.transactionSignature,
symbol: driftSymbol,
direction: body.direction,
entryPrice: entryPrice,
positionSize: positionSizeUSD,
leverage: leverage, // Use actual symbol-specific leverage, not global config
stopLoss: stopLossPrice,
takeProfit1: tp1Price,
takeProfit2: tp2Price,
stopLossPercent: config.stopLossPercent,
tp1Percent: config.takeProfit1Percent,
tp2Percent: config.takeProfit2Percent,
entrySlippage: openResult.slippage,
timestamp: new Date().toISOString(),
}
// Attach exit order signatures to response
if (exitOrderSignatures.length > 0) {
(response as any).exitOrderSignatures = exitOrderSignatures
}
// Save trade to database
// Save trade to database FIRST (CRITICAL: Must succeed before Position Manager)
let qualityResult
try {
// Calculate quality score if metrics available
const qualityResult = scoreSignalQuality({
qualityResult = scoreSignalQuality({
atr: body.atr || 0,
adx: body.adx || 0,
rsi: body.rsi || 0,
@@ -584,8 +556,49 @@ export async function POST(request: NextRequest): Promise<NextResponse<ExecuteTr
console.log(`💾 Trade saved with quality score: ${qualityResult.score}/100`)
console.log(`📊 Quality reasons: ${qualityResult.reasons.join(', ')}`)
} catch (dbError) {
console.error('❌ Failed to save trade to database:', dbError)
// Don't fail the trade if database save fails
console.error('❌ CRITICAL: Failed to save trade to database:', dbError)
console.error(' Position is OPEN on Drift but NOT tracked!')
console.error(' Manual intervention required - close position immediately')
// CRITICAL: If database save fails, we MUST NOT add to Position Manager
// Return error to user so they know to close manually
return NextResponse.json(
{
success: false,
error: 'Database save failed - position unprotected',
message: `Position opened on Drift but database save failed. CLOSE POSITION MANUALLY IMMEDIATELY. Transaction: ${openResult.transactionSignature}`,
},
{ status: 500 }
)
}
// Add to position manager for monitoring ONLY AFTER database save succeeds
await positionManager.addTrade(activeTrade)
console.log('✅ Trade added to position manager for monitoring')
// Create response object
const response: ExecuteTradeResponse = {
success: true,
positionId: openResult.transactionSignature,
symbol: driftSymbol,
direction: body.direction,
entryPrice: entryPrice,
positionSize: positionSizeUSD,
leverage: leverage, // Use actual symbol-specific leverage, not global config
stopLoss: stopLossPrice,
takeProfit1: tp1Price,
takeProfit2: tp2Price,
stopLossPercent: config.stopLossPercent,
tp1Percent: config.takeProfit1Percent,
tp2Percent: config.takeProfit2Percent,
entrySlippage: openResult.slippage,
timestamp: new Date().toISOString(),
}
// Attach exit order signatures to response
if (exitOrderSignatures.length > 0) {
(response as any).exitOrderSignatures = exitOrderSignatures
}
console.log('✅ Trade executed successfully!')