CRITICAL FIX: Prevent unprotected positions via database-first pattern
Root Cause: - Execute endpoint saved to database AFTER adding to Position Manager - Database save failures were silently caught and ignored - API returned success even when DB save failed - Container restarts lost in-memory Position Manager state - Result: Unprotected positions with no TP/SL monitoring Fixes Applied: 1. Database-First Pattern (app/api/trading/execute/route.ts): - MOVED createTrade() BEFORE positionManager.addTrade() - If database save fails, return HTTP 500 with critical error - Error message: 'CLOSE POSITION MANUALLY IMMEDIATELY' - Position Manager only tracks database-persisted trades - Ensures container restarts can restore all positions 2. Transaction Timeout (lib/drift/orders.ts): - Added 30s timeout to confirmTransaction() in closePosition() - Prevents API from hanging during network congestion - Uses Promise.race() pattern for timeout enforcement 3. Telegram Error Messages (telegram_command_bot.py): - Parse JSON for ALL responses (not just 200 OK) - Extract detailed error messages from 'message' field - Shows critical warnings to user immediately - Fail-open: proceeds if analytics check fails 4. Position Manager (lib/trading/position-manager.ts): - Move lastPrice update to TOP of monitoring loop - Ensures /status endpoint always shows current price Verification: - Test trade cmhxj8qxl0000od076m21l58z executed successfully - Database save completed BEFORE Position Manager tracking - SL triggered correctly at -$4.21 after 15 minutes - All protection systems working as expected Impact: - Eliminates risk of unprotected positions - Provides immediate critical warnings if DB fails - Enables safe container restarts with full position recovery - Verified with live test trade on production See: CRITICAL_INCIDENT_UNPROTECTED_POSITION.md for full incident report
This commit is contained in:
@@ -557,17 +557,35 @@ export async function closePosition(
|
||||
console.log(`✅ Close order placed! Transaction: ${txSig}`)
|
||||
|
||||
// CRITICAL: Confirm transaction on-chain to prevent phantom closes
|
||||
console.log('⏳ Confirming transaction on-chain...')
|
||||
// BUT: Use timeout to prevent API hangs during network congestion
|
||||
console.log('⏳ Confirming transaction on-chain (30s timeout)...')
|
||||
const connection = driftService.getConnection()
|
||||
const confirmation = await connection.confirmTransaction(txSig, 'confirmed')
|
||||
|
||||
if (confirmation.value.err) {
|
||||
console.error('❌ Transaction failed on-chain:', confirmation.value.err)
|
||||
throw new Error(`Transaction failed: ${JSON.stringify(confirmation.value.err)}`)
|
||||
try {
|
||||
const confirmationPromise = connection.confirmTransaction(txSig, 'confirmed')
|
||||
const timeoutPromise = new Promise((_, reject) =>
|
||||
setTimeout(() => reject(new Error('Transaction confirmation timeout')), 30000)
|
||||
)
|
||||
|
||||
const confirmation = await Promise.race([confirmationPromise, timeoutPromise]) as any
|
||||
|
||||
if (confirmation.value?.err) {
|
||||
console.error('❌ Transaction failed on-chain:', confirmation.value.err)
|
||||
throw new Error(`Transaction failed: ${JSON.stringify(confirmation.value.err)}`)
|
||||
}
|
||||
|
||||
console.log('✅ Transaction confirmed on-chain')
|
||||
} catch (timeoutError: any) {
|
||||
if (timeoutError.message === 'Transaction confirmation timeout') {
|
||||
console.warn('⚠️ Transaction confirmation timed out after 30s')
|
||||
console.warn(' Order may still execute - check Drift UI')
|
||||
console.warn(` Transaction signature: ${txSig}`)
|
||||
// Continue anyway - order was submitted and will likely execute
|
||||
} else {
|
||||
throw timeoutError
|
||||
}
|
||||
}
|
||||
|
||||
console.log('✅ Transaction confirmed on-chain')
|
||||
|
||||
// Calculate realized P&L with leverage
|
||||
// CRITICAL: P&L must account for leverage and be calculated on USD notional, not base asset size
|
||||
const profitPercent = ((oraclePrice - position.entryPrice) / position.entryPrice) * 100 * (position.side === 'long' ? 1 : -1)
|
||||
|
||||
Reference in New Issue
Block a user