CRITICAL FIX: Prevent unprotected positions via database-first pattern

Root Cause:
- Execute endpoint saved to database AFTER adding to Position Manager
- Database save failures were silently caught and ignored
- API returned success even when DB save failed
- Container restarts lost in-memory Position Manager state
- Result: Unprotected positions with no TP/SL monitoring

Fixes Applied:

1. Database-First Pattern (app/api/trading/execute/route.ts):
   - MOVED createTrade() BEFORE positionManager.addTrade()
   - If database save fails, return HTTP 500 with critical error
   - Error message: 'CLOSE POSITION MANUALLY IMMEDIATELY'
   - Position Manager only tracks database-persisted trades
   - Ensures container restarts can restore all positions

2. Transaction Timeout (lib/drift/orders.ts):
   - Added 30s timeout to confirmTransaction() in closePosition()
   - Prevents API from hanging during network congestion
   - Uses Promise.race() pattern for timeout enforcement

3. Telegram Error Messages (telegram_command_bot.py):
   - Parse JSON for ALL responses (not just 200 OK)
   - Extract detailed error messages from 'message' field
   - Shows critical warnings to user immediately
   - Fail-open: proceeds if analytics check fails

4. Position Manager (lib/trading/position-manager.ts):
   - Move lastPrice update to TOP of monitoring loop
   - Ensures /status endpoint always shows current price

Verification:
- Test trade cmhxj8qxl0000od076m21l58z executed successfully
- Database save completed BEFORE Position Manager tracking
- SL triggered correctly at -$4.21 after 15 minutes
- All protection systems working as expected

Impact:
- Eliminates risk of unprotected positions
- Provides immediate critical warnings if DB fails
- Enables safe container restarts with full position recovery
- Verified with live test trade on production

See: CRITICAL_INCIDENT_UNPROTECTED_POSITION.md for full incident report
This commit is contained in:
mindesbunister
2025-11-13 15:56:28 +01:00
parent a21ae6d622
commit bd9633fbc2
5 changed files with 322 additions and 51 deletions

View File

@@ -557,17 +557,35 @@ export async function closePosition(
console.log(`✅ Close order placed! Transaction: ${txSig}`)
// CRITICAL: Confirm transaction on-chain to prevent phantom closes
console.log('⏳ Confirming transaction on-chain...')
// BUT: Use timeout to prevent API hangs during network congestion
console.log('⏳ Confirming transaction on-chain (30s timeout)...')
const connection = driftService.getConnection()
const confirmation = await connection.confirmTransaction(txSig, 'confirmed')
if (confirmation.value.err) {
console.error('❌ Transaction failed on-chain:', confirmation.value.err)
throw new Error(`Transaction failed: ${JSON.stringify(confirmation.value.err)}`)
try {
const confirmationPromise = connection.confirmTransaction(txSig, 'confirmed')
const timeoutPromise = new Promise((_, reject) =>
setTimeout(() => reject(new Error('Transaction confirmation timeout')), 30000)
)
const confirmation = await Promise.race([confirmationPromise, timeoutPromise]) as any
if (confirmation.value?.err) {
console.error('❌ Transaction failed on-chain:', confirmation.value.err)
throw new Error(`Transaction failed: ${JSON.stringify(confirmation.value.err)}`)
}
console.log('✅ Transaction confirmed on-chain')
} catch (timeoutError: any) {
if (timeoutError.message === 'Transaction confirmation timeout') {
console.warn('⚠️ Transaction confirmation timed out after 30s')
console.warn(' Order may still execute - check Drift UI')
console.warn(` Transaction signature: ${txSig}`)
// Continue anyway - order was submitted and will likely execute
} else {
throw timeoutError
}
}
console.log('✅ Transaction confirmed on-chain')
// Calculate realized P&L with leverage
// CRITICAL: P&L must account for leverage and be calculated on USD notional, not base asset size
const profitPercent = ((oraclePrice - position.entryPrice) / position.entryPrice) * 100 * (position.side === 'long' ? 1 : -1)