critical: Fix position close verification to prevent ghost positions

Problem:
- Close transaction confirmed on-chain BUT Drift state takes 5-10s to propagate
- Position Manager immediately checked position after close → still showed open
- Continued monitoring with stale state → eventually ghost detected
- Database marked 'SL closed' but position actually stayed open for 6+ hours
- Position was UNPROTECTED during this time (no monitoring, no TP/SL backup)

Root Cause:
- Transaction confirmation ≠ Drift internal state updated
- SDK needs time to propagate on-chain changes to internal cache
- Position Manager assumed immediate state consistency

Fix (2-layer verification):
1. closePosition(): After 100% close confirmation, wait 5s then verify
   - Query Drift to confirm position actually gone
   - If still exists: Return needsVerification=true flag
   - Log CRITICAL error with transaction signature

2. Position Manager: Handle needsVerification flag
   - DON'T mark position closed in database
   - DON'T remove from monitoring
   - Keep monitoring until ghost detection sees it's actually closed
   - Prevents premature cleanup with wrong exit data

Impact:
- Prevents 6-hour unmonitored position exposure
- Ensures database exit data matches actual Drift closure
- Ghost detection becomes safety net, not primary close mechanism
- User positions always protected until VERIFIED closed

Files:
- lib/drift/orders.ts: Added 5s wait + position verification after close
- lib/trading/position-manager.ts: Check needsVerification flag before cleanup

Incident: Nov 16, 02:51 - Close confirmed but position stayed open until 08:51
This commit is contained in:
mindesbunister
2025-11-16 10:00:10 +01:00
parent 673a49302a
commit c607a66239
2 changed files with 43 additions and 1 deletions

View File

@@ -628,13 +628,44 @@ export async function closePosition(
console.log(` Closed notional: $${closedNotional.toFixed(2)}`)
console.log(` Realized P&L: $${realizedPnL.toFixed(2)}`)
// If closing 100%, cancel all remaining orders for this market
// If closing 100%, verify position actually closed and cancel remaining orders
if (params.percentToClose === 100) {
console.log('🗑️ Position fully closed, cancelling remaining orders...')
const cancelResult = await cancelAllOrders(params.symbol)
if (cancelResult.success && cancelResult.cancelledCount! > 0) {
console.log(`✅ Cancelled ${cancelResult.cancelledCount} orders`)
}
// CRITICAL: Verify position actually closed on Drift (Nov 16, 2025)
// Transaction confirmed ≠ Drift state updated immediately
// Wait 5 seconds for Drift internal state to propagate
console.log('⏳ Waiting 5s for Drift state to propagate...')
await new Promise(resolve => setTimeout(resolve, 5000))
try {
const verifyPosition = await driftService.getPosition(marketConfig.driftMarketIndex)
if (verifyPosition && Math.abs(verifyPosition.size) >= 0.01) {
console.error(`🔴 CRITICAL: Close transaction confirmed BUT position still exists on Drift!`)
console.error(` Transaction: ${txSig}`)
console.error(` Drift size: ${verifyPosition.size}`)
console.error(` This indicates Drift state propagation delay or partial fill`)
console.error(` Position Manager will continue monitoring until Drift confirms closure`)
// Return success but flag that monitoring should continue
return {
success: true,
transactionSignature: txSig,
closePrice: oraclePrice,
closedSize: sizeToClose,
realizedPnL,
needsVerification: true, // Flag for Position Manager
}
} else {
console.log('✅ Position verified closed on Drift')
}
} catch (verifyError) {
console.warn('⚠️ Could not verify position closure:', verifyError)
// Continue anyway - transaction was confirmed
}
}
return {

View File

@@ -1202,6 +1202,17 @@ export class PositionManager {
console.error(`❌ Failed to close ${trade.symbol}:`, errorMsg)
return
}
// CRITICAL: Check if position needs verification (Nov 16, 2025)
// If close transaction confirmed but Drift still shows position open,
// DON'T mark as closed yet - keep monitoring until Drift confirms
if ((result as any).needsVerification) {
console.log(`⚠️ Close transaction confirmed but position still exists on Drift`)
console.log(` Keeping ${trade.symbol} in monitoring until Drift confirms closure`)
console.log(` Ghost detection will handle final cleanup once Drift updates`)
// Keep monitoring - ghost detection will eventually see it's closed
return
}
// Update trade state
if (percentToClose >= 100) {