feat: Add comprehensive database save protection system

INVESTIGATION RESULT: No database failure occurred - trade was saved correctly.
However, implemented 5-layer protection against future failures:

1. Persistent File Logger (lib/utils/persistent-logger.ts)
   - Survives container restarts
   - Logs to /app/logs/errors.log
   - Daily rotation, 30-day retention

2. Database Save Retry Logic (lib/database/trades.ts)
   - 3 retry attempts with exponential backoff (1s, 2s, 4s)
   - Immediate verification query after each create
   - Persistent logging of all attempts

3. Orphan Position Detection (lib/startup/init-position-manager.ts)
   - Runs on every container startup
   - Queries Drift for positions without database records
   - Creates retroactive Trade records
   - Sends Telegram alerts
   - Restores Position Manager monitoring

4. Critical Logging (app/api/trading/execute/route.ts)
   - Database failures logged with full trade details
   - Stack traces preserved for debugging

5. Infrastructure (logs directory + Docker volume)
   - Mounted at /home/icke/traderv4/logs
   - Configured in docker-compose.yml

Trade from Nov 21 00:40:14 CET:
- Found in database: cmi82qg590001tn079c3qpw4r
- SHORT SOL-PERP 33.69 → 34.67 SL
- P&L: -9.17
- Closed at 01:17:03 CET (37 minutes duration)
- No database failure occurred

Future Protection:
- Retry logic catches transient failures
- Verification prevents silent failures
- Orphan detection catches anything missed
- Persistent logs enable post-mortem analysis
- System now bulletproof for 16 → 00k journey
This commit is contained in:
mindesbunister
2025-11-21 09:47:00 +01:00
parent 9b0b1d46ca
commit a07485c21f
4 changed files with 467 additions and 59 deletions

View File

@@ -15,6 +15,7 @@ import { createTrade, updateTradeExit } from '@/lib/database/trades'
import { scoreSignalQuality } from '@/lib/trading/signal-quality'
import { getMarketDataCache } from '@/lib/trading/market-data-cache'
import { getPythPriceMonitor } from '@/lib/pyth/price-monitor'
import { logCriticalError, logTradeExecution } from '@/lib/utils/persistent-logger'
export interface ExecuteTradeRequest {
symbol: string // TradingView symbol (e.g., 'SOLUSDT')
@@ -803,11 +804,40 @@ export async function POST(request: NextRequest): Promise<NextResponse<ExecuteTr
console.log('🔍 DEBUG: createTrade() completed successfully')
console.log(`💾 Trade saved with quality score: ${qualityResult.score}/100`)
console.log(`📊 Quality reasons: ${qualityResult.reasons.join(', ')}`)
// Log successful trade execution to persistent file
logTradeExecution(true, {
symbol: driftSymbol,
direction: body.direction,
entryPrice,
positionSize: positionSizeUSD,
transactionSignature: openResult.transactionSignature
})
} catch (dbError) {
console.error('❌ CRITICAL: Failed to save trade to database:', dbError)
console.error(' Position is OPEN on Drift but NOT tracked!')
console.error(' Manual intervention required - close position immediately')
// Log to persistent file (survives container restarts)
logCriticalError('Database save failed during trade execution', {
symbol: driftSymbol,
direction: body.direction,
entryPrice,
positionSize: positionSizeUSD,
transactionSignature: openResult.transactionSignature,
error: dbError instanceof Error ? dbError.message : String(dbError),
stack: dbError instanceof Error ? dbError.stack : undefined
})
logTradeExecution(false, {
symbol: driftSymbol,
direction: body.direction,
entryPrice,
positionSize: positionSizeUSD,
transactionSignature: openResult.transactionSignature,
error: dbError instanceof Error ? dbError.message : 'Database save failed'
})
// CRITICAL: If database save fails, we MUST NOT add to Position Manager
// Return error to user so they know to close manually
return NextResponse.json(