fix: Critical rate limit handling + startup position restoration

**Problem 1: Rate Limit Cascade**
- Position Manager tried to close repeatedly, overwhelming Helius RPC (10 req/s limit)
- Base retry delay was too aggressive (2s → 4s → 8s)
- No graceful handling when 429 errors occur

**Problem 2: Orphaned Positions After Restart**
- Container restarts lost Position Manager state
- Positions marked 'closed' in DB but still open on Drift (failed close transactions)
- No cross-validation between database and actual Drift positions

**Solutions Implemented:**

1. **Increased retry delays (orders.ts)**:
   - Base delay: 2s → 5s (progression now 5s → 10s → 20s)
   - Reduces RPC pressure during rate limit situations
   - Gives Helius time to recover between retries
   - Documented Helius limits: 100 req/s burst, 10 req/s sustained (free tier)

2. **Startup position validation (init-position-manager.ts)**:
   - Cross-checks last 24h of 'closed' trades against actual Drift positions
   - If DB says closed but Drift shows open → reopens in DB to restore tracking
   - Prevents unmonitored positions from existing after container restarts
   - Logs detailed mismatch info for debugging

3. **Rate limit-aware exit handling (position-manager.ts)**:
   - Detects 429 errors during position close
   - Keeps trade in monitoring instead of removing it
   - Natural retry on next price update (vs aggressive 2s loop)
   - Prevents marking position as closed when transaction actually failed

**Impact:**
- Eliminates orphaned positions after restarts
- Reduces RPC pressure by 2.5x (5s vs 2s base delay)
- Graceful degradation under rate limits
- Position Manager continues monitoring even during temporary RPC issues

**Testing needed:**
- Monitor next container restart to verify position restoration works
- Check rate limit analytics after next close attempt
- Verify no more phantom 'closed' positions when Drift shows open
This commit is contained in:
mindesbunister
2025-11-14 09:50:13 +01:00
parent ebe5e1ab5f
commit 27eb5d4fe8
3 changed files with 95 additions and 50 deletions

View File

@@ -890,6 +890,9 @@ export class PositionManager {
/**
* Execute exit (close position)
*
* Rate limit handling: If 429 error occurs, marks trade for retry
* instead of removing it from monitoring (prevents orphaned positions)
*/
private async executeExit(
trade: ActiveTrade,
@@ -907,7 +910,18 @@ export class PositionManager {
})
if (!result.success) {
console.error(`❌ Failed to close ${trade.symbol}:`, result.error)
const errorMsg = result.error || 'Unknown error'
// Check if it's a rate limit error
if (errorMsg.includes('429') || errorMsg.toLowerCase().includes('rate limit')) {
console.error(`⚠️ Rate limited while closing ${trade.symbol} - will retry on next price update`)
// DON'T remove trade from monitoring - let it retry naturally
// The retry logic in closePosition() already handled 3 attempts
// Next price update will trigger another exit attempt
return
}
console.error(`❌ Failed to close ${trade.symbol}:`, errorMsg)
return
}