diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 38c23e2..a9a4654 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -994,47 +994,47 @@ ORDER BY MIN(adx) DESC; ## Common Pitfalls 1. **WRONG RPC PROVIDER (CRITICAL - CATASTROPHIC SYSTEM FAILURE):** - - **FINAL CONCLUSION Nov 14, 2025 (after extensive testing):** Helius is the ONLY reliable RPC provider for Drift SDK - - **Alchemy "breakthrough" was NOT sustainable:** First trade appeared perfect (14:25 CET), but Alchemy consistently fails on subsequent trades - - **Symptom #1 (Helius rate limits):** 239+ rate limit errors in 10 minutes → led to initial Alchemy switch - - **Symptom #2 (Alchemy incompatibility):** - * 100+ accountSubscribe errors during init - * SDK claims "initialized successfully" but is fundamentally broken - * First trade might work, then SDK gets into bad state - * Timeouts on subsequent execute calls (60+ seconds) - * Positions open WITHOUT TP/SL orders (NO RISK MANAGEMENT) - * Position Manager doesn't track positions - * Database saves fail silently - - **Root Cause:** Alchemy RPC does NOT support WebSocket subscriptions (accountSubscribe method) which Drift SDK REQUIRES for proper operation - - **ONLY WORKING SOLUTION:** Use Helius RPC with proper rate limit handling (5s exponential backoff retries) - - **Why Helius works:** Supports ALL Solana RPC methods including WebSocket subscriptions, Drift SDK functions correctly - - **Why Alchemy fails:** Despite 10,000 CU/s throughput (Growth plan), lacks WebSocket support = Drift SDK malfunction after first operation - - **Impact with Helius:** System fully operational, TP1/TP2/SL work correctly, Position Manager tracks properly - - **User confirmations:** - * "SO IT WAS THE FUCKING RPC THAT WAS CAUSING ALL THE ISSUES!!!!!!!!!!!!" (referring to Helius rate limits) - * "after changing back the settings it started to act up again" (Alchemy breaks again) - * "telegram works again" (after reverting to Helius final time) - - **Testing timeline Nov 14, 2025:** - * 14:01 - Switched to Alchemy - * 14:25 - First trade perfect (appeared to be breakthrough) - * 15:00-20:00 - Added hybrid/fallback code (broke everything) - * 20:00 - Reverted to pure Alchemy from working commit (still broke - timeouts, no TP/SL) - * 20:05 - Reverted to Helius (works reliably) - - **Critical:** NEVER use RPC provider without WebSocket subscription support for Drift Protocol trading - - **This is definitive:** Helius only, no alternatives work reliably + - **FINAL CONCLUSION Nov 14, 2025 (INVESTIGATION COMPLETE):** Helius is the ONLY reliable RPC provider for Drift SDK + - **Root Cause CONFIRMED:** Alchemy's rate limiting breaks Drift SDK's burst subscription pattern during initialization + - **Definitive Proof (Nov 14, 21:14 CET):** + * Created diagnostic endpoint `/api/testing/drift-init` + * Alchemy: 17-71 subscription errors EVERY init (49 avg over 5 runs), 1644ms avg init time + * Helius: 0 subscription errors EVERY init, 800ms avg init time + * See `docs/ALCHEMY_RPC_INVESTIGATION_RESULTS.md` for full test data - - **TECHNICAL NOTE (for future investigation):** - * Alchemy Growth plan (10,000 CU/s) DOES support WebSocket subscriptions according to official docs - * Supports up to 2,000 concurrent WebSocket connections - * All standard Solana RPC methods supported (getProgramAccounts, getMultipleAccounts, sendTransaction, etc.) - * No known Drift-Alchemy incompatibilities documented by Drift Labs - * Research source: Alchemy Solana docs + Drift Protocol documentation - * **Hypothesis:** accountSubscribe errors might be rate limit 429s (CUPS enforcement), not "method not found" - * **Potential issue:** Drift SDK may not handle Alchemy's rate limit pattern during initialization - * **Why it matters:** First trade works (subscriptions established) → subsequent trades fail (SDK in bad state from rate limit errors) - * **Pragmatic decision:** Helius works reliably NOW, theoretical investigation can wait - * **Future optimization:** Could implement WebSocket-specific retry logic, subscription batching, init delays - * **For now:** Helius free tier + 5s exponential backoff = production-ready solution + - **Why Alchemy Fails:** + * Drift SDK subscribes to 30-50+ accounts simultaneously during init (burst pattern) + * Alchemy's CUPS enforcement rate limits these burst requests + * Drift SDK does NOT retry failed subscriptions + * SDK reports "initialized successfully" but with incomplete subscription set + * Subsequent operations fail/timeout due to missing account data + * Error message: "Received JSON-RPC error calling `accountSubscribe`" + + - **Why "Breakthrough" at 14:25 Wasn't Real:** + * First Alchemy test had 17-71 subscription errors (random variation) + * Sometimes gets lucky with "just enough" subscriptions for one operation + * SDK in degraded state from the start, just not obvious until second operation + * This explains why first trade "worked" but subsequent trades failed + + - **Why Helius Works:** + * Higher burst tolerance for Solana dApp subscription patterns + * Zero subscription errors during init + * Faster initialization (800ms vs 1600ms) + * Stable for continuous operations + + - **Technical Reality vs Documentation:** + * Alchemy DOES support WebSocket subscriptions (research confirmed) + * Alchemy DOES support accountSubscribe method (not -32601 error) + * BUT: Rate limit enforcement model incompatible with Drift's burst pattern + * Documentation doesn't mention burst subscription limits + + - **Production Status:** + * Using: Helius RPC (https://mainnet.helius-rpc.com/?api-key=...) + * Retry logic: 5s exponential backoff for rate limits + * System: Stable, TP1/TP2/SL working, Position Manager tracking correctly + + - **Investigation Closed:** This is DEFINITIVE. Use Helius. Do not use Alchemy. + - **Test Yourself:** `curl 'http://localhost:3001/api/testing/drift-init?rpc=alchemy'` 2. **Prisma not generated in Docker:** Must run `npx prisma generate` in Dockerfile BEFORE `npm run build`