Commit Graph

24 Commits

Author SHA1 Message Date
mindesbunister
5d5868d802 critical: Fix Smart Validation Queue blockReason mismatch (Bug #84)
Root Cause: check-risk endpoint passes blockReason='SMART_VALIDATION_QUEUED'
but addSignal() only accepted 'QUALITY_SCORE_TOO_LOW' → signals blocked but never queued

Impact: Quality 85 LONG signal at 08:40:03 saved to database but never monitored
User missed validation opportunity when price moved favorably

Fix: Accept both blockReason variants in addSignal() validation check

Evidence:
- Database record cmj41pdqu0101pf07mith5s4c has blockReason='SMART_VALIDATION_QUEUED'
- No logs showing addSignal() execution (would log ' Smart validation queued')
- check-risk code line 451 passes 'SMART_VALIDATION_QUEUED'
- addSignal() line 76 rejected signals != 'QUALITY_SCORE_TOO_LOW'

Result: Quality 50-89 signals will now be properly queued for validation
2025-12-13 17:24:38 +01:00
copilot-swe-agent[bot]
86b69ff7c0 fix: Improve enum handling robustness in discover-drift-markets script
Co-authored-by: mindesbunister <32161838+mindesbunister@users.noreply.github.com>
2025-12-05 14:43:06 +00:00
copilot-swe-agent[bot]
e9472175ba feat: Add Drift market discovery script for finding market indices and oracle addresses
Co-authored-by: mindesbunister <32161838+mindesbunister@users.noreply.github.com>
2025-12-05 14:38:16 +00:00
mindesbunister
302511293c feat: Add production logging gating (Phase 1, Task 1.1)
- Created logger utility with environment-based gating (lib/utils/logger.ts)
- Replaced 517 console.log statements with logger.log (71% reduction)
- Fixed import paths in 15 files (resolved comment-trapped imports)
- Added DEBUG_LOGS=false to .env
- Achieves 71% immediate log reduction (517/731 statements)
- Expected 90% reduction in production when deployed

Impact: Reduced I/O blocking, lower log volume in production
Risk: LOW (easy rollback, non-invasive)
Phase: Phase 1, Task 1.1 (Quick Wins - Console.log Production Gating)

Files changed:
- NEW: lib/utils/logger.ts (production-safe logging)
- NEW: scripts/replace-console-logs.js (automation tool)
- Modified: 15 lib/*.ts files (console.log → logger.log)
- Modified: .env (DEBUG_LOGS=false)

Next: Task 1.2 (Image Size Optimization)
2025-12-05 00:32:41 +01:00
mindesbunister
7e1fe1cc30 feat: V9 advanced parameter sweep with MA gap filter (810K configs)
Parameter space expansion:
- Original 15 params: 101K configurations
- NEW: MA gap filter (3 dimensions) = 18× expansion
- Total: ~810,000 configurations across 4 time profiles
- Chunk size: 1,000 configs/chunk = ~810 chunks

MA Gap Filter parameters:
- use_ma_gap: True/False (2 values)
- ma_gap_min_long: -5.0%, 0%, +5.0% (3 values)
- ma_gap_min_short: -5.0%, 0%, +5.0% (3 values)

Implementation:
- money_line_v9.py: Full v9 indicator with MA gap logic
- v9_advanced_worker.py: Chunk processor (1,000 configs)
- v9_advanced_coordinator.py: Work distributor (2 EPYC workers)
- run_v9_advanced_sweep.sh: Startup script (generates + launches)

Infrastructure:
- Uses existing EPYC cluster (64 cores total)
- Worker1: bd-epyc-02 (32 threads)
- Worker2: bd-host01 (32 threads via SSH hop)
- Expected runtime: 70-80 hours
- Database: SQLite (chunk tracking + results)

Goal: Find optimal MA gap thresholds for filtering false breakouts
during MA whipsaw zones while preserving trend entries.
2025-12-01 18:11:47 +01:00
mindesbunister
cc56b72df2 fix: Database-first cluster status detection + Stop button clarification
CRITICAL FIX (Nov 30, 2025):
- Dashboard showed 'idle' despite 22+ worker processes running
- Root cause: SSH-based worker detection timing out
- Solution: Check database for running chunks FIRST

Changes:
1. app/api/cluster/status/route.ts:
   - Query exploration database before SSH detection
   - If running chunks exist, mark workers 'active' even if SSH fails
   - Override worker status: 'offline' → 'active' when chunks running
   - Log: ' Cluster status: ACTIVE (database shows running chunks)'
   - Database is source of truth, SSH only for supplementary metrics

2. app/cluster/page.tsx:
   - Stop button ALREADY EXISTS (conditionally shown)
   - Shows Start when status='idle', Stop when status='active'
   - No code changes needed - fixed by status detection

Result:
- Dashboard now shows 'ACTIVE' with 2 workers (correct)
- Workers show 'active' status (was 'offline')
- Stop button automatically visible when cluster active
- System resilient to SSH timeouts/network issues

Verified:
- Container restarted: Nov 30 21:18 UTC
- API tested: Returns status='active', activeWorkers=2
- Logs confirm: Database-first logic working
- Workers confirmed running: 22+ processes on worker1, workers on worker2
2025-11-30 22:23:01 +01:00
mindesbunister
5f7702469e remove: V10 momentum system - backtest proved it adds no value
- Removed v10 TradingView indicator (moneyline_v10_momentum_dots.pinescript)
- Removed v10 penalty system from signal-quality.ts (-30/-25 point penalties)
- Removed backtest result files (sweep_*.csv)
- Updated copilot-instructions.md to remove v10 references
- Simplified direction-specific quality thresholds (LONG 90+, SHORT 80+)

Rationale:
- 1,944 parameter combinations tested in backtest
- All top results IDENTICAL (568 trades, $498 P&L, 61.09% WR)
- Momentum parameters had ZERO impact on trade selection
- Profit factor 1.027 too low (barely profitable after fees)
- Max drawdown -$1,270 vs +$498 profit = terrible risk-reward
- v10 penalties were blocking good trades (bug: applied to wrong positions)

Keeping v9 as production system - simpler, proven, effective.
2025-11-28 22:35:32 +01:00
mindesbunister
40d69b13ef wip: Emergency order restoration endpoint (has singleton issues)
- Created /api/trading/place-exit-orders endpoint
- Created restore-orders.mjs script
- Issue: Next.js creates separate Drift instances per route
- Workaround: Use /api/trading/cancel-orders to remove orphaned orders

Current situation:
- 32 orphaned orders existed and were cancelled
- Position Manager should auto-place new orders
- Manual order placement endpoint needs refactoring
2025-11-16 22:10:15 +01:00
mindesbunister
abc32d52a0 feat: Add daily rate limit monitoring script
Purpose: Track RPC rate limiting and guide upgrade decision

Features:
- Last 24h summary (hits, recoveries, exhausted events)
- 7-day trend analysis
- Automated decision criteria:
  * >120 exhausted/day: UPGRADE IMMEDIATELY
  * 30-120/day: Monitor 24h more
  * 5-30/day: Acceptable with retry logic
  * <5/day: Keep free tier
- ROI calculator (potential savings vs upgrade cost)

Usage:
  bash scripts/monitor-rate-limits.sh

Run daily to track improvement after retry logic deployment.

Initial Results (Nov 15, 17:40):
- 0 exhausted events in last 24h (was 14/day before)
- Retry logic working perfectly
- Decision: Keep free tier, focus on profitability

Upgrade trigger: If exhausted stays >5/day after 48 hours monitoring

Files: scripts/monitor-rate-limits.sh
2025-11-15 17:41:13 +01:00
mindesbunister
c4c0c63de1 feat: Add Alchemy RPC diagnostic endpoint + complete investigation
- Created /api/testing/drift-init endpoint for systematic RPC testing
- Tested Alchemy: 17-71 subscription errors per init (49 avg over 5 runs)
- Tested Helius: 0 subscription errors, 800ms init time
- DEFINITIVE PROOF: Alchemy rate limits break Drift SDK initialization
- Root cause: Burst subscription pattern hits CUPS limits
- SDK doesn't retry failed subscriptions → unstable state
- Documented complete findings in docs/ALCHEMY_RPC_INVESTIGATION_RESULTS.md
- Investigation CLOSED - Helius is the only reliable solution
2025-11-14 22:20:04 +01:00
mindesbunister
7ff78ee0bd feat: Hybrid RPC fallback system (Alchemy → Helius)
- Automatic fallback after 2 consecutive rate limits
- Primary: Alchemy (300M CU/month, stable for normal ops)
- Fallback: Helius (10 req/sec, backup for startup bursts)
- Reduced startup validation: 6h window, 5 trades (was 24h, 20 trades)
- Multi-position safety check (prevents order cancellation conflicts)
- Rate limit-aware retry logic with exponential backoff

Implementation:
- lib/drift/client.ts: Added fallbackConnection, switchToFallbackRpc()
- .env: SOLANA_FALLBACK_RPC_URL configuration
- lib/startup/init-position-manager.ts: Reduced validation scope
- lib/trading/position-manager.ts: Multi-position order protection

Tested: System switched to fallback on startup, Position Manager active
Result: 1 active trade being monitored after automatic RPC switch
2025-11-14 15:28:07 +01:00
mindesbunister
6dbbe3ea57 feat: add granular phase-level cards to Nextcloud Deck sync
- Updated parser to extract phases from detailed roadmap files
- Cleaner card titles: 'Phase X: Description' instead of file paths
- Improved status detection: CURRENT/DEPLOYED → In Progress, NEXT → Planning
- Code block removal to prevent API 400/500 errors
- Shorter descriptions (400 chars max) for better readability
- All 21 cards created: 3 initiatives + 18 phases

Card distribution:
- Backlog: 6 cards (future work)
- Planning: 1 card (next phase)
- In Progress: 10 cards (active work)
- Complete: 4 cards (done)

Changes:
- scripts/sync-roadmap-to-deck.py: Complete parser rewrite for phase-level granularity
- Handles both ## Phase and ### Phase patterns
- Removes markdown/emojis from titles for clean display
2025-11-14 11:39:03 +01:00
mindesbunister
a49db192f4 feat: complete Nextcloud Deck integration with English emoji stack names
- Renamed all stacks to English with emojis (Backlog, Planning, In Progress, Complete)
- Updated sync script to use new stack names
- Created all 3 initiative cards (IDs 189-191)
- Enhanced error handling with detailed debug output
- Updated documentation with API limitations and troubleshooting
- Fixed stack fallback from 'eingang' to '📥 Backlog'

Changes:
- scripts/sync-roadmap-to-deck.py: Updated STATUS_TO_STACK mapping, added verbose logging
- docs/NEXTCLOUD_DECK_SYNC.md: Updated stack table, added Known Limitations section, enhanced troubleshooting

Note: 6 duplicate/test cards (184-188, 192) must be deleted manually from Nextcloud UI
      due to API limitations (DELETE returns 405)
2025-11-14 11:25:09 +01:00
mindesbunister
77a22bae3f feat: add Nextcloud Deck roadmap sync system
- Create discover-deck-ids.sh to find board/stack configuration
- Implement sync-roadmap-to-deck.py for roadmap → Deck sync
- Parse OPTIMIZATION_MASTER_ROADMAP.md and extract initiatives
- Map roadmap status to Deck stacks (eingang/planung/arbeit/erledigt)
- Create cards with titles, descriptions, due dates, progress
- Support dry-run mode for testing before actual sync
- Add comprehensive documentation in NEXTCLOUD_DECK_SYNC.md

**Benefits:**
- Visual kanban board for roadmap management
- Drag & drop to prioritize tasks
- Single source of truth (markdown files)
- Easy task tracking and status updates
- No manual duplication between systems

**Initial Sync:**
- Created 1 card: Initiative 1 (Signal Quality Optimization)
- Placed in 'eingang' (FUTURE status)

**Future Work:**
- Bidirectional sync (Deck → Roadmap)
- Phase-level cards parsing
- Manual card creation → roadmap entry
- Automated cron sync
2025-11-14 11:09:37 +01:00
mindesbunister
089308a07e Add Position Sync feature for recovering tracking after partial fills
- New /api/trading/sync-positions endpoint (no auth)
- Fetches actual Drift positions and compares with Position Manager
- Removes stale tracking, adds missing positions with calculated TP/SL
- Settings UI: Orange 'Sync Positions' button added
- CLI script: scripts/sync-positions.sh for terminal access
- Full documentation in docs/guides/POSITION_SYNC_GUIDE.md
- Quick reference: POSITION_SYNC_QUICK_REF.md
- Updated AI instructions with pitfall #23

Problem solved: Manual Telegram trades with partial fills can cause
Position Manager to lose tracking, leaving positions without software-
based stop loss protection. This feature restores dual-layer protection.

Note: Docker build not picking up route yet (cache issue), needs investigation
2025-11-10 17:05:32 +01:00
mindesbunister
988fdb9ea4 Fix runner system + strengthen anti-chop filter
Three critical bugs fixed:
1. P&L calculation (65x inflation) - now uses collateralUSD not notional
2. handlePostTp1Adjustments() - checks tp2SizePercent===0 for runner mode
3. JavaScript || operator bug - changed to ?? for proper 0 handling

Signal quality improvements:
- Added anti-chop filter: price position <40% + ADX <25 = -25 points
- Prevents range-bound flip-flops (caught all 3 today)
- Backtest: 43.8% → 55.6% win rate, +86% profit per trade

Changes:
- lib/trading/signal-quality.ts: RANGE-BOUND CHOP penalty
- lib/drift/orders.ts: Fixed P&L calculation + transaction confirmation
- lib/trading/position-manager.ts: Runner system logic
- app/api/trading/execute/route.ts: || to ?? for tp2SizePercent
- app/api/trading/test/route.ts: || to ?? for tp1/tp2SizePercent
- prisma/schema.prisma: Added collateralUSD field
- scripts/fix_pnl_calculations.sql: Historical P&L correction
2025-11-10 15:36:51 +01:00
mindesbunister
d2fbd125a0 fix: Make minSignalQualityScore configurable via settings + anti-chop improvements
CRITICAL BUG FIX:
- Settings page saved MIN_SIGNAL_QUALITY_SCORE to .env but check-risk had hardcoded value
- Now reads from config.minSignalQualityScore (defaults to 65, editable via /settings)
- Prevents settings changes from being ignored after restart

ANTI-CHOP FILTER FIXES:
- Fixed volume breakout bonus conflicting with anti-chop filter
- Volume breakout now requires ADX > 18 (trending market)
- Prevents high volume + low ADX from getting rewarded instead of penalized
- Anti-chop filter now properly blocks whipsaw traps at score 60

TESTING INFRASTRUCTURE:
- Added backtest script showing +17.1% P&L improvement (saved $242 in losses)
- Added test-signals.sh for comprehensive signal quality validation
- Added test-recent-signals.sh for analyzing actual trading session signals
- All tests passing: timeframe awareness, anti-chop, score thresholds

CHANGES:
- config/trading.ts: Added minSignalQualityScore to interface and defaults
- app/api/trading/check-risk/route.ts: Use config value instead of hardcoded 65
- lib/trading/signal-quality.ts: Fixed volume breakout bonus logic
- .env: Added MIN_SIGNAL_QUALITY_SCORE=65
- scripts/: Added comprehensive testing tools

BACKTEST RESULTS (Last 30 trades):
- Old system (score ≥60): $1,412.79 P&L
- New system (score ≥65 + anti-chop): $1,654.79 P&L
- Improvement: +$242.00 (+17.1%)
- Blocked 5 losing trades, missed 0 winners
2025-11-10 11:22:52 +01:00
mindesbunister
22195ed34c Fix P&L calculation and signal flip detection
- Fix external closure P&L using tp1Hit flag instead of currentSize
- Add direction change detection to prevent false TP1 on signal flips
- Signal flips now recorded with accurate P&L as 'manual' exits
- Add retry logic with exponential backoff for Solana RPC rate limits
- Create /api/trading/cancel-orders endpoint for manual cleanup
- Improves data integrity for win/loss statistics
2025-11-09 17:59:50 +01:00
mindesbunister
309cad8108 Add SQL script to fix SHORT position P&L in database
Corrects all historical SHORT trades affected by the P&L calculation bug.

Results:
- 44 SHORT trades corrected
- Total P&L improved from -$388.82 to -$72.98 (+$315.84 recovered)
- SHORT win rate revealed to be 60.5% (not 39% as incorrectly shown)

Script creates backup table before making changes and includes rollback instructions.
2025-11-07 14:55:13 +01:00
mindesbunister
6c7eaf5f04 Add TP1/SL consistency check on trade restore 2025-11-06 12:18:31 +01:00
mindesbunister
18e3e73e83 feat: refresh exit orders after TP1 and add dry-run harness 2025-11-05 10:00:39 +01:00
mindesbunister
881a99242d feat: Add per-symbol trading controls for SOL and ETH
- Add SymbolSettings interface with enabled/positionSize/leverage fields
- Implement per-symbol ENV variables (SOLANA_*, ETHEREUM_*)
- Add SOL and ETH sections to settings UI with enable/disable toggles
- Add symbol-specific test buttons (SOL LONG/SHORT, ETH LONG/SHORT)
- Update execute and test endpoints to check symbol enabled status
- Add real-time risk/reward calculator per symbol
- Rename 'Position Sizing' to 'Global Fallback' for clarity
- Fix position manager P&L calculation for externally closed positions
- Fix zero P&L bug affecting 12 historical trades
- Add SQL scripts for recalculating historical P&L data
- Move archive TypeScript files to .archive to fix build

Defaults:
- SOL: 10 base × 10x leverage = 100 notional (profit trading)
- ETH:  base × 1x leverage =  notional (data collection)
- Global: 10 × 10x for BTC and other symbols

Configuration priority: Per-symbol ENV > Market config > Global ENV > Defaults
2025-11-03 10:28:48 +01:00
mindesbunister
da72b5de04 Phase 3: TP/SL Optimization Analytics API
- Created /api/analytics/tp-sl-optimization endpoint
- Analyzes historical trades using MAE/MFE data
- Calculates optimal TP1/TP2/SL levels based on percentiles
- Provides win rate, profit factor, and hit rate analysis
- Shows money left on table (MFE - realized P&L)
- Projects impact of optimal levels on future performance

Analytics calculated:
- MAE analysis: avg, median, percentiles, worst
- MFE analysis: avg, median, percentiles, best
- Current level performance: TP1/TP2/SL hit rates
- Optimal recommendations: TP1=50% of avg MFE, TP2=80%, SL=70% of avg MAE
- Projected improvements: win rate change, profit factor, total P&L

Requires 10+ closed trades with MAE/MFE data to generate recommendations
Test script: scripts/test-analytics.sh

Next: Phase 4 (visual dashboard) or wait for trades with MAE/MFE data
2025-10-29 21:11:23 +01:00
mindesbunister
14d5de2c64 chore: Organize workspace structure - move docs, workflows, scripts to subdirectories
Organization:
- Created docs/ with setup/, guides/, history/ subdirectories
- Created workflows/ with trading/, analytics/, telegram/, archive/ subdirectories
- Created scripts/ with docker/, setup/, testing/ subdirectories
- Created tests/ for TypeScript test files
- Created archive/ for unused reference files

Moved files:
- 17 documentation files → docs/
- 16 workflow JSON files → workflows/
- 10 shell scripts → scripts/
- 4 test files → tests/
- 5 unused files → archive/

Updated:
- README.md with new file structure and documentation paths

Deleted:
- data/ (empty directory)
- screenshots/ (empty directory)

Critical files remain in root:
- telegram_command_bot.py (active bot - used by Dockerfile)
- watch-restart.sh (systemd service dependency)
- All Dockerfiles and docker-compose files
- All environment files

Validation:
 Containers running (trading-bot-v4, telegram-trade-bot, postgres)
 API responding (positions endpoint tested)
 Telegram bot functional (/status command tested)
 All critical files present in root

No code changes - purely organizational.
System continues running without interruption.

Recovery: git revert HEAD or git reset --hard cleanup-before
2025-10-27 12:59:25 +01:00