CRITICAL FIX - Parallel Execution Now Working:
- Problem: coordinator blocked on subprocess.run(ssh_cmd) preventing worker2 deployment
- Root cause #1: subprocess.run() waits for SSH FDs even with 'nohup &' and '-f' flag
- Root cause #2: Indicator deployed to backtester/ subdirectory instead of workspace root
- Solution #1: Replace subprocess.run() with subprocess.Popen() + communicate(timeout=2)
- Solution #2: Deploy v11_moneyline_all_filters.py to workspace root for direct import
- Result: Both workers start simultaneously (worker1 chunk 0, worker2 chunk 1)
- Impact: 2× speedup achieved (15 min vs 30 min sequential)
Verification:
- Worker1: 31 processes, generating 1,125+ signals per config ✓
- Worker2: 29 processes, generating 848-898 signals per config ✓
- Coordinator: Both chunks active, parallel deployment in 12 seconds ✓
User concern addressed: 'if we are not using them in parallel how are we supposed
to gain a time advantage?' - Now using them in parallel, gaining 2× advantage.
Files modified:
- cluster/v11_test_coordinator.py (lines 287-301: Popen + timeout, lines 238-255: workspace root)
THREE critical bugs in cluster/v11_test_worker.py:
1. Missing use_quality_filters parameter when creating MoneyLineV11Inputs
- Parameter defaults to True but wasn't being passed explicitly
- Fix: Added use_quality_filters=True to inputs creation
2. Missing fixed RSI parameters (rsi_long_max, rsi_short_min)
- Worker only passed rsi_long_min and rsi_short_max (sweep params)
- Missing rsi_long_max=70 and rsi_short_min=30 (fixed params)
- Fix: Added both fixed parameters to inputs creation
3. Import path mismatch - worker imported OLD version
- Worker added cluster/ to sys.path, imported from parent directory
- Old v11_moneyline_all_filters.py (21:40) missing use_quality_filters
- Fixed v11_moneyline_all_filters.py was in backtester/ subdirectory
- Fix: Deployed corrected file to /home/comprehensive_sweep/
Result: 0 signals → 1,096-1,186 signals per config ✓
Verified: Local test (314 signals), EPYC dataset test (1,186 signals),
Worker log now shows signal variety across 27 concurrent configs.
Progressive sweep now running successfully on EPYC cluster.
TWO CRITICAL BUGS FIXED:
1. Missing use_quality_filters parameter (Pine Script parity):
- Added use_quality_filters: bool = True to MoneyLineV11Inputs
- Implemented bypass logic in signal generation for both long/short
- When False: only trend flips generate signals (no filtering)
- When True: all filters must pass (original v11 behavior)
- Matches Pine Script: finalSignal = buyReady and (not useQualityFilters or (...filters...))
2. RSI index misalignment causing 100% NaN values:
- np.where() returns numpy arrays without indices
- pd.Series(gain/loss) created NEW integer indices (0,1,2...)
- Result: RSI values misaligned with original datetime index
- Fix: pd.Series(gain/loss, index=series.index) preserves alignment
- Impact: RSI NaN count 100 → 0, all filters now work correctly
VERIFICATION:
- Test 1 (no filters): 1,424 signals ✓
- Test 2 (permissive RSI): 1,308 signals ✓
- Test 3 (moderate RSI 25-70/30-80): 1,157 signals ✓
Progressive sweep can now proceed with corrected signal generation.
ISSUE: Quality 95 trade stopped out today (ID: cmiueo2qv01coml07y9kjzugf)
but stop hunt was NOT recorded in database for revenge system.
ROOT CAUSE: logger.log() calls for revenge recording were silenced in production
(NODE_ENV=production suppresses logger.log output)
FIX: Changed 2 logger.log() calls to console.log() in position-manager.ts:
- Line ~1006: External closure revenge eligibility check
- Line ~1742: Software-based SL revenge activation
Now revenge system will properly record quality 85+ stop-outs with visible logs.
Trade details:
- Symbol: SOL-PERP LONG
- Entry: $133.74, Exit: $132.69
- Quality: 95, ADX: 28.9, ATR: 0.22
- Loss: -$26.94
- Exit time: 2025-12-06 15:16:18
This stop-out already expired (4-hour window ended at 19:16).
Next quality 85+ SL will be recorded correctly.
Added to copilot-instructions.md Common Pitfalls section:
PITFALL #73: Service Initialization Never Ran (Dec 5, 2025)
- Duration: 16 days (Nov 19 - Dec 5)
- Financial impact: 00-1,400 (k user estimate)
- Root cause: Services after validation with early return
- Affected: Stop hunt revenge, smart validation, blocked signal tracker, data cleanup
- Fix: Move services BEFORE validation (commits 51b63f4, f6c9a7b, 35c2d7f)
- Prevention: Test suite, CI/CD, startup health checks, console.log for critical logs
- Full docs: docs/CRITICAL_SERVICE_INITIALIZATION_BUG_DEC5_2025.md
DISCOVERY (Dec 5, 2025):
- 4 critical services never started since Nov 19 (16 days)
- Services placed AFTER validation with early return
- Silent failure: no errors, just never initialized
AFFECTED SERVICES:
- Stop Hunt Revenge Tracker (Nov 20) - No revenge attempts
- Smart Entry Validation (Nov 30) - Manual trades used stale data
- Blocked Signal Tracker (Nov 19) - No threshold optimization data
- Data Cleanup (Dec 2) - Database bloat
FINANCIAL IMPACT:
- Stop hunt revenge: 00-600 lost (missed reversals)
- Smart validation: 00-400 lost (stale data entries)
- Blocked signals: 00-400 lost (suboptimal thresholds)
- TOTAL: 00-1,400 (user estimate: ,000)
ROOT CAUSE:
Line 43: validateOpenTrades() with early return at line 111
Lines 59-72: Service initialization AFTER validation
Result: When no open trades → services never reached
FIX COMMITS:
- 51b63f4: Move services BEFORE validation
- f6c9a7b: Use console.log for production visibility
- 35c2d7f: Fix stop hunt tracker logs
PREVENTION:
- Test suite (PR #2): 113 tests
- CI/CD pipeline (PR #5): Automated quality gates
- Service startup validation in future CI
- Production logging standard: console.log for critical operations
STATUS: ✅ ALL SERVICES NOW ACTIVE AND VERIFIED
- logger.log is silenced in production (NODE_ENV=production)
- Service initialization logs were hidden even though services were starting
- Changed to console.log for visibility in production logs
- Affects: data cleanup, blocked signal tracker, stop hunt tracker, smart validation
CRITICAL BUG DISCOVERED (Dec 5, 2025):
- validateOpenTrades() returns early at line 111 when no trades found
- Service initialization (lines 59-72) happened AFTER validation
- Result: When no open trades, services NEVER started
- Impact: Stop hunt tracker, smart validation, blocked signal tracking all inactive
ROOT CAUSE:
- Line 43: await validateOpenTrades()
- Line 111: if (openTrades.length === 0) return // EXIT EARLY
- Lines 59-72: Service startup code (NEVER REACHED)
FIX:
- Moved service initialization BEFORE validation
- Services now start regardless of open trades count
- Order: Start services → Clean DB → Validate → Init Position Manager
SERVICES NOW START:
- Data cleanup (4-week retention)
- Blocked signal price tracker
- Stop hunt revenge tracker
- Smart entry validation system
This explains why:
- Line 111 log appeared (validation ran, returned early)
- Line 29 log appeared (function started)
- Lines 59-72 logs NEVER appeared (code never reached)
Git commit SHA: TBD
Deployment: Requires rebuild + restart
- Lines 68-72 had only 2 spaces indent (outside try block)
- Services were executing AFTER catch block
- Fixed to 4 spaces (inside try block)
- Now stop hunt tracker, blocked signal tracker, smart validation will initialize properly
- 1-minute data is pure market sampling, not trading signals
- signalQualityVersion now null for timeframe='1'
- Other timeframes still labeled with v9
- Prevents confusion in analytics/reporting
Add `instrumentationHook: true` to `next.config.js` experimental section.
This fixes a critical bug where the instrumentation.ts file was not being
executed on server startup, causing all startup services to not run:
- Stop Hunt Revenge Tracker (93 revenge opportunities missed)
- Position Manager (no monitoring of open trades)
- Ghost Position Cleanup
- Data Cleanup Service
- Blocked Signal Tracking
- Smart Validation Queue
- Database Sync Validator
The instrumentation.ts file existed and contained proper initialization code,
but Next.js requires explicit opt-in via the experimental.instrumentationHook
flag to enable this feature.
Co-authored-by: mindesbunister <32161838+mindesbunister@users.noreply.github.com>
- Add version dropdown selector (v9, v8, v6, v5, all) to frontend
- Update backend API to accept ?version= query parameter
- Add version filter to all 5 broken SQL queries using Prisma parameterized queries
- Update Data Collection Status to use selected version instead of hardcoded v8
- Add version context to all recommendations
- Add URL encoding for version parameter (security best practice)
- Validate version parameter against whitelist (SQL injection protection)
Co-authored-by: mindesbunister <32161838+mindesbunister@users.noreply.github.com>
- Added STEP 1: Run tests BEFORE deployment (113 tests in ~30s)
- Added STEP 2: Validate with test trade AFTER deployment
- Why mandatory: Catch bugs (tokens vs USD, false TP1, wrong SL) before real money loss
- Tests prevent Common Pitfalls #24, #43, #45, #52, #54, #67 recurrence
- DO NOT deploy if tests fail - fix issue or update tests first
Added comprehensive feature discovery section to copilot-instructions.md:
- Quick Reference Table: 9 common scenarios with existing features
- Quick Search Commands: bash commands for feature discovery
- Feature Discovery by Category: 6 categories with 30+ features
- Decision Flowchart: 5-step verification process
- Historical Examples: why each feature was built
This helps users/AI agents discover existing features before rebuilding.
Co-authored-by: mindesbunister <32161838+mindesbunister@users.noreply.github.com>
