Commit Graph

184 Commits

Author SHA1 Message Date
mindesbunister
7ff78ee0bd feat: Hybrid RPC fallback system (Alchemy → Helius)
- Automatic fallback after 2 consecutive rate limits
- Primary: Alchemy (300M CU/month, stable for normal ops)
- Fallback: Helius (10 req/sec, backup for startup bursts)
- Reduced startup validation: 6h window, 5 trades (was 24h, 20 trades)
- Multi-position safety check (prevents order cancellation conflicts)
- Rate limit-aware retry logic with exponential backoff

Implementation:
- lib/drift/client.ts: Added fallbackConnection, switchToFallbackRpc()
- .env: SOLANA_FALLBACK_RPC_URL configuration
- lib/startup/init-position-manager.ts: Reduced validation scope
- lib/trading/position-manager.ts: Multi-position order protection

Tested: System switched to fallback on startup, Position Manager active
Result: 1 active trade being monitored after automatic RPC switch
2025-11-14 15:28:07 +01:00
mindesbunister
7afd7d5aa1 feat: switch from Helius to Alchemy RPC provider
Changes:
- Updated SOLANA_RPC_URL to use Alchemy (https://solana-mainnet.g.alchemy.com/v2/...)
- Migrated from Helius free tier to Alchemy free tier
- Includes previous rate limit fixes (8s backoff, 2s operation delays)

Context:
- Helius free tier: 10 req/sec sustained, 100 req/sec burst
- Alchemy free tier: 300M compute units/month (more generous)
- User hit 239 rate limit errors in 10 minutes on Helius
- User registered Alchemy account and provided API key

Impact:
- Should significantly reduce 429 rate limit errors
- Better free tier limits for trading bot operations
- Combined with delay fixes for optimal RPC usage
2025-11-14 14:01:52 +01:00
mindesbunister
a0dc80e96b docs: add Docker cleanup instructions to prevent disk full issues
- Document build cache accumulation problem (40-50 GB typical)
- Add cleanup commands: image prune, builder prune, volume prune
- Recommend running after each deployment or weekly
- Typical space freed: 40-55 GB per cleanup
- Clarify what's safe vs not safe to delete
- Part of maintaining healthy development environment
2025-11-14 10:46:15 +01:00
mindesbunister
27eb5d4fe8 fix: Critical rate limit handling + startup position restoration
**Problem 1: Rate Limit Cascade**
- Position Manager tried to close repeatedly, overwhelming Helius RPC (10 req/s limit)
- Base retry delay was too aggressive (2s → 4s → 8s)
- No graceful handling when 429 errors occur

**Problem 2: Orphaned Positions After Restart**
- Container restarts lost Position Manager state
- Positions marked 'closed' in DB but still open on Drift (failed close transactions)
- No cross-validation between database and actual Drift positions

**Solutions Implemented:**

1. **Increased retry delays (orders.ts)**:
   - Base delay: 2s → 5s (progression now 5s → 10s → 20s)
   - Reduces RPC pressure during rate limit situations
   - Gives Helius time to recover between retries
   - Documented Helius limits: 100 req/s burst, 10 req/s sustained (free tier)

2. **Startup position validation (init-position-manager.ts)**:
   - Cross-checks last 24h of 'closed' trades against actual Drift positions
   - If DB says closed but Drift shows open → reopens in DB to restore tracking
   - Prevents unmonitored positions from existing after container restarts
   - Logs detailed mismatch info for debugging

3. **Rate limit-aware exit handling (position-manager.ts)**:
   - Detects 429 errors during position close
   - Keeps trade in monitoring instead of removing it
   - Natural retry on next price update (vs aggressive 2s loop)
   - Prevents marking position as closed when transaction actually failed

**Impact:**
- Eliminates orphaned positions after restarts
- Reduces RPC pressure by 2.5x (5s vs 2s base delay)
- Graceful degradation under rate limits
- Position Manager continues monitoring even during temporary RPC issues

**Testing needed:**
- Monitor next container restart to verify position restoration works
- Check rate limit analytics after next close attempt
- Verify no more phantom 'closed' positions when Drift shows open
2025-11-14 09:50:13 +01:00
mindesbunister
795026aed1 fix: use Pyth price data for flip-flop context check
CRITICAL FIX: Previous implementation showed incorrect price movements
(100% instead of 0.2%) because currentPrice wasn't available in
check-risk endpoint.

Changes:
- app/api/trading/check-risk/route.ts: Fetch current price from Pyth
  price monitor before quality scoring
- lib/trading/signal-quality.ts: Added validation and detailed logging
  - Check if currentPrice available, apply penalty if missing
  - Log actual prices: $X → $Y = Z%
  - Include prices in penalty/allowance messages

Example outputs:
 Flip-flop in tight range: 4min ago, only 0.20% move ($143.86 → $143.58) (-25 pts)
 Direction change after 10.2% move ($170.00 → $153.00, 12min ago) - reversal allowed

This fixes the false positive that allowed a 0.2% flip-flop earlier today.

Deployed: 09:42 CET Nov 14, 2025
2025-11-14 08:23:04 +01:00
mindesbunister
77a9437d26 feat: add price movement context to flip-flop detection
Improved flip-flop penalty logic to distinguish between:
- Chop (bad): <2% price move from opposite signal → -25 penalty
- Reversal (good): ≥2% price move from opposite signal → allowed

Changes:
- lib/database/trades.ts: getRecentSignals() now returns oppositeDirectionPrice
- lib/trading/signal-quality.ts: Added currentPrice parameter, price movement check
- app/api/trading/check-risk/route.ts: Added currentPrice to RiskCheckRequest interface
- app/api/trading/execute/route.ts: Pass openResult.fillPrice as currentPrice
- app/api/analytics/reentry-check/route.ts: Pass currentPrice from metrics

Example scenarios:
- ETH $170 SHORT → $153 LONG (10% move) = reversal allowed 
- ETH $154.50 SHORT → $154.30 LONG (0.13% move) = chop blocked ⚠️

Deployed: 09:18 CET Nov 14, 2025
Container: trading-bot-v4
2025-11-14 07:46:28 +01:00
mindesbunister
111e3ed12a feat: implement signal frequency penalties for flip-flop detection
PHASE 1 IMPLEMENTATION:
Signal quality scoring now checks database for recent trading patterns
and applies penalties to prevent overtrading and flip-flop losses.

NEW PENALTIES:
1. Overtrading: 3+ signals in 30min → -20 points
   - Detects consolidation zones where system generates excessive signals
   - Counts both executed trades AND blocked signals

2. Flip-flop: Opposite direction in last 15min → -25 points
   - Prevents rapid long→short→long whipsaws
   - Example: SHORT at 10:00, LONG at 10:12 = blocked

3. Alternating pattern: Last 3 trades flip directions → -30 points
   - Detects choppy market conditions
   - Pattern like long→short→long = system getting chopped

DATABASE INTEGRATION:
- New function: getRecentSignals() in lib/database/trades.ts
- Queries last 30min of trades + blocked signals
- Checks last 3 executed trades for alternating pattern
- Zero performance impact (fast indexed queries)

ARCHITECTURE:
- scoreSignalQuality() now async (requires database access)
- All callers updated: check-risk, execute, reentry-check
- skipFrequencyCheck flag available for special cases
- Frequency penalties included in qualityResult breakdown

EXPECTED IMPACT:
- Eliminate overnight flip-flop losses (like SOL $141-145 chop)
- Reduce overtrading during sideways consolidation
- Better capital preservation in non-trending markets
- Should improve win rate by 5-10% by avoiding worst setups

TESTING:
- Deploy and monitor next 5 signals in choppy markets
- Check logs for frequency penalty messages
- Analyze if blocked signals would have been losers

Files changed:
- lib/database/trades.ts: Added getRecentSignals()
- lib/trading/signal-quality.ts: Made async, added frequency checks
- app/api/trading/check-risk/route.ts: await + symbol parameter
- app/api/trading/execute/route.ts: await + symbol parameter
- app/api/analytics/reentry-check/route.ts: await + skipFrequencyCheck
2025-11-14 06:41:03 +01:00
mindesbunister
31bc08bed4 fix: TP1/TP2 race condition causing multiple simultaneous closures
CRITICAL BUG FIX:
- Position Manager monitoring loop (every 2s) could trigger TP1/TP2 multiple times
- tp1Hit flag was set AFTER async executeExit() completed
- Multiple concurrent executeExit() calls happened before flag was set
- Result: Position closed 6 times (70% close × 6 = entire position + failed attempts)

ROOT CAUSE:
- Race window: ~0.5-1s between check and flag set
- Multiple monitoring loops entered if statement simultaneously

FIX APPLIED:
- Set tp1Hit = true IMMEDIATELY before calling executeExit()
- Same fix for tp2Hit flag
- Prevents concurrent execution by setting flag synchronously

EVIDENCE:
- Test trade at 04:47:09: TP1 triggered 6 times
- First close: Remaining $13.52 (correct 30%)
- Closes 2-6: Remaining $0.00 (closed entire position)
- Position Manager continued tracking $13.02 runner that didn't exist

IMPACT:
- User had unprotected $42.73 position (Position Manager tracking phantom)
- No TP/SL monitoring, no trailing stop
- Had to manually close position

Files changed:
- lib/trading/position-manager.ts: Move tp1Hit/tp2Hit flag setting before async calls
- Prevents race condition on all future trades

Testing required: Execute test trade and verify TP1 triggers only once.
2025-11-14 06:26:59 +01:00
mindesbunister
6590f4fb1e feat: phantom trade auto-closure system
- Auto-close phantom positions immediately via market order
- Return HTTP 200 (not 500) to allow n8n workflow continuation
- Save phantom trades to database with full P&L tracking
- Exit reason: 'manual' category for phantom auto-closes
- Protects user during unavailable hours (sleeping, no phone)
- Add Docker build best practices to instructions (background + tail)
- Document phantom system as Critical Component #1
- Add Common Pitfall #30: Phantom notification workflow

Why auto-close:
- User can't always respond to phantom alerts
- Unmonitored position = unlimited risk exposure
- Better to exit with small loss/gain than leave exposed
- Re-entry possible if setup actually good

Files changed:
- app/api/trading/execute/route.ts: Auto-close logic
- .github/copilot-instructions.md: Documentation + build pattern
2025-11-14 05:37:51 +01:00
mindesbunister
5e826dee5d Add DNS retry logic to Drift initialization
- Handles transient network failures (EAI_AGAIN, ENOTFOUND, ETIMEDOUT)
- Automatically retries up to 3 times with 2s delay between attempts
- Logs retry attempts for monitoring
- Prevents 500 errors from temporary DNS hiccups
- Fixes: n8n workflow failures during brief network issues

Impact:
- Improves reliability during DNS/network instability
- Reduces false negatives (missed trades due to transient errors)
- User-friendly retry logs for diagnostics
2025-11-13 16:05:42 +01:00
mindesbunister
bd9633fbc2 CRITICAL FIX: Prevent unprotected positions via database-first pattern
Root Cause:
- Execute endpoint saved to database AFTER adding to Position Manager
- Database save failures were silently caught and ignored
- API returned success even when DB save failed
- Container restarts lost in-memory Position Manager state
- Result: Unprotected positions with no TP/SL monitoring

Fixes Applied:

1. Database-First Pattern (app/api/trading/execute/route.ts):
   - MOVED createTrade() BEFORE positionManager.addTrade()
   - If database save fails, return HTTP 500 with critical error
   - Error message: 'CLOSE POSITION MANUALLY IMMEDIATELY'
   - Position Manager only tracks database-persisted trades
   - Ensures container restarts can restore all positions

2. Transaction Timeout (lib/drift/orders.ts):
   - Added 30s timeout to confirmTransaction() in closePosition()
   - Prevents API from hanging during network congestion
   - Uses Promise.race() pattern for timeout enforcement

3. Telegram Error Messages (telegram_command_bot.py):
   - Parse JSON for ALL responses (not just 200 OK)
   - Extract detailed error messages from 'message' field
   - Shows critical warnings to user immediately
   - Fail-open: proceeds if analytics check fails

4. Position Manager (lib/trading/position-manager.ts):
   - Move lastPrice update to TOP of monitoring loop
   - Ensures /status endpoint always shows current price

Verification:
- Test trade cmhxj8qxl0000od076m21l58z executed successfully
- Database save completed BEFORE Position Manager tracking
- SL triggered correctly at -$4.21 after 15 minutes
- All protection systems working as expected

Impact:
- Eliminates risk of unprotected positions
- Provides immediate critical warnings if DB fails
- Enables safe container restarts with full position recovery
- Verified with live test trade on production

See: CRITICAL_INCIDENT_UNPROTECTED_POSITION.md for full incident report
2025-11-13 15:56:28 +01:00
mindesbunister
a21ae6d622 Add v7-momentum indicator (experimental, disabled)
- Created momentum scalper indicator for catching rapid price acceleration
- ROC-based detection: 2.0% threshold over 5 bars
- Volume confirmation: 2.0x spike (checks last 3 bars)
- ADX filter: Requires 12+ minimum directional movement
- Anti-chop filter: Blocks signals in dead markets
- Debug table: Real-time metric display for troubleshooting

Status: Functional but signal quality inferior to v6 HalfTrend
Decision: Shelved for now, continue with proven v6 strategy
File: docs/guides/MOMENTUM_INDICATOR_V1.pine (239 lines)

Lessons learned:
- Momentum indicators inherently noisy (40-50% WR expected)
- Signals either too early (false breakouts) or too late (miss move)
- Volume spike timing issue: Often lags price move by 1-2 bars
- Better to optimize proven strategy than add complexity

Related: Position Manager duplicate update bug fixed (awaiting verification)
2025-11-12 19:55:19 +01:00
mindesbunister
bba58da8fa CRITICAL FIX: position.size is tokens not USD
Fixed Position Manager incorrectly treating position.size as USD when
Drift SDK actually returns base asset tokens (SOL, ETH, BTC).

Impact:
- FALSE TP1 detections (12.28 SOL misinterpreted as 2.28 USD)
- Stop loss moved to breakeven prematurely
- Runner system activated incorrectly
- Positions stuck in wrong state

Changes:
- Line 322: Convert position.size to USD: position.size * currentPrice
- Line 519: Calculate positionSizeUSD before comparison
- Line 558: Use positionSizeUSD directly (already in USD)
- Line 591: Save positionSizeUSD (no price multiplication needed)

Before: Compared 12.28 tokens < 1950 USD = 99.4% reduction = FALSE TP1

This was causing current trade to think TP1 hit when position is still 100% open.
2025-11-12 11:30:47 +01:00
mindesbunister
7f9dcc00e2 feat: add indicatorVersion to CreateTradeParams interface and createTrade function 2025-11-12 08:29:40 +01:00
mindesbunister
03e91fc18d feat: ATR-based trailing stop + rate limit monitoring
MAJOR FIXES:
- ATR-based trailing stop for runners (was fixed 0.3%, now adapts to volatility)
- Fixes runners with +7-9% MFE exiting for losses
- Typical improvement: 2.24x more room (0.3% → 0.67% at 0.45% ATR)
- Enhanced rate limit logging with database tracking
- New /api/analytics/rate-limits endpoint for monitoring

DETAILS:
- Position Manager: Calculate trailing as (atrAtEntry / price × 100) × multiplier
- Config: TRAILING_STOP_ATR_MULTIPLIER=1.5, MIN=0.25%, MAX=0.9%
- Settings UI: Added ATR multiplier controls
- Rate limits: Log hits/recoveries/exhaustions to SystemEvent table
- Documentation: ATR_TRAILING_STOP_FIX.md + RATE_LIMIT_MONITORING.md

IMPACT:
- Runners can now capture big moves (like morning's $172→$162 SOL drop)
- Rate limit visibility prevents silent failures
- Data-driven optimization for RPC endpoint health
2025-11-11 14:51:41 +01:00
mindesbunister
ba13c20c60 feat: implement blocked signals tracking system
- Add BlockedSignal table with 25 fields for comprehensive signal analysis
- Track all blocked signals with metrics (ATR, ADX, RSI, volume, price position)
- Store quality scores, block reasons, and detailed breakdowns
- Include future fields for automated price analysis (priceAfter1/5/15/30Min)
- Restore signalQualityVersion field to Trade table

Database changes:
- New table: BlockedSignal with indexes on symbol, createdAt, score, blockReason
- Fixed schema drift from manual changes

API changes:
- Modified check-risk endpoint to save blocked signals automatically
- Fixed hasContextMetrics variable scope (moved to line 209)
- Save blocks for: quality score too low, cooldown period, hourly limit
- Use config.minSignalQualityScore instead of hardcoded 60

Database helpers:
- Added createBlockedSignal() function with try/catch safety
- Added getRecentBlockedSignals(limit) for queries
- Added getBlockedSignalsForAnalysis(olderThanMinutes) for automation

Documentation:
- Created BLOCKED_SIGNALS_TRACKING.md with SQL queries and analysis workflow
- Created SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md with 5-phase plan
- Documented data-first approach: collect 10-20 signals before optimization

Rationale:
Only 2 historical trades scored 60-64 (insufficient sample size for threshold decision).
Building data collection infrastructure before making premature optimizations.

Phase 1 (current): Collect blocked signals for 1-2 weeks
Phase 2 (next): Analyze patterns and make data-driven threshold decision
Phase 3-5 (future): Automation and ML optimization
2025-11-11 11:49:21 +01:00
mindesbunister
c3a053df63 CRITICAL FIX: Use ?? instead of || for tp2SizePercent to allow 0 value
BUG FOUND:
Line 558: tp2SizePercent: config.takeProfit2SizePercent || 100

When config.takeProfit2SizePercent = 0 (TP2-as-runner system), JavaScript's ||
operator treats 0 as falsy and falls back to 100, causing TP2 to close 100%
of remaining position instead of activating trailing stop.

IMPACT:
- On-chain orders placed correctly (line 481 uses ?? correctly)
- Position Manager reads from DB and expects TP2 to close position
- Result: User sees TWO take-profit orders instead of runner system

FIX:
Changed both tp1SizePercent and tp2SizePercent to use ?? operator:
- tp1SizePercent: config.takeProfit1SizePercent ?? 75
- tp2SizePercent: config.takeProfit2SizePercent ?? 0

This allows 0 value to be saved correctly for TP2-as-runner system.

VERIFICATION NEEDED:
Current open SHORT position in database has tp2SizePercent=100 from before
this fix. Next trade will use correct runner system.
2025-11-10 19:46:03 +01:00
mindesbunister
988fdb9ea4 Fix runner system + strengthen anti-chop filter
Three critical bugs fixed:
1. P&L calculation (65x inflation) - now uses collateralUSD not notional
2. handlePostTp1Adjustments() - checks tp2SizePercent===0 for runner mode
3. JavaScript || operator bug - changed to ?? for proper 0 handling

Signal quality improvements:
- Added anti-chop filter: price position <40% + ADX <25 = -25 points
- Prevents range-bound flip-flops (caught all 3 today)
- Backtest: 43.8% → 55.6% win rate, +86% profit per trade

Changes:
- lib/trading/signal-quality.ts: RANGE-BOUND CHOP penalty
- lib/drift/orders.ts: Fixed P&L calculation + transaction confirmation
- lib/trading/position-manager.ts: Runner system logic
- app/api/trading/execute/route.ts: || to ?? for tp2SizePercent
- app/api/trading/test/route.ts: || to ?? for tp1/tp2SizePercent
- prisma/schema.prisma: Added collateralUSD field
- scripts/fix_pnl_calculations.sql: Historical P&L correction
2025-11-10 15:36:51 +01:00
mindesbunister
6f0a1bb49b feat: Implement percentage-based position sizing
- Add usePercentageSize flag to SymbolSettings and TradingConfig
- Add calculateActualPositionSize() and getActualPositionSizeForSymbol() helpers
- Update execute and test endpoints to calculate position size from free collateral
- Add SOLANA_USE_PERCENTAGE_SIZE, ETHEREUM_USE_PERCENTAGE_SIZE, USE_PERCENTAGE_SIZE env vars
- Configure SOL to use 100% of portfolio (auto-adjusts to available balance)
- Fix TypeScript errors: replace fillNotionalUSD with actualSizeUSD
- Remove signalQualityVersion and fullyClosed references (not in interfaces)
- Add comprehensive documentation in PERCENTAGE_SIZING_FEATURE.md

Benefits:
- Prevents insufficient collateral errors by using available balance
- Auto-scales positions as account grows/shrinks
- Maintains risk proportional to capital
- Flexible per-symbol configuration (SOL percentage, ETH fixed)
2025-11-10 13:35:10 +01:00
mindesbunister
d2fbd125a0 fix: Make minSignalQualityScore configurable via settings + anti-chop improvements
CRITICAL BUG FIX:
- Settings page saved MIN_SIGNAL_QUALITY_SCORE to .env but check-risk had hardcoded value
- Now reads from config.minSignalQualityScore (defaults to 65, editable via /settings)
- Prevents settings changes from being ignored after restart

ANTI-CHOP FILTER FIXES:
- Fixed volume breakout bonus conflicting with anti-chop filter
- Volume breakout now requires ADX > 18 (trending market)
- Prevents high volume + low ADX from getting rewarded instead of penalized
- Anti-chop filter now properly blocks whipsaw traps at score 60

TESTING INFRASTRUCTURE:
- Added backtest script showing +17.1% P&L improvement (saved $242 in losses)
- Added test-signals.sh for comprehensive signal quality validation
- Added test-recent-signals.sh for analyzing actual trading session signals
- All tests passing: timeframe awareness, anti-chop, score thresholds

CHANGES:
- config/trading.ts: Added minSignalQualityScore to interface and defaults
- app/api/trading/check-risk/route.ts: Use config value instead of hardcoded 65
- lib/trading/signal-quality.ts: Fixed volume breakout bonus logic
- .env: Added MIN_SIGNAL_QUALITY_SCORE=65
- scripts/: Added comprehensive testing tools

BACKTEST RESULTS (Last 30 trades):
- Old system (score ≥60): $1,412.79 P&L
- New system (score ≥65 + anti-chop): $1,654.79 P&L
- Improvement: +$242.00 (+17.1%)
- Blocked 5 losing trades, missed 0 winners
2025-11-10 11:22:52 +01:00
mindesbunister
60a0035f56 Add anti-chop filter: Penalize high volume during weak trend
PROBLEM ANALYSIS:
Signal that lost -$32: ADX 14.8, VOL 2.29x → scored 70-90 (PASSED)
Signal that won +3%: ADX 15.7, VOL 1.18x → scored 45-65 (got BLOCKED before fix)

Key insight: High volume during choppy conditions (ADX < 16) indicates
whipsaw/trap, not genuine breakout. Our volume bonus (+15 pts for >1.5x)
was rewarding flip-flop signals instead of real moves.

FIX:
Add anti-chop filter in volume scoring:
- If ADX < 16 AND volume > 1.5x → -15 points (whipsaw trap)
- Overrides the normal +15 bonus for high volume
- Protects against false signals during consolidation

IMPACT ON RECENT SIGNALS:
1. 00:40 SHORT (ADX 17.2, VOL 0.98): 55→75  Still passes
2. 00:55 LONG (ADX 15, VOL 0.47): 35→55  Still blocked (correct, weak vol)
3. 01:05 SHORT (ADX 14.8, VOL 2.29): 70→60 ⚠️ Now flagged as whipsaw trap
4. 01:10 LONG (ADX 15.7, VOL 1.18): 45→65  Catches the +3% runup

Result: Loser signal now barely passes (60) with warning flag,
winner signal passes cleanly (65). Better risk/reward profile.
2025-11-10 07:46:46 +01:00
mindesbunister
4b11186d16 Fix: Add timeframe-aware signal quality scoring for 5min charts
PROBLEM:
- Long signal (ADX 15.7, ATR 0.35%) blocked with score 45/100
- Missed major +3% runup, lost -2 on short that didn't flip
- Scoring logic treated all timeframes identically (daily chart thresholds)

ROOT CAUSE:
- ADX < 18 always scored -15 points regardless of timeframe
- 5min charts naturally have lower ADX (12-22 healthy range)
- copilot-instructions mentioned timeframe awareness but wasn't implemented

FIX:
- Add timeframe parameter to RiskCheckRequest interface
- Update scoreSignalQuality() with timeframe-aware ADX thresholds:
  * 5min/15min: ADX 12-22 healthy (+5), <12 weak (-15), >22 strong (+15)
  * Higher TF: ADX 18-25 healthy (+5), <18 weak (-15), >25 strong (+15)
- Pass timeframe from n8n workflow through check-risk and execute
- Update both Check Risk nodes in Money Machine workflow

IMPACT:
Your blocked signal (ADX 15.7 on 5min) now scores:
- Was: 50 + 5 - 15 + 0 + 0 + 5 = 45 (BLOCKED)
- Now: 50 + 5 + 5 + 0 + 0 + 5 = 65 (PASSES)

This 20-point improvement from timeframe awareness would have caught the runup.
2025-11-10 07:34:21 +01:00
mindesbunister
22195ed34c Fix P&L calculation and signal flip detection
- Fix external closure P&L using tp1Hit flag instead of currentSize
- Add direction change detection to prevent false TP1 on signal flips
- Signal flips now recorded with accurate P&L as 'manual' exits
- Add retry logic with exponential backoff for Solana RPC rate limits
- Create /api/trading/cancel-orders endpoint for manual cleanup
- Improves data integrity for win/loss statistics
2025-11-09 17:59:50 +01:00
mindesbunister
9b767342dc feat: Implement re-entry analytics system with fresh TradingView data
- Add market data cache service (5min expiry) for storing TradingView metrics
- Create /api/trading/market-data webhook endpoint for continuous data updates
- Add /api/analytics/reentry-check endpoint for validating manual trades
- Update execute endpoint to auto-cache metrics from incoming signals
- Enhance Telegram bot with pre-execution analytics validation
- Support --force flag to override analytics blocks
- Use fresh ADX/ATR/RSI data when available, fallback to historical
- Apply performance modifiers: -20 for losing streaks, +10 for winning
- Minimum re-entry score 55 (vs 60 for new signals)
- Fail-open design: proceeds if analytics unavailable
- Show data freshness and source in Telegram responses
- Add comprehensive setup guide in docs/guides/REENTRY_ANALYTICS_QUICKSTART.md

Phase 1 implementation for smart manual trade validation.
2025-11-07 20:40:07 +01:00
mindesbunister
6d5991172a feat: Implement ATR-based dynamic TP2 system and fix P&L calculation
- Add ATR-based dynamic TP2 scaling from 0.7% to 3.0% based on volatility
- New config options: useAtrBasedTargets, atrMultiplierForTp2, minTp2Percent, maxTp2Percent
- Enhanced settings UI with ATR controls and updated risk calculator
- Fix external closure P&L calculation using unrealized P&L instead of volatile current price
- Update execute and test endpoints to use calculateDynamicTp2() function
- Maintain 25% runner system for capturing extended moves (4-5% targets)
- Add environment variables for ATR-based configuration
- Better P&L accuracy for manual position closures
2025-11-07 17:01:22 +01:00
mindesbunister
5acc61cf66 Fix P&L calculation and update Copilot instructions
- Fix P&L calculation in Position Manager to use actual entry vs exit price instead of SDK's potentially incorrect realizedPnL
- Calculate actual profit percentage and apply to closed position size for accurate dollar amounts
- Update database record for last trade from incorrect 6.58 to actual .66 P&L
- Update .github/copilot-instructions.md to reflect TP2-as-runner system changes
- Document 25% runner system (5x larger than old 5%) with ATR-based trailing
- Add critical P&L calculation pattern to common pitfalls section
- Mark Phase 5 complete in development roadmap
2025-11-07 16:24:43 +01:00
mindesbunister
0c644ccabe Make TP2 the runner - no more partial closes
CHANGE: TP2 now activates trailing stop on full 25% remaining instead
of closing 80% and leaving 5% runner.

Benefits:
- 5x larger runner (25% vs 5%) = 25 vs 05 on 100 position
- Eliminates Drift minimum size issues completely
- Simplifies logic - no more canUseRunner() viability checks
- Better R:R on extended moves

New flow:
- TP1 (+0.4%): Close 75%, keep 25%
- TP2 (+0.7%): Skip close, activate trailing stop on full 25%
- Runner: 25% with ATR-based trailing (0.25-0.9%)

Config change: takeProfit2SizePercent: 80 → 0
Position Manager: Remove canUseRunner logic, activate trailing at TP2 hit
2025-11-07 15:29:50 +01:00
mindesbunister
36ba3809a1 Fix runner system by checking minimum position size viability
PROBLEM: Runner never activated because Drift force-closes positions below
minimum size. TP2 would close 80% leaving 5% runner (~$105), but Drift
automatically closed the entire position.

SOLUTION:
1. Created runner-calculator.ts with canUseRunner() to check if remaining
   size would be above Drift minimums BEFORE executing TP2 close
2. If runner not viable: Skip TP2 close entirely, activate trailing stop
   on full 25% remaining (from TP1)
3. If runner viable: Execute TP2 as normal, activate trailing on 5%

Benefits:
- Runner system will now actually work for viable position sizes
- Positions that are too small won't try to force-close below minimums
- Better logs showing why runner did/didn't activate
- Trailing stop works on larger % if runner not viable (better R:R)

Example: $2100 position → $525 after TP1 → $105 runner = VIABLE
         $4 ETH position → $1 after TP1 → $0.20 runner = NOT VIABLE

Runner will trail with ATR-based dynamic % (0.25-0.9%) below peak price.
2025-11-07 15:10:01 +01:00
mindesbunister
4996bc2aad Fix SHORT position P&L calculation bug
CRITICAL BUG FIX: SHORT positions were calculating P&L with inverted logic,
causing profits to be recorded as losses and vice versa.

Problem Example:
- SHORT at $156.58, exit at $154.66 (price dropped $1.92)
- Should be +~$25 profit
- Was recorded as -$499.23 LOSS

Root Cause:
Old formula: profitPercent = (exit - entry) / entry * (side === 'long' ? 1 : -1)
This multiplied the LONG formula by -1 for shorts, but then applied it to
full notional instead of properly accounting for direction.

Fix:
- LONG: priceDiff = (exit - entry) → profit when price rises
- SHORT: priceDiff = (entry - exit) → profit when price falls
- profitPercent = priceDiff / entry * 100
- Proper leverage calculation: realizedPnL = collateral * profitPercent * leverage

This fixes both dry-run and live close position calculations in lib/drift/orders.ts

Impact: All SHORT trades since bot launch have incorrect P&L in database.
Future trades will calculate correctly.
2025-11-07 14:53:03 +01:00
mindesbunister
625dc44c59 Add signal quality version tracking to database
- Added signalQualityVersion field to Trade model
- Tracks which scoring logic version was used for each trade
- v1: Original logic (price position < 5% threshold)
- v2: Added volume compensation for low ADX
- v3: CURRENT - Stricter logic requiring ADX > 18 for extreme positions (< 15%)

This enables future analysis to:
- Compare performance between logic versions
- Filter trades by scoring algorithm
- Data-driven improvements based on clean datasets

All new trades will be marked as v3. Old trades remain null/v1 for comparison.
2025-11-07 12:56:35 +01:00
mindesbunister
3c9da22a8a Add ADX > 18 requirement for extreme price positions
- Shorts/longs at < 15% range require ADX > 18 AND volume > 1.2x
- OR RSI < 35 for shorts, RSI > 60 for longs
- Increased penalty from -10 to -15 when conditions not met
- Changed threshold from < 5% to < 15% to catch more edge cases

Test results:
- Big loser (01:35): ADX 16.1, price 9.3% → Score 60 (was 90) → BLOCKED
- Today's signal (10:05): ADX 17.3, price 0.9% → Score 55 (was 85) → BLOCKED

Rationale: False breakdowns in choppy ranges (ADX < 18) cause losses.
Tradeoff: May block some profitable breakdowns, but prevents chop losses.
2025-11-07 12:19:41 +01:00
mindesbunister
db907d8074 Improve signal quality scoring for breakdowns/breakouts
- Allow shorts at range bottom (<5%) with volume >1.2x OR RSI <40
- Allow longs at range bottom with volume >1.2x OR RSI >60
- Reduce ADX penalty from -15 to -5 when strong volume (>1.2x) present
- Reduce price position penalties from -15 to -10 (less harsh)
- Volume compensation recognizes breakdowns start before ADX strengthens

Test case (blocked signal that would have profited):
- OLD: ATR 0.32, ADX 17.3, RSI 32.5, Vol 1.27x, Price 0.9% → Score 45 (blocked)
- NEW: Same metrics → Score 85 (executes)

Rationale: Breakdowns continue lower, volume confirms conviction, ADX lags price action
2025-11-07 10:58:47 +01:00
mindesbunister
0365560c5b Add timeframe-aware signal quality scoring for 5min charts
- Lower ADX/ATR thresholds for 5min timeframe (ADX 12-22, ATR 0.2-0.7%)
- Add anti-chop filter: -20 points for extreme sideways (ADX<10, ATR<0.25, Vol<0.9)
- Pass timeframe parameter through check-risk and execute endpoints
- Fixes flip-flop losses from overly strict 5min filters
- Higher timeframes unchanged (still use ADX 18+, ATR 0.4+)

5min scoring now:
- ADX 12-15: moderate trend (+5)
- ADX 22+: strong trend (+15)
- ATR 0.2-0.35: acceptable (+5)
- ATR 0.35+: healthy (+10)
- Extreme chop penalty prevents whipsaw trades
2025-11-07 08:56:19 +01:00
mindesbunister
6c7eaf5f04 Add TP1/SL consistency check on trade restore 2025-11-06 12:18:31 +01:00
mindesbunister
7c888282ec Adjust TP detection logic for partial fills 2025-11-05 23:49:41 +01:00
mindesbunister
5241920d44 Prevent repeated TP2 cleanup loops 2025-11-05 16:14:17 +01:00
mindesbunister
a100945864 Enhance trailing stop with ATR-based sizing 2025-11-05 15:28:12 +01:00
mindesbunister
149294084e fix: auto-clean leftovers after stop hits 2025-11-05 11:42:22 +01:00
mindesbunister
b58e08778e fix: correct MFE/MAE tracking after partial exits 2025-11-05 10:29:32 +01:00
mindesbunister
18e3e73e83 feat: refresh exit orders after TP1 and add dry-run harness 2025-11-05 10:00:39 +01:00
mindesbunister
cbb6592153 fix: correct PnL math and add health probe 2025-11-05 07:58:27 +01:00
mindesbunister
02193b7dce fix(critical): Unify quality score calculation across check-risk and execute
PROBLEM:
- check-risk calculated quality score: 60, 70 (PASSED)
- execute calculated quality score: 35, 45 (should have BLOCKED)
- Two different functions with different logic caused trades to bypass validation

ROOT CAUSE:
Two separate scoring functions existed:
1. scoreSignalQuality() in check-risk (detailed, 95% price threshold)
2. calculateQualityScore() in execute (simpler, 90% price threshold)

Example with pricePosition=96.4%, volumeRatio=0.9:
- check-risk: Checks >95, volumeRatio>1.4 failed → -15 + bonuses = 60  PASSED
- execute: Checks >90 → -15 + bonuses = 35  Should block but already opened

SOLUTION:
1. Created lib/trading/signal-quality.ts with unified scoreSignalQuality()
2. Both endpoints now import and use SAME function
3. Consistent scoring logic: 95% price threshold, volume breakout bonus
4. Returns detailed reasons for debugging

IMPACT:
- Quality scores now MATCH between check-risk and execute
- No more trades bypassing validation due to calculation differences
- Better debugging with quality reasons logged

Files changed:
- NEW: lib/trading/signal-quality.ts (unified scoring function)
- MODIFIED: app/api/trading/check-risk/route.ts (import shared function)
- MODIFIED: app/api/trading/execute/route.ts (import shared function)
- REMOVED: Duplicate calculateQualityScore() from execute
- REMOVED: Duplicate scoreSignalQuality() from check-risk
2025-11-04 11:40:25 +01:00
mindesbunister
8bc08955cc feat: Add phantom trade detection and database tracking
- Detect position size mismatches (>50% variance) after opening
- Save phantom trades to database with expectedSizeUSD, actualSizeUSD, phantomReason
- Return error from execute endpoint to prevent Position Manager tracking
- Add comprehensive documentation of phantom trade issue and solution
- Enable data collection for pattern analysis and future optimization

Fixes oracle price lag issue during volatile markets where transactions
confirm but positions don't actually open at expected size.
2025-11-04 10:34:38 +01:00
mindesbunister
1426a9ec2f CRITICAL FIX: P&L calculation using wrong position size for phantom trades
- Position Manager was calculating P&L using tracked size instead of actual on-chain size
- Example: Tracked 100, actual 0.04 SOL () = -99.63% false loss instead of -0.32%
- Fixed external closure detection to use position.size * currentPrice as lastKnownSize
- Manually corrected phantom trade P&L from -092.25 to /bin/bash
- Total P&L corrected: -013.92 → +8.33 (accurate)
- Prevents all future phantom/mismatch trades from wildly incorrect P&L

Modified:
- lib/trading/position-manager.ts lines 421-445 (external closure P&L calculation)
2025-11-03 16:57:53 +01:00
mindesbunister
cfc15cd3b0 fix: Prevent runner positions from being below minimum order size
**Problem:**
When closing small runner positions (5% after TP1+TP2), the calculated size could be below Drift's minimum order size:
- ETH minimum: 0.01 ETH
- After TP1 (75%): 0.0025 ETH left
- After TP2 (80%): 0.0005 ETH runner
- Trailing stop tries to close 0.0005 ETH → ERROR: Below minimum 0.01

n8n showed: "Order size 0.0011 is below minimum 0.01"

**Root Cause:**
closePosition() calculated: sizeToClose = position.size * (percentToClose / 100)
No validation against marketConfig.minOrderSize before submitting to Drift.

**Solution:**
Added minimum size check in closePosition() (lib/drift/orders.ts):
1. Calculate intended close size
2. If below minOrderSize → force 100% close instead
3. Log warning when this happens
4. Prevents Drift API rejection

**Code Change:**
```typescript
let sizeToClose = position.size * (params.percentToClose / 100)

// If calculated size is below minimum, close 100%
if (sizeToClose < marketConfig.minOrderSize) {
  console.log('⚠️ Calculated size below minimum - forcing 100% close')
  sizeToClose = position.size
}
```

**Impact:**
-  Small runner positions close successfully
-  No more "below minimum" errors from Drift
-  Trades complete cleanly
- ⚠️ Runner may close slightly earlier than intended (but better than error)

**Example:**
ETH runner at 0.0005 ETH → tries to close → detects <0.01 → closes entire 0.0005 ETH position at once instead of rejecting.

This is the correct behavior - if the position is already too small, we should close it entirely.
2025-11-03 15:59:31 +01:00
mindesbunister
8a8d4a348c feat: Add position scaling for strong confirmation signals
**Feature: Position Scaling**
Allows adding to existing profitable positions when high-quality signals confirm trend strength.

**Configuration (config/trading.ts):**
- enablePositionScaling: false (disabled by default - enable after testing)
- minScaleQualityScore: 75 (higher bar than initial 60)
- minProfitForScale: 0.4% (must be at/past TP1)
- maxScaleMultiplier: 2.0 (max 200% of original size)
- scaleSizePercent: 50% (add 50% of original position)
- minAdxIncrease: 5 (ADX must strengthen)
- maxPricePositionForScale: 70% (don't chase resistance)

**Validation Logic (check-risk endpoint):**
Same-direction signal triggers scaling check if enabled:
1. Quality score ≥75 (stronger than initial entry)
2. Position profitable ≥0.4% (at/past TP1)
3. ADX increased ≥5 points (trend strengthening)
4. Price position <70% (not near resistance)
5. Total size <2x original (risk management)
6. Returns 'allowed: true, reason: Position scaling' if all pass

**Execution (execute endpoint):**
- Opens additional position at scale size (50% of original)
- Updates ActiveTrade: timesScaled, totalScaleAdded, currentSize
- Tracks originalAdx from first entry for comparison
- Returns 'action: scaled' with scale details

**ActiveTrade Interface:**
Added fields:
- originalAdx?: number (for scaling validation)
- timesScaled?: number (track scaling count)
- totalScaleAdded?: number (total USD added)

**Example Scenario:**
1. LONG SOL at $176 (quality: 45, ADX: 13.4) - weak but entered
2. Price hits $176.70 (+0.4%) - at TP1
3. New LONG signal (quality: 78, ADX: 19) - strong confirmation
4. Scaling validation:  Quality 78  Profit +0.4%  ADX +5.6  Price 68%
5. Adds 50% more position at $176.70
6. Total position: 150% of original size

**Conservative Design:**
- Disabled by default (requires manual enabling)
- Only scales INTO profitable positions (never averaging down)
- Requires significant quality improvement (75 vs 60)
- Requires trend confirmation (ADX increase)
- Hard cap at 2x original size
- Won't chase near resistance levels

**Next Steps:**
1. Enable in settings: ENABLE_POSITION_SCALING=true
2. Test with small positions first
3. Monitor data: do scaled positions outperform?
4. Adjust thresholds based on results

**Safety:**
- All existing duplicate prevention logic intact
- Flip logic unchanged (still requires quality check)
- Position Manager tracks scaling state
- Can be toggled on/off without code changes
2025-11-03 15:35:33 +01:00
mindesbunister
6b1d32a72d fix: Add phantom trade detection and prevention safeguards
**Root Causes:**
1. Auto-flip logic could create phantom trades if close failed
2. Position size mismatches (0.01 SOL vs 11.92 SOL expected) not caught
3. Multiple trades for same symbol+direction in database

**Preventive Measures:**

1. **Startup Validation (lib/startup/init-position-manager.ts)**
   - Validates all open trades against Drift positions on startup
   - Auto-closes phantom trades with <50% expected size
   - Logs size mismatches for manual review
   - Prevents Position Manager from tracking ghost positions

2. **Duplicate Position Prevention (app/api/trading/execute/route.ts)**
   - Blocks opening same-direction position on same symbol
   - Returns 400 error if duplicate detected
   - Only allows auto-flip (opposite direction close + open)

3. **Runtime Phantom Detection (lib/trading/position-manager.ts)**
   - Checks position size every 2s monitoring cycle
   - Auto-closes if size ratio <50% (extreme mismatch)
   - Logs as 'manual' exit with AUTO_CLEANUP tx
   - Removes from monitoring immediately

4. **Quality Score Fix (app/api/trading/check-risk/route.ts)**
   - Hardcoded minScore=60 (removed non-existent config reference)

**Prevention Summary:**
-  Startup validation catches historical phantoms
-  Duplicate check prevents new phantoms
-  Runtime detection catches size mismatches <30s after they occur
-  All three layers work together for defense-in-depth

Issue: User had LONG (phantom) + SHORT (undersized 0.01 SOL vs 11.92 expected)
Fix: Both detected and closed, bot now clean with 0 active trades
2025-11-03 13:53:12 +01:00
mindesbunister
881a99242d feat: Add per-symbol trading controls for SOL and ETH
- Add SymbolSettings interface with enabled/positionSize/leverage fields
- Implement per-symbol ENV variables (SOLANA_*, ETHEREUM_*)
- Add SOL and ETH sections to settings UI with enable/disable toggles
- Add symbol-specific test buttons (SOL LONG/SHORT, ETH LONG/SHORT)
- Update execute and test endpoints to check symbol enabled status
- Add real-time risk/reward calculator per symbol
- Rename 'Position Sizing' to 'Global Fallback' for clarity
- Fix position manager P&L calculation for externally closed positions
- Fix zero P&L bug affecting 12 historical trades
- Add SQL scripts for recalculating historical P&L data
- Move archive TypeScript files to .archive to fix build

Defaults:
- SOL: 10 base × 10x leverage = 100 notional (profit trading)
- ETH:  base × 1x leverage =  notional (data collection)
- Global: 10 × 10x for BTC and other symbols

Configuration priority: Per-symbol ENV > Market config > Global ENV > Defaults
2025-11-03 10:28:48 +01:00
mindesbunister
0ed2e89c7e feat: implement per-symbol cooldown period
CRITICAL: Cooldown was global across ALL symbols, causing missed opportunities
Example: ETH trade at 10:00 blocked SOL trade at 10:04 (5min cooldown)

Changes:
- Added getLastTradeTimeForSymbol() function to query last trade per symbol
- Updated check-risk endpoint to use symbol-specific cooldown
- Each coin (SOL/ETH/BTC) now has independent cooldown timer
- Cooldown message shows symbol: 'Must wait X min before next SOL-PERP trade'

Result: Can trade ETH and SOL simultaneously without interference
Example: ETH LONG at 10:00, SOL SHORT at 10:01 = both allowed
2025-11-03 08:01:30 +01:00
mindesbunister
0ea8773bdc fix: detect exit reason using trade state flags instead of current price
CRITICAL BUG: Position Manager was using current price to determine exit reason,
but on-chain orders filled at a DIFFERENT price in the past!

Example: LONG entry $184.55, TP1 filled at $184.66, but when Position Manager
checked later (price dropped), it saw currentPrice < TP1 and defaulted to 'SL'

Result: Profitable trades incorrectly labeled as SL exits in database

Fix:
- Use trade.tp1Hit and trade.tp2Hit flags to determine exit reason
- If no TP flags set, use realized P&L to distinguish:
  - Profit >0.5% = TP1 filled
  - Negative P&L = SL filled
- Remove duplicate P&L calculation

This ensures exit reasons match actual on-chain order fills
2025-11-03 00:02:19 +01:00