- Add BlockedSignal table with 25 fields for comprehensive signal analysis
- Track all blocked signals with metrics (ATR, ADX, RSI, volume, price position)
- Store quality scores, block reasons, and detailed breakdowns
- Include future fields for automated price analysis (priceAfter1/5/15/30Min)
- Restore signalQualityVersion field to Trade table
Database changes:
- New table: BlockedSignal with indexes on symbol, createdAt, score, blockReason
- Fixed schema drift from manual changes
API changes:
- Modified check-risk endpoint to save blocked signals automatically
- Fixed hasContextMetrics variable scope (moved to line 209)
- Save blocks for: quality score too low, cooldown period, hourly limit
- Use config.minSignalQualityScore instead of hardcoded 60
Database helpers:
- Added createBlockedSignal() function with try/catch safety
- Added getRecentBlockedSignals(limit) for queries
- Added getBlockedSignalsForAnalysis(olderThanMinutes) for automation
Documentation:
- Created BLOCKED_SIGNALS_TRACKING.md with SQL queries and analysis workflow
- Created SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md with 5-phase plan
- Documented data-first approach: collect 10-20 signals before optimization
Rationale:
Only 2 historical trades scored 60-64 (insufficient sample size for threshold decision).
Building data collection infrastructure before making premature optimizations.
Phase 1 (current): Collect blocked signals for 1-2 weeks
Phase 2 (next): Analyze patterns and make data-driven threshold decision
Phase 3-5 (future): Automation and ML optimization
BUG FOUND:
Line 558: tp2SizePercent: config.takeProfit2SizePercent || 100
When config.takeProfit2SizePercent = 0 (TP2-as-runner system), JavaScript's ||
operator treats 0 as falsy and falls back to 100, causing TP2 to close 100%
of remaining position instead of activating trailing stop.
IMPACT:
- On-chain orders placed correctly (line 481 uses ?? correctly)
- Position Manager reads from DB and expects TP2 to close position
- Result: User sees TWO take-profit orders instead of runner system
FIX:
Changed both tp1SizePercent and tp2SizePercent to use ?? operator:
- tp1SizePercent: config.takeProfit1SizePercent ?? 75
- tp2SizePercent: config.takeProfit2SizePercent ?? 0
This allows 0 value to be saved correctly for TP2-as-runner system.
VERIFICATION NEEDED:
Current open SHORT position in database has tp2SizePercent=100 from before
this fix. Next trade will use correct runner system.
- Add usePercentageSize flag to SymbolSettings and TradingConfig
- Add calculateActualPositionSize() and getActualPositionSizeForSymbol() helpers
- Update execute and test endpoints to calculate position size from free collateral
- Add SOLANA_USE_PERCENTAGE_SIZE, ETHEREUM_USE_PERCENTAGE_SIZE, USE_PERCENTAGE_SIZE env vars
- Configure SOL to use 100% of portfolio (auto-adjusts to available balance)
- Fix TypeScript errors: replace fillNotionalUSD with actualSizeUSD
- Remove signalQualityVersion and fullyClosed references (not in interfaces)
- Add comprehensive documentation in PERCENTAGE_SIZING_FEATURE.md
Benefits:
- Prevents insufficient collateral errors by using available balance
- Auto-scales positions as account grows/shrinks
- Maintains risk proportional to capital
- Flexible per-symbol configuration (SOL percentage, ETH fixed)
PROBLEM ANALYSIS:
Signal that lost -$32: ADX 14.8, VOL 2.29x → scored 70-90 (PASSED)
Signal that won +3%: ADX 15.7, VOL 1.18x → scored 45-65 (got BLOCKED before fix)
Key insight: High volume during choppy conditions (ADX < 16) indicates
whipsaw/trap, not genuine breakout. Our volume bonus (+15 pts for >1.5x)
was rewarding flip-flop signals instead of real moves.
FIX:
Add anti-chop filter in volume scoring:
- If ADX < 16 AND volume > 1.5x → -15 points (whipsaw trap)
- Overrides the normal +15 bonus for high volume
- Protects against false signals during consolidation
IMPACT ON RECENT SIGNALS:
1. 00:40 SHORT (ADX 17.2, VOL 0.98): 55→75 ✅ Still passes
2. 00:55 LONG (ADX 15, VOL 0.47): 35→55 ❌ Still blocked (correct, weak vol)
3. 01:05 SHORT (ADX 14.8, VOL 2.29): 70→60 ⚠️ Now flagged as whipsaw trap
4. 01:10 LONG (ADX 15.7, VOL 1.18): 45→65 ✅ Catches the +3% runup
Result: Loser signal now barely passes (60) with warning flag,
winner signal passes cleanly (65). Better risk/reward profile.
- Fix external closure P&L using tp1Hit flag instead of currentSize
- Add direction change detection to prevent false TP1 on signal flips
- Signal flips now recorded with accurate P&L as 'manual' exits
- Add retry logic with exponential backoff for Solana RPC rate limits
- Create /api/trading/cancel-orders endpoint for manual cleanup
- Improves data integrity for win/loss statistics
- Add market data cache service (5min expiry) for storing TradingView metrics
- Create /api/trading/market-data webhook endpoint for continuous data updates
- Add /api/analytics/reentry-check endpoint for validating manual trades
- Update execute endpoint to auto-cache metrics from incoming signals
- Enhance Telegram bot with pre-execution analytics validation
- Support --force flag to override analytics blocks
- Use fresh ADX/ATR/RSI data when available, fallback to historical
- Apply performance modifiers: -20 for losing streaks, +10 for winning
- Minimum re-entry score 55 (vs 60 for new signals)
- Fail-open design: proceeds if analytics unavailable
- Show data freshness and source in Telegram responses
- Add comprehensive setup guide in docs/guides/REENTRY_ANALYTICS_QUICKSTART.md
Phase 1 implementation for smart manual trade validation.
- Add ATR-based dynamic TP2 scaling from 0.7% to 3.0% based on volatility
- New config options: useAtrBasedTargets, atrMultiplierForTp2, minTp2Percent, maxTp2Percent
- Enhanced settings UI with ATR controls and updated risk calculator
- Fix external closure P&L calculation using unrealized P&L instead of volatile current price
- Update execute and test endpoints to use calculateDynamicTp2() function
- Maintain 25% runner system for capturing extended moves (4-5% targets)
- Add environment variables for ATR-based configuration
- Better P&L accuracy for manual position closures
- Fix P&L calculation in Position Manager to use actual entry vs exit price instead of SDK's potentially incorrect realizedPnL
- Calculate actual profit percentage and apply to closed position size for accurate dollar amounts
- Update database record for last trade from incorrect 6.58 to actual .66 P&L
- Update .github/copilot-instructions.md to reflect TP2-as-runner system changes
- Document 25% runner system (5x larger than old 5%) with ATR-based trailing
- Add critical P&L calculation pattern to common pitfalls section
- Mark Phase 5 complete in development roadmap
CHANGE: TP2 now activates trailing stop on full 25% remaining instead
of closing 80% and leaving 5% runner.
Benefits:
- 5x larger runner (25% vs 5%) = 25 vs 05 on 100 position
- Eliminates Drift minimum size issues completely
- Simplifies logic - no more canUseRunner() viability checks
- Better R:R on extended moves
New flow:
- TP1 (+0.4%): Close 75%, keep 25%
- TP2 (+0.7%): Skip close, activate trailing stop on full 25%
- Runner: 25% with ATR-based trailing (0.25-0.9%)
Config change: takeProfit2SizePercent: 80 → 0
Position Manager: Remove canUseRunner logic, activate trailing at TP2 hit
PROBLEM: Runner never activated because Drift force-closes positions below
minimum size. TP2 would close 80% leaving 5% runner (~$105), but Drift
automatically closed the entire position.
SOLUTION:
1. Created runner-calculator.ts with canUseRunner() to check if remaining
size would be above Drift minimums BEFORE executing TP2 close
2. If runner not viable: Skip TP2 close entirely, activate trailing stop
on full 25% remaining (from TP1)
3. If runner viable: Execute TP2 as normal, activate trailing on 5%
Benefits:
- Runner system will now actually work for viable position sizes
- Positions that are too small won't try to force-close below minimums
- Better logs showing why runner did/didn't activate
- Trailing stop works on larger % if runner not viable (better R:R)
Example: $2100 position → $525 after TP1 → $105 runner = VIABLE
$4 ETH position → $1 after TP1 → $0.20 runner = NOT VIABLE
Runner will trail with ATR-based dynamic % (0.25-0.9%) below peak price.
CRITICAL BUG FIX: SHORT positions were calculating P&L with inverted logic,
causing profits to be recorded as losses and vice versa.
Problem Example:
- SHORT at $156.58, exit at $154.66 (price dropped $1.92)
- Should be +~$25 profit
- Was recorded as -$499.23 LOSS
Root Cause:
Old formula: profitPercent = (exit - entry) / entry * (side === 'long' ? 1 : -1)
This multiplied the LONG formula by -1 for shorts, but then applied it to
full notional instead of properly accounting for direction.
Fix:
- LONG: priceDiff = (exit - entry) → profit when price rises
- SHORT: priceDiff = (entry - exit) → profit when price falls
- profitPercent = priceDiff / entry * 100
- Proper leverage calculation: realizedPnL = collateral * profitPercent * leverage
This fixes both dry-run and live close position calculations in lib/drift/orders.ts
Impact: All SHORT trades since bot launch have incorrect P&L in database.
Future trades will calculate correctly.
- Added signalQualityVersion field to Trade model
- Tracks which scoring logic version was used for each trade
- v1: Original logic (price position < 5% threshold)
- v2: Added volume compensation for low ADX
- v3: CURRENT - Stricter logic requiring ADX > 18 for extreme positions (< 15%)
This enables future analysis to:
- Compare performance between logic versions
- Filter trades by scoring algorithm
- Data-driven improvements based on clean datasets
All new trades will be marked as v3. Old trades remain null/v1 for comparison.
- Shorts/longs at < 15% range require ADX > 18 AND volume > 1.2x
- OR RSI < 35 for shorts, RSI > 60 for longs
- Increased penalty from -10 to -15 when conditions not met
- Changed threshold from < 5% to < 15% to catch more edge cases
Test results:
- Big loser (01:35): ADX 16.1, price 9.3% → Score 60 (was 90) → BLOCKED
- Today's signal (10:05): ADX 17.3, price 0.9% → Score 55 (was 85) → BLOCKED
Rationale: False breakdowns in choppy ranges (ADX < 18) cause losses.
Tradeoff: May block some profitable breakdowns, but prevents chop losses.
- Allow shorts at range bottom (<5%) with volume >1.2x OR RSI <40
- Allow longs at range bottom with volume >1.2x OR RSI >60
- Reduce ADX penalty from -15 to -5 when strong volume (>1.2x) present
- Reduce price position penalties from -15 to -10 (less harsh)
- Volume compensation recognizes breakdowns start before ADX strengthens
Test case (blocked signal that would have profited):
- OLD: ATR 0.32, ADX 17.3, RSI 32.5, Vol 1.27x, Price 0.9% → Score 45 (blocked)
- NEW: Same metrics → Score 85 (executes)
Rationale: Breakdowns continue lower, volume confirms conviction, ADX lags price action
- Detect position size mismatches (>50% variance) after opening
- Save phantom trades to database with expectedSizeUSD, actualSizeUSD, phantomReason
- Return error from execute endpoint to prevent Position Manager tracking
- Add comprehensive documentation of phantom trade issue and solution
- Enable data collection for pattern analysis and future optimization
Fixes oracle price lag issue during volatile markets where transactions
confirm but positions don't actually open at expected size.
- Position Manager was calculating P&L using tracked size instead of actual on-chain size
- Example: Tracked 100, actual 0.04 SOL () = -99.63% false loss instead of -0.32%
- Fixed external closure detection to use position.size * currentPrice as lastKnownSize
- Manually corrected phantom trade P&L from -092.25 to /bin/bash
- Total P&L corrected: -013.92 → +8.33 (accurate)
- Prevents all future phantom/mismatch trades from wildly incorrect P&L
Modified:
- lib/trading/position-manager.ts lines 421-445 (external closure P&L calculation)
**Problem:**
When closing small runner positions (5% after TP1+TP2), the calculated size could be below Drift's minimum order size:
- ETH minimum: 0.01 ETH
- After TP1 (75%): 0.0025 ETH left
- After TP2 (80%): 0.0005 ETH runner
- Trailing stop tries to close 0.0005 ETH → ERROR: Below minimum 0.01
n8n showed: "Order size 0.0011 is below minimum 0.01"
**Root Cause:**
closePosition() calculated: sizeToClose = position.size * (percentToClose / 100)
No validation against marketConfig.minOrderSize before submitting to Drift.
**Solution:**
Added minimum size check in closePosition() (lib/drift/orders.ts):
1. Calculate intended close size
2. If below minOrderSize → force 100% close instead
3. Log warning when this happens
4. Prevents Drift API rejection
**Code Change:**
```typescript
let sizeToClose = position.size * (params.percentToClose / 100)
// If calculated size is below minimum, close 100%
if (sizeToClose < marketConfig.minOrderSize) {
console.log('⚠️ Calculated size below minimum - forcing 100% close')
sizeToClose = position.size
}
```
**Impact:**
- ✅ Small runner positions close successfully
- ✅ No more "below minimum" errors from Drift
- Trades complete cleanly
- ⚠️ Runner may close slightly earlier than intended (but better than error)
**Example:**
ETH runner at 0.0005 ETH → tries to close → detects <0.01 → closes entire 0.0005 ETH position at once instead of rejecting.
This is the correct behavior - if the position is already too small, we should close it entirely.
**Feature: Position Scaling**
Allows adding to existing profitable positions when high-quality signals confirm trend strength.
**Configuration (config/trading.ts):**
- enablePositionScaling: false (disabled by default - enable after testing)
- minScaleQualityScore: 75 (higher bar than initial 60)
- minProfitForScale: 0.4% (must be at/past TP1)
- maxScaleMultiplier: 2.0 (max 200% of original size)
- scaleSizePercent: 50% (add 50% of original position)
- minAdxIncrease: 5 (ADX must strengthen)
- maxPricePositionForScale: 70% (don't chase resistance)
**Validation Logic (check-risk endpoint):**
Same-direction signal triggers scaling check if enabled:
1. Quality score ≥75 (stronger than initial entry)
2. Position profitable ≥0.4% (at/past TP1)
3. ADX increased ≥5 points (trend strengthening)
4. Price position <70% (not near resistance)
5. Total size <2x original (risk management)
6. Returns 'allowed: true, reason: Position scaling' if all pass
**Execution (execute endpoint):**
- Opens additional position at scale size (50% of original)
- Updates ActiveTrade: timesScaled, totalScaleAdded, currentSize
- Tracks originalAdx from first entry for comparison
- Returns 'action: scaled' with scale details
**ActiveTrade Interface:**
Added fields:
- originalAdx?: number (for scaling validation)
- timesScaled?: number (track scaling count)
- totalScaleAdded?: number (total USD added)
**Example Scenario:**
1. LONG SOL at $176 (quality: 45, ADX: 13.4) - weak but entered
2. Price hits $176.70 (+0.4%) - at TP1
3. New LONG signal (quality: 78, ADX: 19) - strong confirmation
4. Scaling validation: ✅ Quality 78 ✅ Profit +0.4% ✅ ADX +5.6 ✅ Price 68%
5. Adds 50% more position at $176.70
6. Total position: 150% of original size
**Conservative Design:**
- Disabled by default (requires manual enabling)
- Only scales INTO profitable positions (never averaging down)
- Requires significant quality improvement (75 vs 60)
- Requires trend confirmation (ADX increase)
- Hard cap at 2x original size
- Won't chase near resistance levels
**Next Steps:**
1. Enable in settings: ENABLE_POSITION_SCALING=true
2. Test with small positions first
3. Monitor data: do scaled positions outperform?
4. Adjust thresholds based on results
**Safety:**
- All existing duplicate prevention logic intact
- Flip logic unchanged (still requires quality check)
- Position Manager tracks scaling state
- Can be toggled on/off without code changes
**Root Causes:**
1. Auto-flip logic could create phantom trades if close failed
2. Position size mismatches (0.01 SOL vs 11.92 SOL expected) not caught
3. Multiple trades for same symbol+direction in database
**Preventive Measures:**
1. **Startup Validation (lib/startup/init-position-manager.ts)**
- Validates all open trades against Drift positions on startup
- Auto-closes phantom trades with <50% expected size
- Logs size mismatches for manual review
- Prevents Position Manager from tracking ghost positions
2. **Duplicate Position Prevention (app/api/trading/execute/route.ts)**
- Blocks opening same-direction position on same symbol
- Returns 400 error if duplicate detected
- Only allows auto-flip (opposite direction close + open)
3. **Runtime Phantom Detection (lib/trading/position-manager.ts)**
- Checks position size every 2s monitoring cycle
- Auto-closes if size ratio <50% (extreme mismatch)
- Logs as 'manual' exit with AUTO_CLEANUP tx
- Removes from monitoring immediately
4. **Quality Score Fix (app/api/trading/check-risk/route.ts)**
- Hardcoded minScore=60 (removed non-existent config reference)
**Prevention Summary:**
- ✅ Startup validation catches historical phantoms
- ✅ Duplicate check prevents new phantoms
- ✅ Runtime detection catches size mismatches <30s after they occur
- ✅ All three layers work together for defense-in-depth
Issue: User had LONG (phantom) + SHORT (undersized 0.01 SOL vs 11.92 expected)
Fix: Both detected and closed, bot now clean with 0 active trades
- Add SymbolSettings interface with enabled/positionSize/leverage fields
- Implement per-symbol ENV variables (SOLANA_*, ETHEREUM_*)
- Add SOL and ETH sections to settings UI with enable/disable toggles
- Add symbol-specific test buttons (SOL LONG/SHORT, ETH LONG/SHORT)
- Update execute and test endpoints to check symbol enabled status
- Add real-time risk/reward calculator per symbol
- Rename 'Position Sizing' to 'Global Fallback' for clarity
- Fix position manager P&L calculation for externally closed positions
- Fix zero P&L bug affecting 12 historical trades
- Add SQL scripts for recalculating historical P&L data
- Move archive TypeScript files to .archive to fix build
Defaults:
- SOL: 10 base × 10x leverage = 100 notional (profit trading)
- ETH: base × 1x leverage = notional (data collection)
- Global: 10 × 10x for BTC and other symbols
Configuration priority: Per-symbol ENV > Market config > Global ENV > Defaults
CRITICAL: Cooldown was global across ALL symbols, causing missed opportunities
Example: ETH trade at 10:00 blocked SOL trade at 10:04 (5min cooldown)
Changes:
- Added getLastTradeTimeForSymbol() function to query last trade per symbol
- Updated check-risk endpoint to use symbol-specific cooldown
- Each coin (SOL/ETH/BTC) now has independent cooldown timer
- Cooldown message shows symbol: 'Must wait X min before next SOL-PERP trade'
Result: Can trade ETH and SOL simultaneously without interference
Example: ETH LONG at 10:00, SOL SHORT at 10:01 = both allowed
CRITICAL BUG: Position Manager was using current price to determine exit reason,
but on-chain orders filled at a DIFFERENT price in the past!
Example: LONG entry $184.55, TP1 filled at $184.66, but when Position Manager
checked later (price dropped), it saw currentPrice < TP1 and defaulted to 'SL'
Result: Profitable trades incorrectly labeled as SL exits in database
Fix:
- Use trade.tp1Hit and trade.tp2Hit flags to determine exit reason
- If no TP flags set, use realized P&L to distinguish:
- Profit >0.5% = TP1 filled
- Negative P&L = SL filled
- Remove duplicate P&L calculation
This ensures exit reasons match actual on-chain order fills
Added Maximum Favorable/Adverse Excursion tracking:
- Track maxFavorableExcursion: best profit % reached during trade
- Track maxAdverseExcursion: worst loss % reached during trade
- Track maxFavorablePrice and maxAdversePrice
- Update every price check (2s interval)
- Save to database on trade exit for optimization analysis
Benefits:
- Identify if TP levels are too conservative (MFE consistently higher)
- Determine if SL is too tight (MAE < SL but trade recovers)
- Optimize runner size based on how often MFE >> TP2
- Data-driven exit strategy tuning after collecting 10-20 trades
Display in monitoring logs: Shows MFE/MAE % every 20 seconds
CRITICAL: After TP1 closes 75%, the on-chain stop loss orders were NOT being updated
- Position Manager was tracking new SL price internally but not updating Drift orders
- Old SL orders (e.g., $181.69) remained active even after TP1 at $185.28
- This prevented the 'move SL to breakeven after TP1' logic from working
Fix:
- After TP1 hits, cancel ALL old orders on-chain
- Place new SL orders at updated price (breakeven + configured %)
- Place remaining TP2 order for the 25% runner position
- Maintains dual-stop system if enabled
Result: SL will now actually move up on Drift UI after TP1 fires
- Convert closePosition.closedSize (base asset) to USD when updating trade.currentSize
- Fix conversion when position.size detected from Drift: set currentSize = position.size * currentPrice
- Prevent trade.currentSize from being reduced to tiny values due to unit mismatch
- Position Manager now detects TP1/TP2 fills by monitoring position size reductions instead of entry price mismatches
- When position size reduces by ~75%, marks TP1 as filled and updates currentSize
- When position size reduces by ~95%, marks TP2 as filled and activates trailing stop for 5% runner
- Entry price mismatch check now skipped after TP fills (Drift shows weighted average entry price after partial closes)
- Fixes bug where runners were incorrectly closed after TP1/TP2 fired on-chain
- Adds grace period for new trades (<30s) to avoid false positives during blockchain propagation delays
- This unblocks Phase 1 data collection for signal quality optimization (need 10+ trades with MAE/MFE data)
- Add qualityScore to ExecuteTradeResponse interface and response object
- Update analytics page to always show Signal Quality card (N/A if unavailable)
- Fix n8n workflow to pass context metrics and qualityScore to execute endpoint
- Fix timezone in Telegram notifications (Europe/Berlin)
- Fix symbol normalization in /api/trading/close endpoint
- Update Drift ETH-PERP minimum order size (0.002 ETH not 0.01)
- Add transaction confirmation to closePosition() to prevent phantom closes
- Add 30-second grace period for new trades in Position Manager
- Fix execution order: database save before Position Manager.addTrade()
- Update copilot instructions with transaction confirmation pattern
- Added getConnection() method to DriftService
- Added proper transaction confirmation in openPosition()
- Check confirmation.value.err to detect on-chain failures
- Return error if transaction fails instead of assuming success
- Prevents phantom trades that never actually execute
This fixes the issue where bot was recording trades with transaction
signatures that don't exist on-chain (like 2gqrPxnvGzdRp56...).
- Add signalQualityScore field to Trade model (0-100)
- Calculate quality score in execute endpoint using same logic as check-risk
- Save score with every trade for correlation analysis
- Create database migration for new field
- Enables future analysis: score vs win rate, P&L, etc.
This allows data-driven decisions on dynamic position sizing
- Add getLastTrade() function to database service
- Create /api/analytics/last-trade endpoint
- Display last trade with full details on analytics page
- Show entry/exit prices, P&L, position size, targets
- Visual indicators for trade direction and exit reason
- Helps quickly diagnose where trades went (TP1, TP2, or SL)
Bug: Position Manager was comparing ANY position on the symbol to the trade being
tracked, without verifying entry price match. When a new position opened, it would
think the old tracked trade 'closed externally' and cancel ALL orders - including
the new position's exit orders.
Fix: Added entry price verification (0.5% tolerance). If position entry price doesn't
match the tracked trade, mark the old trade as 'lost tracking' and remove from
monitoring WITHOUT cancelling orders (they belong to the new position).
This prevents the catastrophic scenario where exit orders are repeatedly cancelled,
leaving positions unprotected.
