19 Commits

Author SHA1 Message Date
mindesbunister
6990f20d6f feat: Orderbook shadow logging system - Phase 1 complete
Implementation:
- Added 7 orderbook fields to Trade model (spreadBps, imbalanceRatio, depths, impact, walls)
- Oracle-based estimates with 2bps spread assumption
- ENV flag: ENABLE_ORDERBOOK_LOGGING (defaults true)
- Execute wrapper lines 1037-1053 guards orderbook logic

Database:
- Direct SQL ALTER TABLE (avoided migration drift issues)
- All columns nullable DOUBLE PRECISION
- Prisma schema synced via db pull + generate

Deployment:
- Container rebuilt and deployed successfully
- All 7 columns verified accessible
- System operational, ready for live trade validation

Files changed:
- config/trading.ts (enableOrderbookLogging flag, line 127)
- types/trading.ts (orderbook interfaces)
- lib/database/trades.ts (createTrade saves orderbook data)
- app/api/trading/execute/route.ts (ENV wrapper lines 1037-1053)
- prisma/schema.prisma (7 orderbook fields)
- docs/ORDERBOOK_SHADOW_LOGGING.md (complete documentation)

Status:  PRODUCTION READY - awaiting first trade for validation
2025-12-19 08:51:36 +01:00
mindesbunister
b11da009eb critical: Bug #89 - Detect and handle Drift fractional position remnants (3-part fix)
- Part 1: Position Manager fractional remnant detection after close attempts
  * Check if position < 1.5× minOrderSize after close transaction
  * Log to persistent logger with FRACTIONAL_REMNANT_DETECTED
  * Track closeAttempts, limit to 3 maximum
  * Mark exitReason='FRACTIONAL_REMNANT' in database
  * Remove from monitoring after 3 failed attempts

- Part 2: Pre-close validation in closePosition()
  * Check if position viable before attempting close
  * Reject positions < 1.5× minOrderSize with specific error
  * Prevent wasted transaction attempts on too-small positions
  * Return POSITION_TOO_SMALL_TO_CLOSE error with manual instructions

- Part 3: Health monitor detection for fractional remnants
  * Query Trade table for FRACTIONAL_REMNANT exits in last 24h
  * Alert operators with position details and manual cleanup instructions
  * Provide trade IDs, symbols, and Drift UI link

- Database schema: Added closeAttempts Int? field to Track attempts

Root cause: Drift protocol exchange constraints can leave fractional positions
Evidence: 3 close transactions confirmed but 0.15 SOL remnant persisted
Financial impact: ,000+ risk from unprotected fractional positions
Status: Fix implemented, awaiting deployment verification

See: docs/COMMON_PITFALLS.md Bug #89 for complete incident details
2025-12-16 22:05:12 +01:00
mindesbunister
93dd950821 critical: Fix ghost detection P&L compounding - delete from Map BEFORE check
Bug: Multiple monitoring loops detect ghost simultaneously
- Loop 1: has(tradeId) → true → proceeds
- Loop 2: has(tradeId) → true → ALSO proceeds (race condition)
- Both send Telegram notifications with compounding P&L

Real incident (Dec 2, 2025):
- Manual SHORT at $138.84
- 23 duplicate notifications
- P&L compounded: -$47.96 → -$1,129.24 (23× accumulation)
- Database shows single trade with final compounded value

Fix: Map.delete() returns true if key existed, false if already removed
- Call delete() FIRST
- Check return value
 proceeds
- All other loops get false → skip immediately
- Atomic operation prevents race condition

Pattern: This is variant of Common Pitfalls #48, #49, #59, #60, #61
- All had "check then delete" pattern
- All vulnerable to async timing issues
- Solution: "delete then check" pattern
- Map.delete() is synchronous and atomic

Files changed:
- lib/trading/position-manager.ts lines 390-410

Related: DUPLICATE PREVENTED message was working but too late
2025-12-02 18:25:56 +01:00
mindesbunister
5773d7d36d feat: Extend 1-minute data retention from 4 weeks to 1 year
- Updated lib/maintenance/data-cleanup.ts retention period: 28 days → 365 days
- Storage requirements validated: 251 MB/year (negligible)
- Rationale: 13× more historical data for better pattern analysis
- Benefits: 260-390 blocked signals/year vs 20-30/month
- Cleanup cutoff: Now Dec 2, 2024 (vs Nov 4, 2025 previously)
- Deployment verified: Container restarted, cleanup scheduled for 3 AM daily
2025-12-02 11:55:36 +01:00
mindesbunister
db52299b55 feat: Enhancement #6 data collection + #1 implementation plan
Enhancement #6 - SL Distance Validation (Data Collection Phase):
- Added slDistanceAtEntry field to StopHunt schema
- Calculates distance from revenge entry to stop zone (LONG vs SHORT logic)
- Logs distance in dollars + × ATR multiplier
- Purpose: Collect 20+ revenge trade samples for optimal multiplier analysis
- Created comprehensive analysis guide with SQL queries
- Decision deferred until empirical data collected

Enhancement #1 - ADX Confirmation (Implementation Plan):
- Documented complete 1-minute TradingView alert strategy
- Storage analysis: 19.44 MB/month for 3 symbols (negligible)
- Two-phase approach: Cache-only MVP → Optional DB persistence
- Provided TradingView Pine Script (ready to use)
- Cost breakdown: Pro subscription $49.95/month required
- Benefits: Real-time ADX, pattern recognition, ML features
- Implementation checklist with validation phases

Files Changed:
- prisma/schema.prisma: +1 field (slDistanceAtEntry)
- lib/trading/stop-hunt-tracker.ts: +10 lines (distance calculation + logging)
- docs/1MIN_MARKET_DATA_IMPLEMENTATION.md: NEW (comprehensive plan)
- docs/ENHANCEMENT_6_ANALYSIS_GUIDE.md: NEW (SQL queries + decision matrix)

Status:
 Enhancement #4 and #10 deployed (previous commit)
 Enhancement #6 data collection enabled (this commit)
   Awaiting 20+ revenge trades for Enhancement #6 decision
2025-11-27 08:26:45 +01:00
mindesbunister
ceb84c3bc1 feat: Revenge system enhancements #4 and #10 - IMPLEMENTED
Enhancement #4: Failed Revenge Tracking
- Added 3 database fields: revengeOutcome, revengePnL, revengeFailedReason
- Added updateRevengeOutcome() method in stop-hunt-tracker.ts
- Position Manager hooks revenge trade closes, records outcome
- Enables data-driven analysis of revenge success rate

Enhancement #10: Metadata Persistence
- Added 4 database fields: firstCrossTime, lowestInZone, highestInZone, zoneResetCount
- Migrated 90-second zone tracking from in-memory to database
- Rewrote shouldExecuteRevenge() with database persistence
- Container restarts now preserve exact zone tracking state

Technical Details:
- Prisma schema updated with 7 new StopHunt fields
- Added signalSource field to ActiveTrade interface
- All zone metadata persisted in real-time to database
- Build verified successful (no TypeScript errors)

Files Changed:
- prisma/schema.prisma (StopHunt model + index)
- lib/trading/stop-hunt-tracker.ts (DB persistence + outcome tracking)
- lib/trading/position-manager.ts (revenge hook + interface)
- docs/REVENGE_ENHANCEMENTS_EXPLAINED.md (comprehensive guide)

Pending User Decision:
- Enhancement #1: ADX confirmation (3 options explained in docs)
- Enhancement #6: SL distance validation (2× ATR recommended)

Status: Ready for deployment after Prisma migration
Date: Nov 27, 2025
2025-11-27 08:08:37 +01:00
mindesbunister
702e027aba feat: Stop Hunt Revenge System - DEPLOYED (Nov 20, 2025)
Automatically re-enters positions after high-quality signals get stopped out

Features:
- Tracks quality 85+ signals that get stopped out
- Monitors for price reversal through original entry (4-hour window)
- Executes revenge trade at 1.2x size (recover losses faster)
- Telegram notification: 🔥 REVENGE TRADE ACTIVATED
- Database: StopHunt table with 20 fields, 4 indexes
- Monitoring: 30-second checks for active stop hunts

Technical:
- Fixed: Database query hanging in startStopHuntTracking()
- Solution: Added try-catch with error handling
- Import path: Corrected to use '../database/trades'
- Singleton pattern: Single tracker instance per server
- Integration: Position Manager records on SL close

Files:
- lib/trading/stop-hunt-tracker.ts (293 lines, 8 methods)
- lib/startup/init-position-manager.ts (startup integration)
- lib/trading/position-manager.ts (recording logic, ready for next deployment)
- prisma/schema.prisma (StopHunt model)

Commits: Import fix, debug logs, error handling, cleanup
Tested: Container starts successfully, tracker initializes, database query works
Status: 100% operational, waiting for first quality 85+ stop-out to test live
2025-11-20 19:17:43 +01:00
mindesbunister
60fc571aa6 feat: Automated multi-timeframe price tracking system
Implemented comprehensive price tracking for multi-timeframe signal analysis.

**Components Added:**
- lib/analysis/blocked-signal-tracker.ts - Background job tracking prices
- app/api/analytics/signal-tracking/route.ts - Status/metrics endpoint

**Features:**
- Automatic price tracking at 1min, 5min, 15min, 30min intervals
- TP1/TP2/SL hit detection using ATR-based targets
- Max favorable/adverse excursion tracking (MFE/MAE)
- Analysis completion after 30 minutes
- Background job runs every 5 minutes
- Entry price captured from signal time

**Database Changes:**
- Added entryPrice field to BlockedSignal (for price tracking baseline)
- Added maxFavorablePrice, maxAdversePrice fields
- Added maxFavorableExcursion, maxAdverseExcursion fields

**Integration:**
- Auto-starts on container startup
- Tracks all DATA_COLLECTION_ONLY signals
- Uses same TP/SL calculation as live trades (ATR-based)
- Calculates profit % based on direction (long vs short)

**API Endpoints:**
- GET /api/analytics/signal-tracking - View tracking status and metrics
- POST /api/analytics/signal-tracking - Manually trigger update (auth required)

**Purpose:**
Enables data-driven multi-timeframe comparison. After 50+ signals per
timeframe, can analyze which timeframe (5min vs 15min vs 1H vs 4H vs Daily)
has best win rate, profit potential, and signal quality.

**What It Tracks:**
- Price at 1min, 5min, 15min, 30min after signal
- Would TP1/TP2/SL have been hit?
- Maximum profit/loss during 30min window
- Complete analysis of signal profitability

**How It Works:**
1. Signal comes in (15min, 1H, 4H, Daily) → saved to BlockedSignal
2. Background job runs every 5min
3. Queries current price from Pyth
4. Calculates profit % from entry
5. Checks if TP/SL thresholds crossed
6. Updates MFE/MAE if new highs/lows
7. After 30min, marks analysisComplete=true

**Future Analysis:**
After 50+ signals per timeframe:
- Compare TP1 hit rates across timeframes
- Identify which timeframe has highest win rate
- Determine optimal signal frequency vs quality trade-off
- Switch production to best-performing timeframe

User requested: "i want all the bells and whistles. lets make the
powerhouse more powerfull. i cant see any reason why we shouldnt"
2025-11-19 17:18:47 +01:00
mindesbunister
0700daf8ff feat: add indicator version tracking system
Database changes:
- Added indicatorVersion field to Trade table
- Added indicatorVersion field to BlockedSignal table
- Tracks which Pine Script version (v5, v6, etc.) generated each signal

Pine Script changes:
- v6 now includes '| IND:v6' in alert messages
- Enables differentiation between v5 and v6 signals in database

Documentation:
- Created INDICATOR_VERSION_TRACKING.md with full implementation guide
- Includes n8n workflow update instructions
- Includes SQL analysis queries for v5 vs v6 comparison
- Includes rollback plan if needed

Next steps (manual):
1. Update n8n workflow Parse Signal Enhanced node to extract IND field
2. Update n8n HTTP requests to pass indicatorVersion
3. Update API endpoints to accept and save indicatorVersion
4. Rebuild Docker container

Benefits:
- Compare v5 vs v6 Pine Script effectiveness
- Track which version generated winning/losing trades
- Validate that v6 price position filter reduces blocked signals
- Data-driven decisions on Pine Script improvements
2025-11-11 12:53:33 +01:00
mindesbunister
ba13c20c60 feat: implement blocked signals tracking system
- Add BlockedSignal table with 25 fields for comprehensive signal analysis
- Track all blocked signals with metrics (ATR, ADX, RSI, volume, price position)
- Store quality scores, block reasons, and detailed breakdowns
- Include future fields for automated price analysis (priceAfter1/5/15/30Min)
- Restore signalQualityVersion field to Trade table

Database changes:
- New table: BlockedSignal with indexes on symbol, createdAt, score, blockReason
- Fixed schema drift from manual changes

API changes:
- Modified check-risk endpoint to save blocked signals automatically
- Fixed hasContextMetrics variable scope (moved to line 209)
- Save blocks for: quality score too low, cooldown period, hourly limit
- Use config.minSignalQualityScore instead of hardcoded 60

Database helpers:
- Added createBlockedSignal() function with try/catch safety
- Added getRecentBlockedSignals(limit) for queries
- Added getBlockedSignalsForAnalysis(olderThanMinutes) for automation

Documentation:
- Created BLOCKED_SIGNALS_TRACKING.md with SQL queries and analysis workflow
- Created SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md with 5-phase plan
- Documented data-first approach: collect 10-20 signals before optimization

Rationale:
Only 2 historical trades scored 60-64 (insufficient sample size for threshold decision).
Building data collection infrastructure before making premature optimizations.

Phase 1 (current): Collect blocked signals for 1-2 weeks
Phase 2 (next): Analyze patterns and make data-driven threshold decision
Phase 3-5 (future): Automation and ML optimization
2025-11-11 11:49:21 +01:00
mindesbunister
988fdb9ea4 Fix runner system + strengthen anti-chop filter
Three critical bugs fixed:
1. P&L calculation (65x inflation) - now uses collateralUSD not notional
2. handlePostTp1Adjustments() - checks tp2SizePercent===0 for runner mode
3. JavaScript || operator bug - changed to ?? for proper 0 handling

Signal quality improvements:
- Added anti-chop filter: price position <40% + ADX <25 = -25 points
- Prevents range-bound flip-flops (caught all 3 today)
- Backtest: 43.8% → 55.6% win rate, +86% profit per trade

Changes:
- lib/trading/signal-quality.ts: RANGE-BOUND CHOP penalty
- lib/drift/orders.ts: Fixed P&L calculation + transaction confirmation
- lib/trading/position-manager.ts: Runner system logic
- app/api/trading/execute/route.ts: || to ?? for tp2SizePercent
- app/api/trading/test/route.ts: || to ?? for tp1/tp2SizePercent
- prisma/schema.prisma: Added collateralUSD field
- scripts/fix_pnl_calculations.sql: Historical P&L correction
2025-11-10 15:36:51 +01:00
mindesbunister
9b767342dc feat: Implement re-entry analytics system with fresh TradingView data
- Add market data cache service (5min expiry) for storing TradingView metrics
- Create /api/trading/market-data webhook endpoint for continuous data updates
- Add /api/analytics/reentry-check endpoint for validating manual trades
- Update execute endpoint to auto-cache metrics from incoming signals
- Enhance Telegram bot with pre-execution analytics validation
- Support --force flag to override analytics blocks
- Use fresh ADX/ATR/RSI data when available, fallback to historical
- Apply performance modifiers: -20 for losing streaks, +10 for winning
- Minimum re-entry score 55 (vs 60 for new signals)
- Fail-open design: proceeds if analytics unavailable
- Show data freshness and source in Telegram responses
- Add comprehensive setup guide in docs/guides/REENTRY_ANALYTICS_QUICKSTART.md

Phase 1 implementation for smart manual trade validation.
2025-11-07 20:40:07 +01:00
mindesbunister
625dc44c59 Add signal quality version tracking to database
- Added signalQualityVersion field to Trade model
- Tracks which scoring logic version was used for each trade
- v1: Original logic (price position < 5% threshold)
- v2: Added volume compensation for low ADX
- v3: CURRENT - Stricter logic requiring ADX > 18 for extreme positions (< 15%)

This enables future analysis to:
- Compare performance between logic versions
- Filter trades by scoring algorithm
- Data-driven improvements based on clean datasets

All new trades will be marked as v3. Old trades remain null/v1 for comparison.
2025-11-07 12:56:35 +01:00
mindesbunister
8bc08955cc feat: Add phantom trade detection and database tracking
- Detect position size mismatches (>50% variance) after opening
- Save phantom trades to database with expectedSizeUSD, actualSizeUSD, phantomReason
- Return error from execute endpoint to prevent Position Manager tracking
- Add comprehensive documentation of phantom trade issue and solution
- Enable data collection for pattern analysis and future optimization

Fixes oracle price lag issue during volatile markets where transactions
confirm but positions don't actually open at expected size.
2025-11-04 10:34:38 +01:00
mindesbunister
090b79a07f Store signal quality score in database for future analysis
- Add signalQualityScore field to Trade model (0-100)
- Calculate quality score in execute endpoint using same logic as check-risk
- Save score with every trade for correlation analysis
- Create database migration for new field
- Enables future analysis: score vs win rate, P&L, etc.

This allows data-driven decisions on dynamic position sizing
2025-10-31 11:12:07 +01:00
mindesbunister
830468d524 Implement signal quality scoring system
- Updated execute endpoint to store context metrics in database
- Updated CreateTradeParams interface with 5 context metrics
- Updated Prisma schema with rsiAtEntry and pricePositionAtEntry
- Ran migration: add_rsi_and_price_position_metrics
- Complete flow: TradingView → n8n → check-risk (scores) → execute (stores)
2025-10-30 19:31:32 +01:00
mindesbunister
65e6a8efed Phase 1: Add MAE/MFE tracking and analytics schema
- Added 20+ analytics fields to Trade model (MAE/MFE, fill tracking, timing, market context, slippage)
- Implemented real-time MAE/MFE tracking in Position Manager (updates every 5s)
- Enhanced database schema with comprehensive trade analytics
- Updated all API endpoints to initialize MAE/MFE fields
- Modified updateTradeState() to persist MAE/MFE in configSnapshot

Database changes:
- maxFavorableExcursion/maxAdverseExcursion track best/worst profit %
- maxFavorablePrice/maxAdversePrice track exact price levels
- Fill tracking: tp1Filled, tp2Filled, softSlFilled, hardSlFilled
- Timing metrics: timeToTp1, timeToTp2, timeToSl
- Market context: atrAtEntry, adxAtEntry, volumeAtEntry, fundingRateAtEntry, basisAtEntry
- Slippage tracking: expectedEntryPrice, entrySlippagePct, expectedExitPrice, exitSlippagePct

Position Manager changes:
- Track MAE/MFE on every price check (2s interval)
- Throttled database updates (5s interval) via updateTradeMetrics()
- Persist MAE/MFE in trade state snapshots for recovery

Next: Phase 2 (market context capture) or Phase 3 (analytics API)
2025-10-29 20:34:03 +01:00
mindesbunister
8e5c592cac Fix database persistence and add analytics
- Fixed Prisma client not being available in Docker container
- Added isTestTrade flag to exclude test trades from analytics
- Created analytics views for net positions (matches Drift UI netting)
- Added API endpoints: /api/analytics/positions and /api/analytics/stats
- Added test trade endpoint: /api/trading/test-db
- Updated Dockerfile to properly copy Prisma client from builder stage
- Database now successfully stores all trades with full details
- Supports position netting calculations to match Drift perpetuals behavior
2025-10-27 09:35:01 +01:00
mindesbunister
d64f6d84c4 feat: implement dual stop system and database tracking
- Add PostgreSQL database with Prisma ORM
  - Trade model: tracks entry/exit, P&L, order signatures, config snapshots
  - PriceUpdate model: tracks price movements for drawdown analysis
  - SystemEvent model: logs errors and system events
  - DailyStats model: aggregated performance metrics

- Implement dual stop loss system (enabled by default)
  - Soft stop (TRIGGER_LIMIT) at -1.5% to avoid wicks
  - Hard stop (TRIGGER_MARKET) at -2.5% to guarantee exit
  - Configurable via USE_DUAL_STOPS, SOFT_STOP_PERCENT, HARD_STOP_PERCENT
  - Backward compatible with single stop modes

- Add database service layer (lib/database/trades.ts)
  - createTrade(): save new trades with all details
  - updateTradeExit(): close trades with P&L calculations
  - addPriceUpdate(): track price movements during trade
  - getTradeStats(): calculate win rate, profit factor, avg win/loss
  - logSystemEvent(): log errors and system events

- Update execute endpoint to use dual stops and save to database
  - Calculate dual stop prices when enabled
  - Pass dual stop parameters to placeExitOrders
  - Save complete trade record to database after execution

- Add test trade button to settings page
  - New /api/trading/test endpoint for executing test trades
  - Displays detailed results including dual stop prices
  - Confirmation dialog before execution
  - Shows entry price, position size, stops, and TX signature

- Generate Prisma client in Docker build
- Update DATABASE_URL for container networking
2025-10-26 21:29:27 +01:00