trading_bot_v3/.github/copilot-instructions.md

GitHub Copilot Instructions for Trading Bot Development

🎯 Project Context & Architecture

This is an AI-powered trading automation system with advanced learning capabilities built with Next.js 15 App Router, TypeScript, Tailwind CSS, and integrated with Drift Protocol and Jupiter DEX for automated trading execution.

Core System Components

1. Superior Parallel Screenshot System - 60% faster than sequential capture (71s vs 180s)
2. AI Learning System - Adapts trading decisions based on outcomes with pattern recognition
3. Orphaned Order Cleanup - Automatic cleanup when positions close via position monitor
4. Position Monitoring - Frequent checks with integrated cleanup triggers
5. Dual-Session Screenshot Automation - AI and DIY layouts with session persistence
6. Robust Cleanup System - Prevents Chromium process accumulation

Critical File Relationships

app/api/automation/position-monitor/route.js → Monitors positions + triggers cleanup
lib/simplified-stop-loss-learner.js → AI learning core with pattern recognition
lib/superior-screenshot-service.ts → Parallel screenshot capture system (AVOID in APIs - causes recursion)
lib/enhanced-screenshot.ts → Real screenshot service (USE THIS in API routes)
lib/enhanced-autonomous-risk-manager.js → Risk management with AI integration
lib/enhanced-screenshot-robust.ts → Guaranteed cleanup with finally blocks
lib/automated-cleanup-service.ts → Background process monitoring
app/api/enhanced-screenshot/route.js → CRITICAL: Real screenshot API (fixed from recursive calls)
app/api/ai-analysis/latest/route.js → Real analysis endpoint (depends on enhanced-screenshot)
lib/ai-analysis.ts → AI analysis service (use analyzeScreenshot/analyzeMultipleScreenshots)

<EFBFBD> CRITICAL API INTEGRATION DEBUGGING (Essential Knowledge)

Real vs Mock Data Integration Issues (Critical Learning)

MAJOR ISSUE PATTERN: APIs can appear to work but return fake data due to recursive calls, missing methods, or import failures.

🔥 Enhanced Screenshot API Recursion Problem (Solved)

Root Cause: /api/enhanced-screenshot was calling superiorScreenshotService.captureQuick() which internally called /api/enhanced-screenshot → infinite recursion = 500 errors.

Solution Pattern:

// ❌ WRONG: Causes recursive API calls
import { superiorScreenshotService } from '../../../lib/superior-screenshot-service'
const screenshots = await superiorScreenshotService.captureQuick(symbol, timeframe, layouts)

// ✅ CORRECT: Direct screenshot service usage
const { EnhancedScreenshotService } = await import('../../../lib/enhanced-screenshot')
const service = new EnhancedScreenshotService()
const screenshots = await service.captureWithLogin(config)

🔥 TypeScript/JavaScript Import Issues (Critical)

Problem: Importing .ts files in .js API routes causes "Cannot read properties of undefined" errors.

Solution Pattern:

// ❌ WRONG: Static import of TypeScript in JavaScript
import { EnhancedScreenshotService } from '../../../lib/enhanced-screenshot'

// ✅ CORRECT: Dynamic import for TypeScript modules
const { EnhancedScreenshotService } = await import('../../../lib/enhanced-screenshot')
const service = new EnhancedScreenshotService()

// ✅ CORRECT: Use at call time, not module level

🔥 Progress Tracker Method Issues (Critical)

Problem: Calling non-existent methods crashes API routes silently.

Detection: Error "Cannot read properties of undefined (reading 'length')" often means method doesn't exist.

Solution Pattern:

// ❌ WRONG: Method doesn't exist
sessionId = progressTracker.createSession() // Missing required params
progressTracker.initializeSteps(sessionId, steps) // Method doesn't exist

// ✅ CORRECT: Check actual method signatures
sessionId = `session_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`
const progress = progressTracker.createSession(sessionId, progressSteps)

🔥 AI Analysis Method Issues (Critical)

Problem: Calling wrong method names in AI analysis service.

Solution Pattern:

// ❌ WRONG: Method doesn't exist
analysis = await aiAnalysisService.analyzeScreenshots(config)

// ✅ CORRECT: Use actual methods
if (screenshots.length === 1) {
  analysis = await aiAnalysisService.analyzeScreenshot(screenshots[0])
} else {
  analysis = await aiAnalysisService.analyzeMultipleScreenshots(screenshots)
}

🛠️ Critical Debugging Workflow

Step 1: Verify API Endpoint Functionality

# Test basic API response
curl -X POST http://localhost:9001/api/enhanced-screenshot \
  -H "Content-Type: application/json" \
  -d '{"symbol":"SOLUSD","timeframe":"60","layouts":["ai"],"analyze":false}' \
  --connect-timeout 30 --max-time 60

# Expected: Success response with screenshots array
# Red Flag: 500 error or "Cannot read properties of undefined"

Step 2: Check Container Logs for Import Errors

# Look for import/method errors
docker compose -f docker-compose.dev.yml logs --since="1m" | grep -E "Cannot find module|is not a function|undefined"

# Check for recursive call patterns
docker compose -f docker-compose.dev.yml logs --since="5m" | grep -E "enhanced-screenshot.*enhanced-screenshot"

Step 3: Test Method Existence in Container

# Test TypeScript imports work
docker compose -f docker-compose.dev.yml exec app bash -c \
  "node -e \"import('./lib/enhanced-screenshot').then(m => console.log(Object.keys(m)))\""

# Test method signatures
docker compose -f docker-compose.dev.yml exec app bash -c \
  "node -e \"const pt = require('./lib/progress-tracker'); console.log(Object.getOwnPropertyNames(pt.progressTracker.__proto__))\""

Step 4: Add Debugging to Pinpoint Issues

// Add extensive debugging to API routes
console.log('🔍 API starting...')
console.log('🔍 Config received:', JSON.stringify(config, null, 2))
console.log('🔍 About to call service method...')

// Test each step individually
try {
  const service = new EnhancedScreenshotService()
  console.log('✅ Service instantiated')
  const result = await service.captureWithLogin(config)
  console.log('✅ Method called successfully, result:', typeof result, Array.isArray(result))
} catch (error) {
  console.error('❌ Method call failed:', error.message, error.stack)
}

🎯 Real Data Integration Validation

Verify Real Analysis Working

# Test real analysis endpoint
curl "http://localhost:9001/api/ai-analysis/latest?symbol=SOLUSD&timeframe=60" | jq '.data.analysis.confidence'

# Should return: number between 50-95 (real confidence)
# Red Flag: Always same number, "MOCK" in response, or < 30 second response time

Real Screenshot Integration Checklist

• Screenshots directory exists and contains recent .png files
• Analysis contains specific technical indicators (RSI, MACD, VWAP, OBV)
• Confidence scores vary realistically (60-90%, not always 75%)
• Entry/exit prices are near current market price
• Response time is 30-180 seconds (real analysis takes time)
• Layout analysis mentions "AI Layout" and "DIY Layout"

Mock Data Detection Patterns

// 🚨 RED FLAGS: Patterns that indicate mock data
if (analysis.confidence === 75 && analysis.recommendation === 'HOLD') {
  // Likely mock data - real analysis varies more
}

if (response_time < 5000) {
  // Real analysis takes 30-180 seconds for screenshots + AI
}

if (analysis.reasoning.includes('mock') || analysis.reasoning.includes('demo')) {
  // Obviously mock data
}

if (!analysis.layoutsAnalyzed || analysis.layoutsAnalyzed.length < 2) {
  // Real analysis should have multi-layout comparison
}

🔄 API Integration Dependencies (Critical Chain)

Understanding the Critical Chain:

1. app/automation-v2/page.js GET SIGNAL button calls →
2. app/api/ai-analysis/latest/route.js which calls →
3. app/api/enhanced-screenshot/route.js which uses →
4. lib/enhanced-screenshot.ts EnhancedScreenshotService which calls →
5. lib/ai-analysis.ts aiAnalysisService methods

Failure Points:

• If #3 fails (500 error), #2 throws "Failed to get real screenshot analysis"
• If #2 fails, #1 shows "Error getting latest AI analysis"
• If automation uses #2, automation shows "Waiting for Live Analysis Data"

Testing the Chain:

# Test each link in the chain
curl "http://localhost:9001/api/enhanced-screenshot" -X POST -H "Content-Type: application/json" -d '{"symbol":"SOLUSD","timeframe":"60","analyze":true}'
curl "http://localhost:9001/api/ai-analysis/latest?symbol=SOLUSD&timeframe=60"
# Both should return success with real analysis data

<EFBFBD>🚀 Development Environment (Critical)

Docker Container Development (Required)

All development happens inside Docker containers using Docker Compose v2. Browser automation requires specific system dependencies only available in containerized environment.

IMPORTANT: Use Docker Compose v2 syntax - All commands use docker compose (with space) instead of docker-compose (with hyphen).

# Development environment - Docker Compose v2 dev setup
npm run docker:dev        # Port 9001:3000, hot reload, debug mode
# Direct v2 command: docker compose -f docker-compose.dev.yml up --build

# Production environment  
npm run docker:up         # Port 9000:3000, optimized build
# Direct v2 command: docker compose -f docker-compose.prod.yml up --build

# Debugging commands
npm run docker:logs       # View container logs
npm run docker:exec       # Shell access for debugging inside container

Port Configuration:

• Development: External port 9001 → Internal port 3000 (http://localhost:9001)
• Production: External port 9000 → Internal port 3000 (http://localhost:9000)

Container-First Development Workflow

Common Issue: File edits not reflecting in container due to volume mount sync issues.

Solution - Container Development Workflow:

# 1. Access running container for immediate edits
docker compose -f docker-compose.dev.yml exec app bash

# 2. Edit files directly in container (immediate effect)
nano /app/lib/enhanced-screenshot.ts
echo "console.log('Debug: immediate test');" >> /app/debug.js

# 3. Test changes immediately (no rebuild needed)
# Changes take effect instantly for hot reload

# 4. Once everything works, copy changes back to host
docker cp container_name:/app/modified-file.js ./modified-file.js

# 5. Commit successful changes to git BEFORE rebuilding
git add .
git commit -m "feat: implement working solution for [specific feature]"
git push origin development

# 6. Rebuild container for persistence
docker compose -f docker-compose.dev.yml down
docker compose -f docker-compose.dev.yml up --build -d

# 7. Final validation and commit completion
curl http://localhost:9001  # Verify functionality
git add . && git commit -m "chore: confirm container persistence" && git push

Docker Volume Mount Debugging (Critical Learning)

Problem: Code changes don't reflect in container, or container has different file content than host.

Root Cause: Volume mounts in docker-compose.dev.yml synchronize host directories to container:

volumes:
  - ./app:/app/app:cached        # Host ./app → Container /app/app
  - ./lib:/app/lib:cached        # Host ./lib → Container /app/lib

Debugging Workflow:

# 1. Always check what's actually in the container vs host
docker compose -f docker-compose.dev.yml exec app bash -c "grep -n 'problem_pattern' /app/app/api/file.js"
grep -n 'problem_pattern' app/api/file.js

# 2. Compare file checksums to verify sync
sha256sum app/api/file.js
docker compose -f docker-compose.dev.yml exec app bash -c "sha256sum /app/app/api/file.js"

# 3. Check if Next.js compiled cache is stale
docker compose -f docker-compose.dev.yml exec app bash -c "ls -la /app/.next/server/"
docker compose -f docker-compose.dev.yml exec app bash -c "grep -r 'problem_pattern' /app/.next/server/ || echo 'Not in compiled cache'"

# 4. Clear Next.js cache if files match but behavior doesn't
docker compose -f docker-compose.dev.yml exec app bash -c "rm -rf /app/.next"
docker compose -f docker-compose.dev.yml restart

Key Insights:

• Host edits sync to container automatically via volume mounts
• Container file copies are overwritten by volume mounts on restart
• Next.js compilation cache in .next/ can persist old code even after file changes
• Always verify container content matches expectations before debugging logic
• Compiled webpack bundles may differ from source files - check both

Troubleshooting Steps:

1. Verify host file has expected changes (grep, cat, sed -n '90,100p')
2. Confirm container file matches host (checksums, direct comparison)
3. Check if .next cache is stale (search compiled files for old patterns)
4. Clear compilation cache and restart if source is correct but behavior wrong
5. Use container logs to trace actual execution vs expected code paths

Git Branch Strategy (Required)

Primary development workflow:

• development branch: Use for all active development and feature work
• main branch: Stable, production-ready code only
• Workflow: Develop on development → test thoroughly → commit progress → merge to main when stable

# Standard development workflow with frequent commits
git checkout development        # Always start here
git pull origin development     # Get latest changes

# Make your changes and test in container...

# Commit working progress BEFORE rebuilding container
git add .
git commit -m "feat: [specific achievement] - tested and working"
git push origin development

# After successful container rebuild and validation
git add .
git commit -m "chore: confirm [feature] persistence after rebuild"
git push origin development

# Only merge to main when features are stable and tested
git checkout main
git merge development          # When ready for production
git push origin main

<EFBFBD> System Architecture

Dual-Session Screenshot Automation

• AI Layout: Z1TzpUrf - RSI (top), EMAs, MACD (bottom)
• DIY Layout: vWVvjLhP - Stochastic RSI (top), VWAP, OBV (bottom)
• Parallel browser sessions for multi-layout capture in lib/enhanced-screenshot.ts
• TradingView automation with session persistence in lib/tradingview-automation.ts
• Session data stored in .tradingview-session/ volume mount to avoid captchas

AI Analysis Pipeline

• OpenAI GPT-4o mini for cost-effective chart analysis (~$0.006 per analysis)
• Multi-layout comparison and consensus detection in lib/ai-analysis.ts
• Professional trading setups with exact entry/exit levels and risk management
• Layout-specific indicator analysis (RSI vs Stochastic RSI, MACD vs OBV)

Trading Integration

• Drift Protocol: Perpetual futures trading via @drift-labs/sdk
• Jupiter DEX: Spot trading on Solana
• Position management and P&L tracking in lib/drift-trading-final.ts
• Real-time account balance and collateral monitoring

Browser Process Management & Cleanup System

Critical Issue: Chromium processes accumulate during automated trading, consuming system resources over time.

Robust Cleanup Implementation:

1. Enhanced Screenshot Service (lib/enhanced-screenshot-robust.ts)

   • Guaranteed cleanup via finally blocks in all browser operations
   • Active session tracking to prevent orphaned browsers
   • Session cleanup tasks array for systematic teardown

2. Automated Cleanup Service (lib/automated-cleanup-service.ts)

   • Background monitoring service for orphaned processes
   • Multiple kill strategies: graceful → force → system cleanup
   • Periodic cleanup of temporary files and browser data

3. Aggressive Cleanup Utilities (lib/aggressive-cleanup.ts)

   • System-level process killing for stubborn Chromium processes
   • Port cleanup and temporary directory management
   • Emergency cleanup functions for resource recovery

Implementation Patterns:

// Always use finally blocks for guaranteed cleanup
try {
  const browser = await puppeteer.launch(options);
  // ... browser operations
} finally {
  // Guaranteed cleanup regardless of success/failure
  await ensureBrowserCleanup(browser, sessionId);
  await cleanupSessionTasks(sessionId);
}

// Background monitoring for long-running operations
const cleanupService = new AutomatedCleanupService();
cleanupService.startPeriodicCleanup(); // Every 10 minutes

🚨 Automation Interference Patterns (Critical Learning)

Auto-Restart Loop Detection & Prevention

Problem Pattern: Position monitors with hardcoded "START_TRADING" recommendations create infinite restart loops when no positions are detected, causing rapid order cancellations.

Root Cause Symptoms:

# Log patterns indicating auto-restart loops
docker logs trader_dev | grep "AUTO-RESTART.*START_TRADING"
docker logs trader_dev | grep "No position detected.*recommendation"
docker logs trader_dev | grep "triggering auto-restart"

Detection Commands:

# Check for restart loop patterns
docker logs trader_dev --since="10m" | grep -E "(CYCLE|recommendation|AUTOMATION)" | tail -15

# Monitor order cancellation frequency  
curl -s http://localhost:9001/api/drift/orders | jq '.orders | map(select(.status == "CANCELED")) | length'

# Check position monitor behavior
curl -s http://localhost:9001/api/automation/position-monitor | jq '.monitor.recommendation'

Solution Pattern:

// ❌ WRONG: Hardcoded recommendation causes loops
const result = {
  recommendation: 'START_TRADING', // Always triggers restart
  hasPosition: false // When combined, creates infinite loop
};

// ✅ CORRECT: Context-aware recommendations
const result = {
  recommendation: hasPosition ? 'MONITOR_POSITION' : 'MONITOR_ONLY',
  hasPosition: false // Safe - no auto-restart trigger
};

// ✅ CORRECT: Disable auto-restart entirely for manual control
/* Auto-restart logic disabled to prevent interference with manual trading */

Prevention Checklist:

• Position monitor recommendations are context-aware, not hardcoded
• Auto-restart logic includes manual override capabilities
• Order placement doesn't trigger immediate cleanup cycles
• System allows manual trading without automation interference
• Logs show clean monitoring without constant restart attempts

<EFBFBD>🏗️ System Architecture

Dual-Session Screenshot Automation

• AI Layout: Z1TzpUrf - RSI (top), EMAs, MACD (bottom)
• DIY Layout: vWVvjLhP - Stochastic RSI (top), VWAP, OBV (bottom)
• Parallel browser sessions for multi-layout capture in lib/enhanced-screenshot.ts
• TradingView automation with session persistence in lib/tradingview-automation.ts
• Session data stored in .tradingview-session/ volume mount to avoid captchas

AI Analysis Pipeline

• OpenAI GPT-4o mini for cost-effective chart analysis (~$0.006 per analysis)
• Multi-layout comparison and consensus detection in lib/ai-analysis.ts
• Professional trading setups with exact entry/exit levels and risk management
• Layout-specific indicator analysis (RSI vs Stochastic RSI, MACD vs OBV)

Trading Integration

• Drift Protocol: Perpetual futures trading via @drift-labs/sdk
• Jupiter DEX: Spot trading on Solana
• Position management and P&L tracking in lib/drift-trading-final.ts
• Real-time account balance and collateral monitoring

Browser Process Management & Cleanup System

Critical Issue: Chromium processes accumulate during automated trading, consuming system resources over time.

Robust Cleanup Implementation:

1. Enhanced Screenshot Service (lib/enhanced-screenshot-robust.ts)

   • Guaranteed cleanup via finally blocks in all browser operations
   • Active session tracking to prevent orphaned browsers
   • Session cleanup tasks array for systematic teardown

2. Automated Cleanup Service (lib/automated-cleanup-service.ts)

   • Background monitoring service for orphaned processes
   • Multiple kill strategies: graceful → force → system cleanup
   • Periodic cleanup of temporary files and browser data

3. Aggressive Cleanup Utilities (lib/aggressive-cleanup.ts)

   • System-level process killing for stubborn Chromium processes
   • Port cleanup and temporary directory management
   • Emergency cleanup functions for resource recovery

Implementation Patterns:

// Always use finally blocks for guaranteed cleanup
try {
  const browser = await puppeteer.launch(options);
  // ... browser operations
} finally {
  // Guaranteed cleanup regardless of success/failure
  await ensureBrowserCleanup(browser, sessionId);
  await cleanupSessionTasks(sessionId);
}

// Background monitoring for long-running operations
const cleanupService = new AutomatedCleanupService();
cleanupService.startPeriodicCleanup(); // Every 10 minutes

API Route Structure

All core functionality exposed via Next.js API routes:

// Enhanced screenshot with progress tracking and robust cleanup
POST /api/enhanced-screenshot
{
  symbol: "SOLUSD", 
  timeframe: "240", 
  layouts: ["ai", "diy"],
  analyze: true
}
// Returns: { screenshots, analysis, sessionId }
// Note: Includes automatic Chromium process cleanup via finally blocks

// Drift trading endpoints
GET /api/balance          # Account balance/collateral
POST /api/trading         # Execute trades
GET /api/status          # Trading status
GET /api/automation/position-monitor  # Position monitoring with orphaned cleanup
POST /api/drift/cleanup-orders        # Manual order cleanup

Progress Tracking System

Real-time operation tracking for long-running tasks:

• lib/progress-tracker.ts manages EventEmitter-based progress
• SessionId-based tracking for multi-step operations
• Steps: init → auth → navigation → loading → capture → analysis
• Stream endpoint: /api/progress/[sessionId]/stream

Page Structure & Multi-Timeframe Implementation

• app/analysis/page.js - Original analysis page with multi-timeframe functionality
• app/automation/page.js - Original automation page (legacy, may have issues)
• app/automation-v2/page.js - NEW: Clean automation page with full multi-timeframe support
• app/automation/page-v2.js - Alternative implementation, same functionality as automation-v2

Multi-Timeframe Architecture Pattern:

// Standard timeframes array - use this exact format
const timeframes = ['5m', '15m', '30m', '1h', '2h', '4h', '1d'];

// State management for multi-timeframe selection
const [selectedTimeframes, setSelectedTimeframes] = useState(['1h', '4h']);

// Toggle function with proper array handling
const toggleTimeframe = (tf) => {
  setSelectedTimeframes(prev => 
    prev.includes(tf) 
      ? prev.filter(t => t !== tf)  // Remove if selected
      : [...prev, tf]                // Add if not selected
  );
};

// Preset configurations for trading styles
const presets = {
  scalping: ['5m', '15m', '1h'],
  day: ['1h', '4h', '1d'],
  swing: ['4h', '1d']
};

Component Architecture

• app/layout.js - Root layout with gradient styling and navigation
• components/Navigation.tsx - Multi-page navigation system
• components/AIAnalysisPanel.tsx - Multi-timeframe analysis interface
• components/Dashboard.tsx - Main trading dashboard with real Drift positions
• components/AdvancedTradingPanel.tsx - Drift Protocol trading interface

Critical timeframe handling to avoid TradingView confusion:

// ALWAYS use minute values first, then alternatives
'4h': ['240', '240m', '4h', '4H'] // 240 minutes FIRST
'1h': ['60', '60m', '1h', '1H']   // 60 minutes FIRST
'15m': ['15', '15m']

Layout URL mappings for direct navigation:

const LAYOUT_URLS = {
  'ai': 'Z1TzpUrf',    // RSI + EMAs + MACD
  'diy': 'vWVvjLhP'    // Stochastic RSI + VWAP + OBV
}

🧠 AI Learning System Patterns

async generateLearningReport() {
  // Return comprehensive learning status
  return {
    summary: { totalDecisions, systemConfidence, successRate },
    insights: { thresholds, confidenceLevel },
    recommendations: []
  };
}

async getSmartRecommendation(requestData) {
  // Analyze patterns and return AI recommendation
  const { distanceFromSL, symbol, marketConditions } = requestData;
  // Return: { action, confidence, reasoning }
}

async recordDecision(decisionData) {
  // Log decision for learning with unique ID
  const id = `decision_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
  // Store in database for pattern analysis
}

async assessDecisionOutcome(outcomeData) {
  // Update decision with actual result for learning
  // Calculate if decision was correct based on outcome
}

Database Operations Best Practices:

// ALWAYS provide unique IDs for Prisma records
await prisma.ai_learning_data.create({
  data: {
    id: `${prefix}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,
    // ... other fields
  }
});

// Use correct import path
const { getDB } = require('./db');  // NOT './database-util'

Prisma Table Name Debugging (Critical)

Problem: Database queries fail with "Cannot read properties of undefined (reading 'findMany')" despite correct syntax.

Root Cause: Prisma model names vs database table names can differ, causing silent failures that fall back to alternative APIs (like Binance instead of CoinGecko).

Common Issues:

// ❌ Wrong - will cause undefined errors
await prisma.trade.findMany()              // Should be 'trades' 
await prisma.automationSession.findMany()  // Should be 'automation_sessions'

// ✅ Correct - matches actual database schema
await prisma.trades.findMany()
await prisma.automation_sessions.findMany()

Debugging Steps:

1. Check Prisma schema (prisma/schema.prisma) for actual model names
2. Verify table names in database: PRAGMA table_info(trades);
3. Test queries directly: node -e "const prisma = new PrismaClient(); prisma.trades.findMany().then(console.log);"
4. Look for fallback behavior - API might silently use backup data sources when DB fails
5. Monitor logs for price source errors - DB failures often cause price fetching fallbacks

Impact: Database errors can cause price monitors to fail and fall back to wrong price sources (Binance instead of CoinGecko), affecting trading accuracy.

Always Include These Functions in Learning Classes:

async generateLearningReport() {
  // Return comprehensive learning status
  return {
    summary: { totalDecisions, systemConfidence, successRate },
    insights: { thresholds, confidenceLevel },
    recommendations: []
  };
}

async getSmartRecommendation(requestData) {
  // Analyze patterns and return AI recommendation
  const { distanceFromSL, symbol, marketConditions } = requestData;
  // Return: { action, confidence, reasoning }
}

async recordDecision(decisionData) {
  // Log decision for learning with unique ID
  const id = `decision_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
  // Store in database for pattern analysis
}

async assessDecisionOutcome(outcomeData) {
  // Update decision with actual result for learning
  // Calculate if decision was correct based on outcome
}

Database Operations Best Practices:

// ALWAYS provide unique IDs for Prisma records
await prisma.ai_learning_data.create({
  data: {
    id: `${prefix}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,
    // ... other fields
  }
});

// Use correct import path
const { getDB } = require('./db');  // NOT './database-util'

Prisma Table Name Debugging (Critical)

Problem: Database queries fail with "Cannot read properties of undefined (reading 'findMany')" despite correct syntax.

Root Cause: Prisma model names vs database table names can differ, causing silent failures that fall back to alternative APIs (like Binance instead of CoinGecko).

Common Issues:

// ❌ Wrong - will cause undefined errors
await prisma.trade.findMany()              // Should be 'trades' 
await prisma.automationSession.findMany()  // Should be 'automation_sessions'

// ✅ Correct - matches actual database schema
await prisma.trades.findMany()
await prisma.automation_sessions.findMany()

Debugging Steps:

1. Check Prisma schema (prisma/schema.prisma) for actual model names
2. Verify table names in database: PRAGMA table_info(trades);
3. Test queries directly: node -e "const prisma = new PrismaClient(); prisma.trades.findMany().then(console.log);"
4. Look for fallback behavior - API might silently use backup data sources when DB fails
5. Monitor logs for price source errors - DB failures often cause price fetching fallbacks

Impact: Database errors can cause price monitors to fail and fall back to wrong price sources (Binance instead of CoinGecko), affecting trading accuracy.

Always Include These Functions in Learning Classes:

// ALWAYS provide unique IDs for Prisma records
await prisma.ai_learning_data.create({
  data: {
    id: `${prefix}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,
    // ... other fields
  }
});

// Use correct import path
const { getDB } = require('./db');  // NOT './database-util'

🔧 Error Handling Patterns

🔧 Error Handling Patterns

Function Existence Checks:

// Always check if functions exist before calling
if (typeof this.learner.generateLearningReport === 'function') {
  const report = await this.learner.generateLearningReport();
} else {
  // Fallback to alternative method
  const status = await this.learner.getLearningStatus();
}

Comprehensive Try-Catch:

try {
  const result = await aiFunction();
  return result;
} catch (error) {
  await this.log(`❌ AI function error: ${error.message}`);
  return fallbackResult(); // Always provide fallback
}

📊 Integration Patterns

// Always check if functions exist before calling
if (typeof this.learner.generateLearningReport === 'function') {
  const report = await this.learner.generateLearningReport();
} else {
  // Fallback to alternative method
  const status = await this.learner.getLearningStatus();
}

Comprehensive Try-Catch:

try {
  const result = await aiFunction();
  return result;
} catch (error) {
  await this.log(`❌ AI function error: ${error.message}`);
  return fallbackResult(); // Always provide fallback
}

📊 Integration Patterns

Position Monitor Integration:

// When no position detected, check for orphaned orders
if (!result.hasPosition) {
  console.log('📋 No active positions detected - checking for orphaned orders...');
  
  try {
    const ordersResponse = await fetch(`${baseUrl}/api/drift/orders`);
    if (ordersResponse.ok) {
      const ordersData = await ordersResponse.json();
      if (ordersData.orders?.length > 0) {
        // Trigger cleanup
        const cleanupResponse = await fetch(`${baseUrl}/api/drift/cleanup-orders`, {
          method: 'POST'
        });
        // Handle cleanup result
      }
    }
  } catch (error) {
    // Handle error gracefully
  }
}

Parallel Processing for Screenshots:

// Use Promise.allSettled for parallel processing
const promises = timeframes.map(timeframe => 
  captureTimeframe(timeframe, symbol, layoutType)
);
const results = await Promise.allSettled(promises);

// Process results with error isolation
results.forEach((result, index) => {
  if (result.status === 'fulfilled') {
    // Handle success
  } else {
    // Handle individual failure without breaking others
  }
});

🎯 Performance Optimization Rules

Screenshot Capture:

• Always use parallel processing for multiple timeframes
• Reuse browser sessions to avoid login/captcha
• Isolate errors so one failure doesn't break others
• Prefer Promise.allSettled over Promise.all

Database Queries:

• Use indexed fields for frequent searches (symbol, createdAt)
• Batch operations when possible
• Include proper error handling for connection issues

Container Optimization:

• Check syntax before deployment: node -c filename.js
• Use health checks for monitoring
• Implement graceful shutdown handling

🧪 Testing Requirements

Always Include These Tests:

// Test AI learning functions
const learner = new SimplifiedStopLossLearner();
const report = await learner.generateLearningReport();
console.log('Learning report:', report.summary);

// Test API endpoints
const response = await fetch('/api/automation/position-monitor');
const result = await response.json();
console.log('Position monitor working:', result.success);

// Test error scenarios
try {
  await riskyFunction();
} catch (error) {
  console.log('Error handling working:', error.message);
}

🎨 UI/UX Patterns

Preset Configuration:

// Frontend presets MUST match backend exactly
const TRADING_PRESETS = {
  scalp: ['5m', '15m', '30m'],    // NOT ['5m', '15m', '1h']
  day: ['1h', '2h'],              // NOT ['1h', '4h', '1d']  
  swing: ['4h', '1D'],
  extended: ['1m', '3m', '5m', '15m', '30m', '1h', '4h', '1D']
};

Status Display:

// Always provide detailed feedback
return {
  success: true,
  monitor: {
    hasPosition: false,
    orphanedOrderCleanup: {
      triggered: true,
      success: true,
      message: 'Cleaned up 2 orphaned orders',
      summary: { totalCanceled: 2 }
    }
  }
};

🔍 Debugging Strategies

Container Issues:

# Check for syntax errors
find . -name "*.js" -exec node -c {} \;

# Monitor logs for patterns
docker logs trader_dev --since="1m" | grep -E "(Error|unhandled|crash)"

# Test specific components
node test-learning-system.js

Integration Issues:

# Test API endpoints individually
curl -s http://localhost:9001/api/automation/position-monitor | jq .

# Verify database connectivity
node -e "const {getDB} = require('./lib/db'); getDB().then(() => console.log('DB OK'));"

Automation Loop Debugging:

# Track automation cycles and recommendations
docker logs trader_dev --since="5m" | grep -E "(AUTO-RESTART|recommendation|CYCLE)" | tail -10

# Monitor order behavior patterns
curl -s http://localhost:9001/api/drift/orders | jq '.orders | map(select(.status == "CANCELED")) | length'

# Check if position detection is working
curl -s http://localhost:9001/api/drift/positions | jq '.positions | length'

# Verify cleanup operations
curl -s http://localhost:9001/api/automation/position-monitor | jq '.monitor.orphanedOrderCleanup'

Docker Volume Mount Debugging (Critical Learning)

Problem: Code changes don't reflect in container, or container has different file content than host.

Root Cause: Volume mounts in docker-compose.dev.yml synchronize host directories to container:

volumes:
  - ./app:/app/app:cached        # Host ./app → Container /app/app
  - ./lib:/app/lib:cached        # Host ./lib → Container /app/lib

Debugging Workflow:

# 1. Always check what's actually in the container vs host
docker compose -f docker-compose.dev.yml exec app bash -c "grep -n 'problem_pattern' /app/app/api/file.js"
grep -n 'problem_pattern' app/api/file.js

# 2. Compare file checksums to verify sync
sha256sum app/api/file.js
docker compose -f docker-compose.dev.yml exec app bash -c "sha256sum /app/app/api/file.js"

# 3. Check if Next.js compiled cache is stale
docker compose -f docker-compose.dev.yml exec app bash -c "ls -la /app/.next/server/"
docker compose -f docker-compose.dev.yml exec app bash -c "grep -r 'problem_pattern' /app/.next/server/ || echo 'Not in compiled cache'"

# 4. Clear Next.js cache if files match but behavior doesn't
docker compose -f docker-compose.dev.yml exec app bash -c "rm -rf /app/.next"
docker compose -f docker-compose.dev.yml restart

Key Insights:

• Host edits sync to container automatically via volume mounts
• Container file copies are overwritten by volume mounts on restart
• Next.js compilation cache in .next/ can persist old code even after file changes
• Always verify container content matches expectations before debugging logic
• Compiled webpack bundles may differ from source files - check both

Troubleshooting Steps:

1. Verify host file has expected changes (grep, cat, sed -n '90,100p')
2. Confirm container file matches host (checksums, direct comparison)
3. Check if .next cache is stale (search compiled files for old patterns)
4. Clear compilation cache and restart if source is correct but behavior wrong
5. Use container logs to trace actual execution vs expected code paths

<EFBFBD> Automation Interference Patterns (Critical Learning)

Auto-Restart Loop Detection & Prevention

Problem Pattern: Position monitors with hardcoded "START_TRADING" recommendations create infinite restart loops when no positions are detected, causing rapid order cancellations.

Root Cause Symptoms:

# Log patterns indicating auto-restart loops
docker logs trader_dev | grep "AUTO-RESTART.*START_TRADING"
docker logs trader_dev | grep "No position detected.*recommendation"
docker logs trader_dev | grep "triggering auto-restart"

Detection Commands:

# Check for restart loop patterns
docker logs trader_dev --since="10m" | grep -E "(CYCLE|recommendation|AUTOMATION)" | tail -15

# Monitor order cancellation frequency  
curl -s http://localhost:9001/api/drift/orders | jq '.orders | map(select(.status == "CANCELED")) | length'

# Check position monitor behavior
curl -s http://localhost:9001/api/automation/position-monitor | jq '.monitor.recommendation'

Solution Pattern:

// ❌ WRONG: Hardcoded recommendation causes loops
const result = {
  recommendation: 'START_TRADING', // Always triggers restart
  hasPosition: false // When combined, creates infinite loop
};

// ✅ CORRECT: Context-aware recommendations
const result = {
  recommendation: hasPosition ? 'MONITOR_POSITION' : 'MONITOR_ONLY',
  hasPosition: false // Safe - no auto-restart trigger
};

// ✅ CORRECT: Disable auto-restart entirely for manual control
/* Auto-restart logic disabled to prevent interference with manual trading */

Prevention Checklist:

• Position monitor recommendations are context-aware, not hardcoded
• Auto-restart logic includes manual override capabilities
• Order placement doesn't trigger immediate cleanup cycles
• System allows manual trading without automation interference
• Logs show clean monitoring without constant restart attempts

<EFBFBD>🚨 Critical Anti-Patterns to Avoid

❌ Don't Do This:

// Missing error handling
const report = await this.learner.generateLearningReport(); // Will crash if function missing

// Redundant polling
setInterval(checkOrders, 60000); // When position monitor already runs frequently

// Auto-restart loops that interfere with trading
recommendation: 'START_TRADING', // Hardcoded - causes constant restart triggers
if (!hasPosition && recommendation === 'START_TRADING') {
  // Auto-restart logic that triggers rapid cleanup cycles
}

// Frontend/backend preset mismatch  
backend: ['5m', '15m', '1h']
frontend: ['5m', '15m', '30m'] // Will cause confusion

// Missing unique IDs
await prisma.create({ data: { symbol, timeframe } }); // Will fail validation

✅ Do This Instead:

// Defensive programming
if (typeof this.learner.generateLearningReport === 'function') {
  try {
    const report = await this.learner.generateLearningReport();
  } catch (error) {
    await this.log(`Report generation failed: ${error.message}`);
  }
}

// Leverage existing infrastructure
// Add cleanup to existing position monitor instead of new polling

// Smart recommendations that don't trigger loops
recommendation: hasPosition ? 'MONITOR_POSITION' : 'MONITOR_ONLY', // Context-aware
// Disable auto-restart for manual control
/* Auto-restart logic disabled to prevent interference */

// Ensure consistency
const PRESETS = { scalp: ['5m', '15m', '30m'] }; // Same in frontend and backend

// Always provide unique IDs
const id = `${type}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;

🚨 CRITICAL ANTI-PATTERNS TO AVOID

❌ Don't Do This:

// Missing error handling
const report = await this.learner.generateLearningReport(); // Will crash if function missing

// Redundant polling
setInterval(checkOrders, 60000); // When position monitor already runs frequently

// Auto-restart loops that interfere with trading
recommendation: 'START_TRADING', // Hardcoded - causes constant restart triggers
if (!hasPosition && recommendation === 'START_TRADING') {
  // Auto-restart logic that triggers rapid cleanup cycles
}

// Frontend/backend preset mismatch  
backend: ['5m', '15m', '1h']
frontend: ['5m', '15m', '30m'] // Will cause confusion

// Missing unique IDs
await prisma.create({ data: { symbol, timeframe } }); // Will fail validation

// Recursive API calls (CRITICAL)
// In /api/enhanced-screenshot calling superiorScreenshotService which calls /api/enhanced-screenshot
import { superiorScreenshotService } from '../../../lib/superior-screenshot-service'
const screenshots = await superiorScreenshotService.captureQuick() // CAUSES INFINITE RECURSION

// TypeScript imports in JavaScript API routes
import { EnhancedScreenshotService } from '../../../lib/enhanced-screenshot' // FAILS SILENTLY

// Wrong AI analysis method names
analysis = await aiAnalysisService.analyzeScreenshots(config) // METHOD DOESN'T EXIST

// Wrong progress tracker usage
sessionId = progressTracker.createSession() // MISSING REQUIRED PARAMETERS
progressTracker.initializeSteps(sessionId, steps) // METHOD DOESN'T EXIST

✅ Do This Instead:

// Defensive programming
if (typeof this.learner.generateLearningReport === 'function') {
  try {
    const report = await this.learner.generateLearningReport();
  } catch (error) {
    await this.log(`Report generation failed: ${error.message}`);
  }
}

// Leverage existing infrastructure
// Add cleanup to existing position monitor instead of new polling

// Smart recommendations that don't trigger loops
recommendation: hasPosition ? 'MONITOR_POSITION' : 'MONITOR_ONLY', // Context-aware
// Disable auto-restart for manual control
/* Auto-restart logic disabled to prevent interference */

// Ensure consistency
const PRESETS = { scalp: ['5m', '15m', '30m'] }; // Same in frontend and backend

// Always provide unique IDs
const id = `${type}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;

// Correct service usage (NO RECURSION)
const { EnhancedScreenshotService } = await import('../../../lib/enhanced-screenshot')
const service = new EnhancedScreenshotService()
const screenshots = await service.captureWithLogin(config)

// Dynamic imports for TypeScript in JavaScript
const { EnhancedScreenshotService } = await import('../../../lib/enhanced-screenshot')

// Correct AI analysis method names
if (screenshots.length === 1) {
  analysis = await aiAnalysisService.analyzeScreenshot(screenshots[0])
} else {
  analysis = await aiAnalysisService.analyzeMultipleScreenshots(screenshots)
}

// Correct progress tracker usage
sessionId = `session_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`
const progress = progressTracker.createSession(sessionId, progressSteps)

🎯 Configuration Standards

Environment Variables:

// Always provide fallbacks
const apiKey = process.env.OPENAI_API_KEY || '';
if (!apiKey) {
  throw new Error('OPENAI_API_KEY is required');
}

Next.js Configuration:

// Use new format, not deprecated
const nextConfig: NextConfig = {
  serverExternalPackages: ['puppeteer-core'], // NOT experimental.serverComponentsExternalPackages
  transpilePackages: ['next-font'],
  typescript: { ignoreBuildErrors: true },
  eslint: { ignoreDuringBuilds: true }
};

📈 Enhancement Guidelines

When adding new features:

1. Check Existing Infrastructure - Can it be integrated vs creating new?
2. Add Comprehensive Error Handling - Assume functions may not exist
3. Include Fallback Mechanisms - System should work without AI/learning
4. Test in Isolation - Create test scripts for new components
5. Document Integration Points - How does it connect to existing systems?
6. Maintain Consistency - Frontend and backend must match exactly
7. Use Defensive Programming - Check before calling, handle gracefully
8. Avoid Recursive API Calls - Never call same endpoint from service layer
9. Use Dynamic Imports - For TypeScript modules in JavaScript API routes
10. Verify Method Existence - Check actual method signatures before calling
11. Test API Chain Dependencies - Ensure entire call chain works end-to-end
12. Validate Real vs Mock Data - Confirm actual analysis data, not fallback responses

🔧 Critical Debugging Workflow for New Features:

# 1. Test API endpoint directly
curl -X POST http://localhost:9001/api/your-endpoint \
  -H "Content-Type: application/json" \
  -d '{"test":"data"}' --max-time 30

# 2. Check container logs for errors
docker compose -f docker-compose.dev.yml logs --since="1m" | grep -E "ERROR|undefined|is not a function"

# 3. Verify method existence in container
docker compose -f docker-compose.dev.yml exec app bash -c \
  "node -e \"import('./lib/your-service').then(m => console.log(Object.keys(m)))\""

# 4. Test complete integration chain
curl "http://localhost:9001/api/ai-analysis/latest?symbol=SOLUSD&timeframe=60"

🚨 Pre-Deployment Checklist:

• API endpoints return real data, not mock/fallback responses
• No recursive API calls in service layers
• TypeScript imports use dynamic imports in JavaScript files
• All method calls verified to exist with correct signatures
• Error handling includes function existence checks
• Progress tracking uses correct method signatures
• Database queries use correct table/model names
• Container logs show no import/method errors
• End-to-end API chain tested and working
• Real analysis data verified (proper confidence scores, realistic timing)

📚 Documentation References

Technical Documentation

• ADVANCED_SYSTEM_KNOWLEDGE.md - Deep technical architecture, session management, cleanup systems
• README.md - Main project overview with current feature status and setup
• AI_LEARNING_EXPLAINED.md - AI learning system implementation details
• DRIFT_FEEDBACK_LOOP_COMPLETE.md - Complete Drift trading integration
• ROBUST_CLEANUP_IMPLEMENTATION.md - Browser process cleanup system details

Implementation Guides

• MULTI_LAYOUT_IMPLEMENTATION.md - Dual-session screenshot system
• SESSION_PERSISTENCE.md - TradingView session management
• DOCKER_AUTOMATION.md - Container development workflow
• DEVELOPMENT_GUIDE.md - Complete development setup instructions

Analysis & Troubleshooting

• MULTI_LAYOUT_TROUBLESHOOTING.md - Screenshot automation debugging
• CLEANUP_IMPROVEMENTS.md - Process management enhancements
• SCREENSHOT_PATH_FIXES.md - Screenshot capture issue resolution

🎓 CRITICAL LESSONS LEARNED (Session: July 30, 2025)

🚨 Major Bug Pattern Discovered: API Recursion Loops

Problem: Enhanced Screenshot API was calling itself infinitely through service layer Root Cause: superiorScreenshotService.captureQuick() internally called /api/enhanced-screenshot Solution: Switch to direct EnhancedScreenshotService with captureWithLogin() Detection: 500 errors, container logs showing recursive call patterns Prevention: Never import services that call the same API endpoint

🔧 TypeScript/JavaScript Import Issues in API Routes

Problem: Static imports of .ts files in .js API routes fail silently Symptoms: "Cannot read properties of undefined" errors, missing method errors Solution: Use dynamic imports: const { Service } = await import('./service') Critical: Always test imports in container before deployment

📊 Progress Tracker Method Signature Errors

Problem: Calling progressTracker.createSession() without parameters Error: "Cannot read properties of undefined (reading 'length')" Solution: Always provide sessionId and steps: createSession(sessionId, progressSteps) Learning: Check actual method signatures, don't assume parameter patterns

🤖 AI Analysis Service Method Name Confusion

Problem: aiAnalysisService.analyzeScreenshots() doesn't exist Correct Methods: analyzeScreenshot() (single) and analyzeMultipleScreenshots() (array) Pattern: Always verify method names in TypeScript service files Debugging: Use container exec to test imports and list available methods

🔄 Real vs Mock Data Integration Validation

Critical Insight: APIs can appear to work but return fake data due to failed imports Validation Techniques:

• Response timing: Real analysis takes 30-180 seconds, mock returns in <5 seconds
• Confidence variance: Real analysis varies 60-90%, mock often fixed at 75%
• Technical detail depth: Real includes specific indicators, entry/exit levels
• Layout analysis: Real mentions "AI Layout" and "DIY Layout" comparison

🐛 Container Development Debugging Workflow

Key Learning: Always verify what's actually running in the container Essential Commands:

# Check file sync between host and container
sha256sum app/api/file.js
docker compose exec app bash -c "sha256sum /app/app/api/file.js"

# Clear Next.js compilation cache when behavior doesn't match source
docker compose exec app bash -c "rm -rf /app/.next"

# Test imports directly in container
docker compose exec app bash -c "node -e \"import('./lib/service').then(console.log)\""

🎯 API Integration Chain Dependencies

Critical Understanding: Failure in one link breaks entire automation Chain: GET SIGNAL → ai-analysis/latest → enhanced-screenshot → EnhancedScreenshotService → AI Analysis Testing Strategy: Test each link individually, then validate end-to-end Monitoring: Watch for "Failed to get real screenshot analysis" errors

---

These patterns represent the most common and critical issues encountered during real data integration. Understanding these will prevent weeks of debugging in future development.

---

Follow these patterns to maintain system stability and avoid the complex debugging issues that were resolved in this session.