trading_bot_v3/.github/copilot-instructions.md

# GitHub Copilot Instructions for Trading Bot Development

## 🎯 Project Context & Architecture

This is an AI-powered trading automation system with advanced learning capabilities built with Next.js 15 App Router, TypeScript, Tailwind CSS, and integrated with Drift Protocol and Jupiter DEX for automated trading execution.

### Core System Components
1. **Superior Parallel Screenshot System** - 60% faster than sequential capture (71s vs 180s)
2. **AI Learning System** - Adapts trading decisions based on outcomes with pattern recognition  
3. **Orphaned Order Cleanup** - Automatic cleanup when positions close via position monitor
4. **Position Monitoring** - Frequent checks with integrated cleanup triggers
5. **Dual-Session Screenshot Automation** - AI and DIY layouts with session persistence
6. **Robust Cleanup System** - Prevents Chromium process accumulation

### Critical File Relationships
```
app/api/automation/position-monitor/route.js → Monitors positions + triggers cleanup
lib/simplified-stop-loss-learner.js → AI learning core with pattern recognition
lib/superior-screenshot-service.ts → Parallel screenshot capture system (AVOID in APIs - causes recursion)
lib/enhanced-screenshot.ts → Real screenshot service (USE THIS in API routes)
lib/enhanced-autonomous-risk-manager.js → Risk management with AI integration
lib/enhanced-screenshot-robust.ts → Guaranteed cleanup with finally blocks
lib/automated-cleanup-service.ts → Background process monitoring
app/api/enhanced-screenshot/route.js → CRITICAL: Real screenshot API (fixed from recursive calls)
app/api/ai-analysis/latest/route.js → Real analysis endpoint (depends on enhanced-screenshot)
lib/ai-analysis.ts → AI analysis service (use analyzeScreenshot/analyzeMultipleScreenshots)
```

## <20> CRITICAL API INTEGRATION DEBUGGING (Essential Knowledge)

### Real vs Mock Data Integration Issues (Critical Learning)

**MAJOR ISSUE PATTERN**: APIs can appear to work but return fake data due to recursive calls, missing methods, or import failures.

#### 🔥 Enhanced Screenshot API Recursion Problem (Solved)
**Root Cause**: `/api/enhanced-screenshot` was calling `superiorScreenshotService.captureQuick()` which internally called `/api/enhanced-screenshot` → infinite recursion = 500 errors.

**Solution Pattern**:
```javascript
// ❌ WRONG: Causes recursive API calls
import { superiorScreenshotService } from '../../../lib/superior-screenshot-service'
const screenshots = await superiorScreenshotService.captureQuick(symbol, timeframe, layouts)

// ✅ CORRECT: Direct screenshot service usage
const { EnhancedScreenshotService } = await import('../../../lib/enhanced-screenshot')
const service = new EnhancedScreenshotService()
const screenshots = await service.captureWithLogin(config)
```

#### 🔥 TypeScript/JavaScript Import Issues (Critical)
**Problem**: Importing `.ts` files in `.js` API routes causes "Cannot read properties of undefined" errors.

**Solution Pattern**:
```javascript
// ❌ WRONG: Static import of TypeScript in JavaScript
import { EnhancedScreenshotService } from '../../../lib/enhanced-screenshot'

// ✅ CORRECT: Dynamic import for TypeScript modules
const { EnhancedScreenshotService } = await import('../../../lib/enhanced-screenshot')
const service = new EnhancedScreenshotService()

// ✅ CORRECT: Use at call time, not module level
```

#### 🔥 Progress Tracker Method Issues (Critical)
**Problem**: Calling non-existent methods crashes API routes silently.

**Detection**: Error "Cannot read properties of undefined (reading 'length')" often means method doesn't exist.

**Solution Pattern**:
```javascript
// ❌ WRONG: Method doesn't exist
sessionId = progressTracker.createSession() // Missing required params
progressTracker.initializeSteps(sessionId, steps) // Method doesn't exist

// ✅ CORRECT: Check actual method signatures
sessionId = `session_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`
const progress = progressTracker.createSession(sessionId, progressSteps)
```

#### 🔥 AI Analysis Method Issues (Critical)
**Problem**: Calling wrong method names in AI analysis service.

**Solution Pattern**:
```javascript
// ❌ WRONG: Method doesn't exist
analysis = await aiAnalysisService.analyzeScreenshots(config)

// ✅ CORRECT: Use actual methods
if (screenshots.length === 1) {
  analysis = await aiAnalysisService.analyzeScreenshot(screenshots[0])
} else {
  analysis = await aiAnalysisService.analyzeMultipleScreenshots(screenshots)
}
```

### 🛠️ Critical Debugging Workflow

#### Step 1: Verify API Endpoint Functionality
```bash
# Test basic API response
curl -X POST http://localhost:9001/api/enhanced-screenshot \
  -H "Content-Type: application/json" \
  -d '{"symbol":"SOLUSD","timeframe":"60","layouts":["ai"],"analyze":false}' \
  --connect-timeout 30 --max-time 60

# Expected: Success response with screenshots array
# Red Flag: 500 error or "Cannot read properties of undefined"
```

#### Step 2: Check Container Logs for Import Errors
```bash
# Look for import/method errors
docker compose -f docker-compose.dev.yml logs --since="1m" | grep -E "Cannot find module|is not a function|undefined"

# Check for recursive call patterns
docker compose -f docker-compose.dev.yml logs --since="5m" | grep -E "enhanced-screenshot.*enhanced-screenshot"
```

#### Step 3: Test Method Existence in Container
```bash
# Test TypeScript imports work
docker compose -f docker-compose.dev.yml exec app bash -c \
  "node -e \"import('./lib/enhanced-screenshot').then(m => console.log(Object.keys(m)))\""

# Test method signatures
docker compose -f docker-compose.dev.yml exec app bash -c \
  "node -e \"const pt = require('./lib/progress-tracker'); console.log(Object.getOwnPropertyNames(pt.progressTracker.__proto__))\""
```

#### Step 4: Add Debugging to Pinpoint Issues
```javascript
// Add extensive debugging to API routes
console.log('🔍 API starting...')
console.log('🔍 Config received:', JSON.stringify(config, null, 2))
console.log('🔍 About to call service method...')

// Test each step individually
try {
  const service = new EnhancedScreenshotService()
  console.log('✅ Service instantiated')
  const result = await service.captureWithLogin(config)
  console.log('✅ Method called successfully, result:', typeof result, Array.isArray(result))
} catch (error) {
  console.error('❌ Method call failed:', error.message, error.stack)
}
```

### 🎯 Real Data Integration Validation

#### Verify Real Analysis Working
```bash
# Test real analysis endpoint
curl "http://localhost:9001/api/ai-analysis/latest?symbol=SOLUSD&timeframe=60" | jq '.data.analysis.confidence'

# Should return: number between 50-95 (real confidence)
# Red Flag: Always same number, "MOCK" in response, or < 30 second response time
```

#### Real Screenshot Integration Checklist
- [ ] Screenshots directory exists and contains recent `.png` files
- [ ] Analysis contains specific technical indicators (RSI, MACD, VWAP, OBV)
- [ ] Confidence scores vary realistically (60-90%, not always 75%)
- [ ] Entry/exit prices are near current market price
- [ ] Response time is 30-180 seconds (real analysis takes time)
- [ ] Layout analysis mentions "AI Layout" and "DIY Layout"

#### Mock Data Detection Patterns
```javascript
// 🚨 RED FLAGS: Patterns that indicate mock data
if (analysis.confidence === 75 && analysis.recommendation === 'HOLD') {
  // Likely mock data - real analysis varies more
}

if (response_time < 5000) {
  // Real analysis takes 30-180 seconds for screenshots + AI
}

if (analysis.reasoning.includes('mock') || analysis.reasoning.includes('demo')) {
  // Obviously mock data
}

if (!analysis.layoutsAnalyzed || analysis.layoutsAnalyzed.length < 2) {
  // Real analysis should have multi-layout comparison
}
```

### 🔄 API Integration Dependencies (Critical Chain)

**Understanding the Critical Chain**:
1. `app/automation-v2/page.js` GET SIGNAL button calls →
2. `app/api/ai-analysis/latest/route.js` which calls →
3. `app/api/enhanced-screenshot/route.js` which uses →
4. `lib/enhanced-screenshot.ts` EnhancedScreenshotService which calls →
5. `lib/ai-analysis.ts` aiAnalysisService methods

**Failure Points**:
- If #3 fails (500 error), #2 throws "Failed to get real screenshot analysis"
- If #2 fails, #1 shows "Error getting latest AI analysis"
- If automation uses #2, automation shows "Waiting for Live Analysis Data"

**Testing the Chain**:
```bash
# Test each link in the chain
curl "http://localhost:9001/api/enhanced-screenshot" -X POST -H "Content-Type: application/json" -d '{"symbol":"SOLUSD","timeframe":"60","analyze":true}'
curl "http://localhost:9001/api/ai-analysis/latest?symbol=SOLUSD&timeframe=60"
# Both should return success with real analysis data
```

## <20>🚀 Development Environment (Critical)

### Docker Container Development (Required)
**All development happens inside Docker containers** using Docker Compose v2. Browser automation requires specific system dependencies only available in containerized environment.

**IMPORTANT: Use Docker Compose v2 syntax** - All commands use `docker compose` (with space) instead of `docker-compose` (with hyphen).

```bash
# Development environment - Docker Compose v2 dev setup
npm run docker:dev        # Port 9001:3000, hot reload, debug mode
# Direct v2 command: docker compose -f docker-compose.dev.yml up --build

# Production environment  
npm run docker:up         # Port 9000:3000, optimized build
# Direct v2 command: docker compose -f docker-compose.prod.yml up --build

# Debugging commands
npm run docker:logs       # View container logs
npm run docker:exec       # Shell access for debugging inside container
```

**Port Configuration:**
- **Development**: External port `9001` → Internal port `3000` (http://localhost:9001)
- **Production**: External port `9000` → Internal port `3000` (http://localhost:9000)

### Container-First Development Workflow
**Common Issue**: File edits not reflecting in container due to volume mount sync issues.

**Solution - Container Development Workflow:**
```bash
# 1. Access running container for immediate edits
docker compose -f docker-compose.dev.yml exec app bash

# 2. Edit files directly in container (immediate effect)
nano /app/lib/enhanced-screenshot.ts
echo "console.log('Debug: immediate test');" >> /app/debug.js

# 3. Test changes immediately (no rebuild needed)
# Changes take effect instantly for hot reload

# 4. Once everything works, copy changes back to host
docker cp container_name:/app/modified-file.js ./modified-file.js

# 5. Commit successful changes to git BEFORE rebuilding
git add .
git commit -m "feat: implement working solution for [specific feature]"
git push origin development

# 6. Rebuild container for persistence
docker compose -f docker-compose.dev.yml down
docker compose -f docker-compose.dev.yml up --build -d

# 7. Final validation and commit completion
curl http://localhost:9001  # Verify functionality
git add . && git commit -m "chore: confirm container persistence" && git push
```

### Docker Volume Mount Debugging (Critical Learning)
**Problem**: Code changes don't reflect in container, or container has different file content than host.

**Root Cause**: Volume mounts in `docker-compose.dev.yml` synchronize host directories to container:
```yaml
volumes:
  - ./app:/app/app:cached        # Host ./app → Container /app/app
  - ./lib:/app/lib:cached        # Host ./lib → Container /app/lib
```

**Debugging Workflow**:
```bash
# 1. Always check what's actually in the container vs host
docker compose -f docker-compose.dev.yml exec app bash -c "grep -n 'problem_pattern' /app/app/api/file.js"
grep -n 'problem_pattern' app/api/file.js

# 2. Compare file checksums to verify sync
sha256sum app/api/file.js
docker compose -f docker-compose.dev.yml exec app bash -c "sha256sum /app/app/api/file.js"

# 3. Check if Next.js compiled cache is stale
docker compose -f docker-compose.dev.yml exec app bash -c "ls -la /app/.next/server/"
docker compose -f docker-compose.dev.yml exec app bash -c "grep -r 'problem_pattern' /app/.next/server/ || echo 'Not in compiled cache'"

# 4. Clear Next.js cache if files match but behavior doesn't
docker compose -f docker-compose.dev.yml exec app bash -c "rm -rf /app/.next"
docker compose -f docker-compose.dev.yml restart
```

**Key Insights**:
- **Host edits sync to container** automatically via volume mounts
- **Container file copies are overwritten** by volume mounts on restart
- **Next.js compilation cache** in `.next/` can persist old code even after file changes
- **Always verify container content** matches expectations before debugging logic
- **Compiled webpack bundles** may differ from source files - check both

**Troubleshooting Steps**:
1. **Verify host file has expected changes** (`grep`, `cat`, `sed -n '90,100p'`)
2. **Confirm container file matches host** (checksums, direct comparison)
3. **Check if .next cache is stale** (search compiled files for old patterns)
4. **Clear compilation cache and restart** if source is correct but behavior wrong
5. **Use container logs to trace actual execution** vs expected code paths

### Git Branch Strategy (Required)
**Primary development workflow:**
- **`development` branch**: Use for all active development and feature work
- **`main` branch**: Stable, production-ready code only
- **Workflow**: Develop on `development` → test thoroughly → commit progress → merge to `main` when stable

```bash
# Standard development workflow with frequent commits
git checkout development        # Always start here
git pull origin development     # Get latest changes

# Make your changes and test in container...

# Commit working progress BEFORE rebuilding container
git add .
git commit -m "feat: [specific achievement] - tested and working"
git push origin development

# After successful container rebuild and validation
git add .
git commit -m "chore: confirm [feature] persistence after rebuild"
git push origin development

# Only merge to main when features are stable and tested
git checkout main
git merge development          # When ready for production
git push origin main
```

## <20> System Architecture

### Dual-Session Screenshot Automation
- **AI Layout**: `Z1TzpUrf` - RSI (top), EMAs, MACD (bottom)
- **DIY Layout**: `vWVvjLhP` - Stochastic RSI (top), VWAP, OBV (bottom)
- Parallel browser sessions for multi-layout capture in `lib/enhanced-screenshot.ts`
- TradingView automation with session persistence in `lib/tradingview-automation.ts`
- Session data stored in `.tradingview-session/` volume mount to avoid captchas

### AI Analysis Pipeline  
- OpenAI GPT-4o mini for cost-effective chart analysis (~$0.006 per analysis)
- Multi-layout comparison and consensus detection in `lib/ai-analysis.ts`
- Professional trading setups with exact entry/exit levels and risk management
- Layout-specific indicator analysis (RSI vs Stochastic RSI, MACD vs OBV)

### Trading Integration
- **Drift Protocol**: Perpetual futures trading via `@drift-labs/sdk`
- **Jupiter DEX**: Spot trading on Solana
- Position management and P&L tracking in `lib/drift-trading-final.ts`
- Real-time account balance and collateral monitoring

### Browser Process Management & Cleanup System
**Critical Issue**: Chromium processes accumulate during automated trading, consuming system resources over time.

**Robust Cleanup Implementation:**
1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`)
   - Guaranteed cleanup via `finally` blocks in all browser operations
   - Active session tracking to prevent orphaned browsers
   - Session cleanup tasks array for systematic teardown

2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`)
   - Background monitoring service for orphaned processes
   - Multiple kill strategies: graceful → force → system cleanup
   - Periodic cleanup of temporary files and browser data

3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`)
   - System-level process killing for stubborn Chromium processes
   - Port cleanup and temporary directory management
   - Emergency cleanup functions for resource recovery

**Implementation Patterns:**
```typescript
// Always use finally blocks for guaranteed cleanup
try {
  const browser = await puppeteer.launch(options);
  // ... browser operations
} finally {
  // Guaranteed cleanup regardless of success/failure
  await ensureBrowserCleanup(browser, sessionId);
  await cleanupSessionTasks(sessionId);
}

// Background monitoring for long-running operations
const cleanupService = new AutomatedCleanupService();
cleanupService.startPeriodicCleanup(); // Every 10 minutes
```

## 🚨 Automation Interference Patterns (Critical Learning)

### Auto-Restart Loop Detection & Prevention
**Problem Pattern**: Position monitors with hardcoded "START_TRADING" recommendations create infinite restart loops when no positions are detected, causing rapid order cancellations.

**Root Cause Symptoms**:
```bash
# Log patterns indicating auto-restart loops
docker logs trader_dev | grep "AUTO-RESTART.*START_TRADING"
docker logs trader_dev | grep "No position detected.*recommendation"
docker logs trader_dev | grep "triggering auto-restart"
```

**Detection Commands**:
```bash
# Check for restart loop patterns
docker logs trader_dev --since="10m" | grep -E "(CYCLE|recommendation|AUTOMATION)" | tail -15

# Monitor order cancellation frequency  
curl -s http://localhost:9001/api/drift/orders | jq '.orders | map(select(.status == "CANCELED")) | length'

# Check position monitor behavior
curl -s http://localhost:9001/api/automation/position-monitor | jq '.monitor.recommendation'
```

**Solution Pattern**:
```javascript
// ❌ WRONG: Hardcoded recommendation causes loops
const result = {
  recommendation: 'START_TRADING', // Always triggers restart
  hasPosition: false // When combined, creates infinite loop
};

// ✅ CORRECT: Context-aware recommendations
const result = {
  recommendation: hasPosition ? 'MONITOR_POSITION' : 'MONITOR_ONLY',
  hasPosition: false // Safe - no auto-restart trigger
};

// ✅ CORRECT: Disable auto-restart entirely for manual control
/* Auto-restart logic disabled to prevent interference with manual trading */
```

**Prevention Checklist**:
- [ ] Position monitor recommendations are context-aware, not hardcoded
- [ ] Auto-restart logic includes manual override capabilities  
- [ ] Order placement doesn't trigger immediate cleanup cycles
- [ ] System allows manual trading without automation interference
- [ ] Logs show clean monitoring without constant restart attempts

## <20>🏗️ System Architecture

### Dual-Session Screenshot Automation
- **AI Layout**: `Z1TzpUrf` - RSI (top), EMAs, MACD (bottom)
- **DIY Layout**: `vWVvjLhP` - Stochastic RSI (top), VWAP, OBV (bottom)
- Parallel browser sessions for multi-layout capture in `lib/enhanced-screenshot.ts`
- TradingView automation with session persistence in `lib/tradingview-automation.ts`
- Session data stored in `.tradingview-session/` volume mount to avoid captchas

### AI Analysis Pipeline  
- OpenAI GPT-4o mini for cost-effective chart analysis (~$0.006 per analysis)
- Multi-layout comparison and consensus detection in `lib/ai-analysis.ts`
- Professional trading setups with exact entry/exit levels and risk management
- Layout-specific indicator analysis (RSI vs Stochastic RSI, MACD vs OBV)

### Trading Integration
- **Drift Protocol**: Perpetual futures trading via `@drift-labs/sdk`
- **Jupiter DEX**: Spot trading on Solana
- Position management and P&L tracking in `lib/drift-trading-final.ts`
- Real-time account balance and collateral monitoring

### Browser Process Management & Cleanup System
**Critical Issue**: Chromium processes accumulate during automated trading, consuming system resources over time.

**Robust Cleanup Implementation:**
1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`)
   - Guaranteed cleanup via `finally` blocks in all browser operations
   - Active session tracking to prevent orphaned browsers
   - Session cleanup tasks array for systematic teardown

2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`)
   - Background monitoring service for orphaned processes
   - Multiple kill strategies: graceful → force → system cleanup
   - Periodic cleanup of temporary files and browser data

3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`)
   - System-level process killing for stubborn Chromium processes
   - Port cleanup and temporary directory management
   - Emergency cleanup functions for resource recovery

**Implementation Patterns:**
```typescript
// Always use finally blocks for guaranteed cleanup
try {
  const browser = await puppeteer.launch(options);
  // ... browser operations
} finally {
  // Guaranteed cleanup regardless of success/failure
  await ensureBrowserCleanup(browser, sessionId);
  await cleanupSessionTasks(sessionId);
}

// Background monitoring for long-running operations
const cleanupService = new AutomatedCleanupService();
cleanupService.startPeriodicCleanup(); // Every 10 minutes
```


### API Route Structure
All core functionality exposed via Next.js API routes:
```typescript
// Enhanced screenshot with progress tracking and robust cleanup
POST /api/enhanced-screenshot
{
  symbol: "SOLUSD", 
  timeframe: "240", 
  layouts: ["ai", "diy"],
  analyze: true
}
// Returns: { screenshots, analysis, sessionId }
// Note: Includes automatic Chromium process cleanup via finally blocks

// Drift trading endpoints
GET /api/balance          # Account balance/collateral
POST /api/trading         # Execute trades
GET /api/status          # Trading status
GET /api/automation/position-monitor  # Position monitoring with orphaned cleanup
POST /api/drift/cleanup-orders        # Manual order cleanup
```

### Progress Tracking System
Real-time operation tracking for long-running tasks:
- `lib/progress-tracker.ts` manages EventEmitter-based progress
- SessionId-based tracking for multi-step operations
- Steps: init → auth → navigation → loading → capture → analysis
- Stream endpoint: `/api/progress/[sessionId]/stream`

### Page Structure & Multi-Timeframe Implementation
- `app/analysis/page.js` - Original analysis page with multi-timeframe functionality
- `app/automation/page.js` - Original automation page (legacy, may have issues)
- `app/automation-v2/page.js` - **NEW**: Clean automation page with full multi-timeframe support
- `app/automation/page-v2.js` - Alternative implementation, same functionality as automation-v2

**Multi-Timeframe Architecture Pattern:**
```javascript
// Standard timeframes array - use this exact format
const timeframes = ['5m', '15m', '30m', '1h', '2h', '4h', '1d'];

// State management for multi-timeframe selection
const [selectedTimeframes, setSelectedTimeframes] = useState(['1h', '4h']);

// Toggle function with proper array handling
const toggleTimeframe = (tf) => {
  setSelectedTimeframes(prev => 
    prev.includes(tf) 
      ? prev.filter(t => t !== tf)  // Remove if selected
      : [...prev, tf]                // Add if not selected
  );
};

// Preset configurations for trading styles
const presets = {
  scalping: ['5m', '15m', '1h'],
  day: ['1h', '4h', '1d'],
  swing: ['4h', '1d']
};
```

### Component Architecture
- `app/layout.js` - Root layout with gradient styling and navigation
- `components/Navigation.tsx` - Multi-page navigation system
- `components/AIAnalysisPanel.tsx` - Multi-timeframe analysis interface
- `components/Dashboard.tsx` - Main trading dashboard with real Drift positions
- `components/AdvancedTradingPanel.tsx` - Drift Protocol trading interface

Critical timeframe handling to avoid TradingView confusion:
```typescript
// ALWAYS use minute values first, then alternatives
'4h': ['240', '240m', '4h', '4H'] // 240 minutes FIRST
'1h': ['60', '60m', '1h', '1H']   // 60 minutes FIRST
'15m': ['15', '15m']
```

Layout URL mappings for direct navigation:
```typescript
const LAYOUT_URLS = {
  'ai': 'Z1TzpUrf',    // RSI + EMAs + MACD
  'diy': 'vWVvjLhP'    // Stochastic RSI + VWAP + OBV
}
```

## 🧠 AI Learning System Patterns
```javascript
async generateLearningReport() {
  // Return comprehensive learning status
  return {
    summary: { totalDecisions, systemConfidence, successRate },
    insights: { thresholds, confidenceLevel },
    recommendations: []
  };
}

async getSmartRecommendation(requestData) {
  // Analyze patterns and return AI recommendation
  const { distanceFromSL, symbol, marketConditions } = requestData;
  // Return: { action, confidence, reasoning }
}

async recordDecision(decisionData) {
  // Log decision for learning with unique ID
  const id = `decision_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
  // Store in database for pattern analysis
}

async assessDecisionOutcome(outcomeData) {
  // Update decision with actual result for learning
  // Calculate if decision was correct based on outcome
}
```

### Database Operations Best Practices:
```javascript
// ALWAYS provide unique IDs for Prisma records
await prisma.ai_learning_data.create({
  data: {
    id: `${prefix}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,
    // ... other fields
  }
});

// Use correct import path
const { getDB } = require('./db');  // NOT './database-util'
```

### Prisma Table Name Debugging (Critical)
**Problem**: Database queries fail with "Cannot read properties of undefined (reading 'findMany')" despite correct syntax.

**Root Cause**: Prisma model names vs database table names can differ, causing silent failures that fall back to alternative APIs (like Binance instead of CoinGecko).

**Common Issues**:
```javascript
// ❌ Wrong - will cause undefined errors
await prisma.trade.findMany()              // Should be 'trades' 
await prisma.automationSession.findMany()  // Should be 'automation_sessions'

// ✅ Correct - matches actual database schema
await prisma.trades.findMany()
await prisma.automation_sessions.findMany()
```

**Debugging Steps**:
1. **Check Prisma schema** (`prisma/schema.prisma`) for actual model names
2. **Verify table names** in database: `PRAGMA table_info(trades);`
3. **Test queries directly**: `node -e "const prisma = new PrismaClient(); prisma.trades.findMany().then(console.log);"`
4. **Look for fallback behavior** - API might silently use backup data sources when DB fails
5. **Monitor logs for price source errors** - DB failures often cause price fetching fallbacks

**Impact**: Database errors can cause price monitors to fail and fall back to wrong price sources (Binance instead of CoinGecko), affecting trading accuracy.

### Always Include These Functions in Learning Classes:
```javascript
async generateLearningReport() {
  // Return comprehensive learning status
  return {
    summary: { totalDecisions, systemConfidence, successRate },
    insights: { thresholds, confidenceLevel },
    recommendations: []
  };
}

async getSmartRecommendation(requestData) {
  // Analyze patterns and return AI recommendation
  const { distanceFromSL, symbol, marketConditions } = requestData;
  // Return: { action, confidence, reasoning }
}

async recordDecision(decisionData) {
  // Log decision for learning with unique ID
  const id = `decision_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
  // Store in database for pattern analysis
}

async assessDecisionOutcome(outcomeData) {
  // Update decision with actual result for learning
  // Calculate if decision was correct based on outcome
}
```

### Database Operations Best Practices:
```javascript
// ALWAYS provide unique IDs for Prisma records
await prisma.ai_learning_data.create({
  data: {
    id: `${prefix}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,
    // ... other fields
  }
});

// Use correct import path
const { getDB } = require('./db');  // NOT './database-util'
```

### Prisma Table Name Debugging (Critical)
**Problem**: Database queries fail with "Cannot read properties of undefined (reading 'findMany')" despite correct syntax.

**Root Cause**: Prisma model names vs database table names can differ, causing silent failures that fall back to alternative APIs (like Binance instead of CoinGecko).

**Common Issues**:
```javascript
// ❌ Wrong - will cause undefined errors
await prisma.trade.findMany()              // Should be 'trades' 
await prisma.automationSession.findMany()  // Should be 'automation_sessions'

// ✅ Correct - matches actual database schema
await prisma.trades.findMany()
await prisma.automation_sessions.findMany()
```

**Debugging Steps**:
1. **Check Prisma schema** (`prisma/schema.prisma`) for actual model names
2. **Verify table names** in database: `PRAGMA table_info(trades);`
3. **Test queries directly**: `node -e "const prisma = new PrismaClient(); prisma.trades.findMany().then(console.log);"`
4. **Look for fallback behavior** - API might silently use backup data sources when DB fails
5. **Monitor logs for price source errors** - DB failures often cause price fetching fallbacks

**Impact**: Database errors can cause price monitors to fail and fall back to wrong price sources (Binance instead of CoinGecko), affecting trading accuracy.

### Always Include These Functions in Learning Classes:
```javascript
// ALWAYS provide unique IDs for Prisma records
await prisma.ai_learning_data.create({
  data: {
    id: `${prefix}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,
    // ... other fields
  }
});

// Use correct import path
const { getDB } = require('./db');  // NOT './database-util'
```

## 🔧 Error Handling Patterns

## 🔧 Error Handling Patterns

### Function Existence Checks:
```javascript
// Always check if functions exist before calling
if (typeof this.learner.generateLearningReport === 'function') {
  const report = await this.learner.generateLearningReport();
} else {
  // Fallback to alternative method
  const status = await this.learner.getLearningStatus();
}
```

### Comprehensive Try-Catch:
```javascript
try {
  const result = await aiFunction();
  return result;
} catch (error) {
  await this.log(`❌ AI function error: ${error.message}`);
  return fallbackResult(); // Always provide fallback
}
```

## 📊 Integration Patterns
```javascript
// Always check if functions exist before calling
if (typeof this.learner.generateLearningReport === 'function') {
  const report = await this.learner.generateLearningReport();
} else {
  // Fallback to alternative method
  const status = await this.learner.getLearningStatus();
}
```

### Comprehensive Try-Catch:
```javascript
try {
  const result = await aiFunction();
  return result;
} catch (error) {
  await this.log(`❌ AI function error: ${error.message}`);
  return fallbackResult(); // Always provide fallback
}
```

## 📊 Integration Patterns

### Position Monitor Integration:
```javascript
// When no position detected, check for orphaned orders
if (!result.hasPosition) {
  console.log('📋 No active positions detected - checking for orphaned orders...');
  
  try {
    const ordersResponse = await fetch(`${baseUrl}/api/drift/orders`);
    if (ordersResponse.ok) {
      const ordersData = await ordersResponse.json();
      if (ordersData.orders?.length > 0) {
        // Trigger cleanup
        const cleanupResponse = await fetch(`${baseUrl}/api/drift/cleanup-orders`, {
          method: 'POST'
        });
        // Handle cleanup result
      }
    }
  } catch (error) {
    // Handle error gracefully
  }
}
```

### Parallel Processing for Screenshots:
```javascript
// Use Promise.allSettled for parallel processing
const promises = timeframes.map(timeframe => 
  captureTimeframe(timeframe, symbol, layoutType)
);
const results = await Promise.allSettled(promises);

// Process results with error isolation
results.forEach((result, index) => {
  if (result.status === 'fulfilled') {
    // Handle success
  } else {
    // Handle individual failure without breaking others
  }
});
```

## 🎯 Performance Optimization Rules

### Screenshot Capture:
- Always use parallel processing for multiple timeframes
- Reuse browser sessions to avoid login/captcha
- Isolate errors so one failure doesn't break others
- Prefer `Promise.allSettled` over `Promise.all`

### Database Queries:
- Use indexed fields for frequent searches (symbol, createdAt)
- Batch operations when possible
- Include proper error handling for connection issues

### Container Optimization:
- Check syntax before deployment: `node -c filename.js`
- Use health checks for monitoring
- Implement graceful shutdown handling

## 🧪 Testing Requirements

### Always Include These Tests:
```javascript
// Test AI learning functions
const learner = new SimplifiedStopLossLearner();
const report = await learner.generateLearningReport();
console.log('Learning report:', report.summary);

// Test API endpoints
const response = await fetch('/api/automation/position-monitor');
const result = await response.json();
console.log('Position monitor working:', result.success);

// Test error scenarios
try {
  await riskyFunction();
} catch (error) {
  console.log('Error handling working:', error.message);
}
```

## 🎨 UI/UX Patterns

### Preset Configuration:
```javascript
// Frontend presets MUST match backend exactly
const TRADING_PRESETS = {
  scalp: ['5m', '15m', '30m'],    // NOT ['5m', '15m', '1h']
  day: ['1h', '2h'],              // NOT ['1h', '4h', '1d']  
  swing: ['4h', '1D'],
  extended: ['1m', '3m', '5m', '15m', '30m', '1h', '4h', '1D']
};
```

### Status Display:
```javascript
// Always provide detailed feedback
return {
  success: true,
  monitor: {
    hasPosition: false,
    orphanedOrderCleanup: {
      triggered: true,
      success: true,
      message: 'Cleaned up 2 orphaned orders',
      summary: { totalCanceled: 2 }
    }
  }
};
```

## 🔍 Debugging Strategies

### Container Issues:
```bash
# Check for syntax errors
find . -name "*.js" -exec node -c {} \;

# Monitor logs for patterns
docker logs trader_dev --since="1m" | grep -E "(Error|unhandled|crash)"

# Test specific components
node test-learning-system.js
```

### Integration Issues:
```bash
# Test API endpoints individually
curl -s http://localhost:9001/api/automation/position-monitor | jq .

# Verify database connectivity
node -e "const {getDB} = require('./lib/db'); getDB().then(() => console.log('DB OK'));"
```

### Automation Loop Debugging:
```bash
# Track automation cycles and recommendations
docker logs trader_dev --since="5m" | grep -E "(AUTO-RESTART|recommendation|CYCLE)" | tail -10

# Monitor order behavior patterns
curl -s http://localhost:9001/api/drift/orders | jq '.orders | map(select(.status == "CANCELED")) | length'

# Check if position detection is working
curl -s http://localhost:9001/api/drift/positions | jq '.positions | length'

# Verify cleanup operations
curl -s http://localhost:9001/api/automation/position-monitor | jq '.monitor.orphanedOrderCleanup'
```

### Docker Volume Mount Debugging (Critical Learning)
**Problem**: Code changes don't reflect in container, or container has different file content than host.

**Root Cause**: Volume mounts in `docker-compose.dev.yml` synchronize host directories to container:
```yaml
volumes:
  - ./app:/app/app:cached        # Host ./app → Container /app/app
  - ./lib:/app/lib:cached        # Host ./lib → Container /app/lib
```

**Debugging Workflow**:
```bash
# 1. Always check what's actually in the container vs host
docker compose -f docker-compose.dev.yml exec app bash -c "grep -n 'problem_pattern' /app/app/api/file.js"
grep -n 'problem_pattern' app/api/file.js

# 2. Compare file checksums to verify sync
sha256sum app/api/file.js
docker compose -f docker-compose.dev.yml exec app bash -c "sha256sum /app/app/api/file.js"

# 3. Check if Next.js compiled cache is stale
docker compose -f docker-compose.dev.yml exec app bash -c "ls -la /app/.next/server/"
docker compose -f docker-compose.dev.yml exec app bash -c "grep -r 'problem_pattern' /app/.next/server/ || echo 'Not in compiled cache'"

# 4. Clear Next.js cache if files match but behavior doesn't
docker compose -f docker-compose.dev.yml exec app bash -c "rm -rf /app/.next"
docker compose -f docker-compose.dev.yml restart
```

**Key Insights**:
- **Host edits sync to container** automatically via volume mounts
- **Container file copies are overwritten** by volume mounts on restart
- **Next.js compilation cache** in `.next/` can persist old code even after file changes
- **Always verify container content** matches expectations before debugging logic
- **Compiled webpack bundles** may differ from source files - check both

**Troubleshooting Steps**:
1. **Verify host file has expected changes** (`grep`, `cat`, `sed -n '90,100p'`)
2. **Confirm container file matches host** (checksums, direct comparison)
3. **Check if .next cache is stale** (search compiled files for old patterns)
4. **Clear compilation cache and restart** if source is correct but behavior wrong
5. **Use container logs to trace actual execution** vs expected code paths

## <20> Automation Interference Patterns (Critical Learning)

### Auto-Restart Loop Detection & Prevention
**Problem Pattern**: Position monitors with hardcoded "START_TRADING" recommendations create infinite restart loops when no positions are detected, causing rapid order cancellations.

**Root Cause Symptoms**:
```bash
# Log patterns indicating auto-restart loops
docker logs trader_dev | grep "AUTO-RESTART.*START_TRADING"
docker logs trader_dev | grep "No position detected.*recommendation"
docker logs trader_dev | grep "triggering auto-restart"
```

**Detection Commands**:
```bash
# Check for restart loop patterns
docker logs trader_dev --since="10m" | grep -E "(CYCLE|recommendation|AUTOMATION)" | tail -15

# Monitor order cancellation frequency  
curl -s http://localhost:9001/api/drift/orders | jq '.orders | map(select(.status == "CANCELED")) | length'

# Check position monitor behavior
curl -s http://localhost:9001/api/automation/position-monitor | jq '.monitor.recommendation'
```

**Solution Pattern**:
```javascript
// ❌ WRONG: Hardcoded recommendation causes loops
const result = {
  recommendation: 'START_TRADING', // Always triggers restart
  hasPosition: false // When combined, creates infinite loop
};

// ✅ CORRECT: Context-aware recommendations
const result = {
  recommendation: hasPosition ? 'MONITOR_POSITION' : 'MONITOR_ONLY',
  hasPosition: false // Safe - no auto-restart trigger
};

// ✅ CORRECT: Disable auto-restart entirely for manual control
/* Auto-restart logic disabled to prevent interference with manual trading */
```

**Prevention Checklist**:
- [ ] Position monitor recommendations are context-aware, not hardcoded
- [ ] Auto-restart logic includes manual override capabilities  
- [ ] Order placement doesn't trigger immediate cleanup cycles
- [ ] System allows manual trading without automation interference
- [ ] Logs show clean monitoring without constant restart attempts

## <20>🚨 Critical Anti-Patterns to Avoid

### ❌ Don't Do This:
```javascript
// Missing error handling
const report = await this.learner.generateLearningReport(); // Will crash if function missing

// Redundant polling
setInterval(checkOrders, 60000); // When position monitor already runs frequently

// Auto-restart loops that interfere with trading
recommendation: 'START_TRADING', // Hardcoded - causes constant restart triggers
if (!hasPosition && recommendation === 'START_TRADING') {
  // Auto-restart logic that triggers rapid cleanup cycles
}

// Frontend/backend preset mismatch  
backend: ['5m', '15m', '1h']
frontend: ['5m', '15m', '30m'] // Will cause confusion

// Missing unique IDs
await prisma.create({ data: { symbol, timeframe } }); // Will fail validation
```

### ✅ Do This Instead:
```javascript
// Defensive programming
if (typeof this.learner.generateLearningReport === 'function') {
  try {
    const report = await this.learner.generateLearningReport();
  } catch (error) {
    await this.log(`Report generation failed: ${error.message}`);
  }
}

// Leverage existing infrastructure
// Add cleanup to existing position monitor instead of new polling

// Smart recommendations that don't trigger loops
recommendation: hasPosition ? 'MONITOR_POSITION' : 'MONITOR_ONLY', // Context-aware
// Disable auto-restart for manual control
/* Auto-restart logic disabled to prevent interference */

// Ensure consistency
const PRESETS = { scalp: ['5m', '15m', '30m'] }; // Same in frontend and backend

// Always provide unique IDs
const id = `${type}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
```

## 🚨 CRITICAL ANTI-PATTERNS TO AVOID

### ❌ Don't Do This:
```javascript
// Missing error handling
const report = await this.learner.generateLearningReport(); // Will crash if function missing

// Redundant polling
setInterval(checkOrders, 60000); // When position monitor already runs frequently

// Auto-restart loops that interfere with trading
recommendation: 'START_TRADING', // Hardcoded - causes constant restart triggers
if (!hasPosition && recommendation === 'START_TRADING') {
  // Auto-restart logic that triggers rapid cleanup cycles
}

// Frontend/backend preset mismatch  
backend: ['5m', '15m', '1h']
frontend: ['5m', '15m', '30m'] // Will cause confusion

// Missing unique IDs
await prisma.create({ data: { symbol, timeframe } }); // Will fail validation

// Recursive API calls (CRITICAL)
// In /api/enhanced-screenshot calling superiorScreenshotService which calls /api/enhanced-screenshot
import { superiorScreenshotService } from '../../../lib/superior-screenshot-service'
const screenshots = await superiorScreenshotService.captureQuick() // CAUSES INFINITE RECURSION

// TypeScript imports in JavaScript API routes
import { EnhancedScreenshotService } from '../../../lib/enhanced-screenshot' // FAILS SILENTLY

// Wrong AI analysis method names
analysis = await aiAnalysisService.analyzeScreenshots(config) // METHOD DOESN'T EXIST

// Wrong progress tracker usage
sessionId = progressTracker.createSession() // MISSING REQUIRED PARAMETERS
progressTracker.initializeSteps(sessionId, steps) // METHOD DOESN'T EXIST
```

### ✅ Do This Instead:
```javascript
// Defensive programming
if (typeof this.learner.generateLearningReport === 'function') {
  try {
    const report = await this.learner.generateLearningReport();
  } catch (error) {
    await this.log(`Report generation failed: ${error.message}`);
  }
}

// Leverage existing infrastructure
// Add cleanup to existing position monitor instead of new polling

// Smart recommendations that don't trigger loops
recommendation: hasPosition ? 'MONITOR_POSITION' : 'MONITOR_ONLY', // Context-aware
// Disable auto-restart for manual control
/* Auto-restart logic disabled to prevent interference */

// Ensure consistency
const PRESETS = { scalp: ['5m', '15m', '30m'] }; // Same in frontend and backend

// Always provide unique IDs
const id = `${type}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;

// Correct service usage (NO RECURSION)
const { EnhancedScreenshotService } = await import('../../../lib/enhanced-screenshot')
const service = new EnhancedScreenshotService()
const screenshots = await service.captureWithLogin(config)

// Dynamic imports for TypeScript in JavaScript
const { EnhancedScreenshotService } = await import('../../../lib/enhanced-screenshot')

// Correct AI analysis method names
if (screenshots.length === 1) {
  analysis = await aiAnalysisService.analyzeScreenshot(screenshots[0])
} else {
  analysis = await aiAnalysisService.analyzeMultipleScreenshots(screenshots)
}

// Correct progress tracker usage
sessionId = `session_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`
const progress = progressTracker.createSession(sessionId, progressSteps)
```

## 🎯 Configuration Standards

### Environment Variables:
```javascript
// Always provide fallbacks
const apiKey = process.env.OPENAI_API_KEY || '';
if (!apiKey) {
  throw new Error('OPENAI_API_KEY is required');
}
```

### Next.js Configuration:
```typescript
// Use new format, not deprecated
const nextConfig: NextConfig = {
  serverExternalPackages: ['puppeteer-core'], // NOT experimental.serverComponentsExternalPackages
  transpilePackages: ['next-font'],
  typescript: { ignoreBuildErrors: true },
  eslint: { ignoreDuringBuilds: true }
};
```

## 📈 Enhancement Guidelines

When adding new features:

1. **Check Existing Infrastructure** - Can it be integrated vs creating new?
2. **Add Comprehensive Error Handling** - Assume functions may not exist
3. **Include Fallback Mechanisms** - System should work without AI/learning
4. **Test in Isolation** - Create test scripts for new components
5. **Document Integration Points** - How does it connect to existing systems?
6. **Maintain Consistency** - Frontend and backend must match exactly
7. **Use Defensive Programming** - Check before calling, handle gracefully
8. **Avoid Recursive API Calls** - Never call same endpoint from service layer
9. **Use Dynamic Imports** - For TypeScript modules in JavaScript API routes
10. **Verify Method Existence** - Check actual method signatures before calling
11. **Test API Chain Dependencies** - Ensure entire call chain works end-to-end
12. **Validate Real vs Mock Data** - Confirm actual analysis data, not fallback responses

### 🔧 Critical Debugging Workflow for New Features:
```bash
# 1. Test API endpoint directly
curl -X POST http://localhost:9001/api/your-endpoint \
  -H "Content-Type: application/json" \
  -d '{"test":"data"}' --max-time 30

# 2. Check container logs for errors
docker compose -f docker-compose.dev.yml logs --since="1m" | grep -E "ERROR|undefined|is not a function"

# 3. Verify method existence in container
docker compose -f docker-compose.dev.yml exec app bash -c \
  "node -e \"import('./lib/your-service').then(m => console.log(Object.keys(m)))\""

# 4. Test complete integration chain
curl "http://localhost:9001/api/ai-analysis/latest?symbol=SOLUSD&timeframe=60"
```

### 🚨 Pre-Deployment Checklist:
- [ ] API endpoints return real data, not mock/fallback responses
- [ ] No recursive API calls in service layers
- [ ] TypeScript imports use dynamic imports in JavaScript files
- [ ] All method calls verified to exist with correct signatures
- [ ] Error handling includes function existence checks
- [ ] Progress tracking uses correct method signatures
- [ ] Database queries use correct table/model names
- [ ] Container logs show no import/method errors
- [ ] End-to-end API chain tested and working
- [ ] Real analysis data verified (proper confidence scores, realistic timing)

## 📚 Documentation References

### Technical Documentation
- **`ADVANCED_SYSTEM_KNOWLEDGE.md`** - Deep technical architecture, session management, cleanup systems
- **`README.md`** - Main project overview with current feature status and setup
- **`AI_LEARNING_EXPLAINED.md`** - AI learning system implementation details
- **`DRIFT_FEEDBACK_LOOP_COMPLETE.md`** - Complete Drift trading integration
- **`ROBUST_CLEANUP_IMPLEMENTATION.md`** - Browser process cleanup system details

### Implementation Guides
- **`MULTI_LAYOUT_IMPLEMENTATION.md`** - Dual-session screenshot system
- **`SESSION_PERSISTENCE.md`** - TradingView session management
- **`DOCKER_AUTOMATION.md`** - Container development workflow
- **`DEVELOPMENT_GUIDE.md`** - Complete development setup instructions

### Analysis & Troubleshooting
- **`MULTI_LAYOUT_TROUBLESHOOTING.md`** - Screenshot automation debugging
- **`CLEANUP_IMPROVEMENTS.md`** - Process management enhancements
- **`SCREENSHOT_PATH_FIXES.md`** - Screenshot capture issue resolution

## 🎓 CRITICAL LESSONS LEARNED (Session: July 30, 2025)

### 🚨 Major Bug Pattern Discovered: API Recursion Loops
**Problem**: Enhanced Screenshot API was calling itself infinitely through service layer
**Root Cause**: `superiorScreenshotService.captureQuick()` internally called `/api/enhanced-screenshot`
**Solution**: Switch to direct `EnhancedScreenshotService` with `captureWithLogin()`
**Detection**: 500 errors, container logs showing recursive call patterns
**Prevention**: Never import services that call the same API endpoint

### 🔧 TypeScript/JavaScript Import Issues in API Routes
**Problem**: Static imports of `.ts` files in `.js` API routes fail silently
**Symptoms**: "Cannot read properties of undefined" errors, missing method errors
**Solution**: Use dynamic imports: `const { Service } = await import('./service')`
**Critical**: Always test imports in container before deployment

### 📊 Progress Tracker Method Signature Errors
**Problem**: Calling `progressTracker.createSession()` without parameters
**Error**: "Cannot read properties of undefined (reading 'length')"
**Solution**: Always provide sessionId and steps: `createSession(sessionId, progressSteps)`
**Learning**: Check actual method signatures, don't assume parameter patterns

### 🤖 AI Analysis Service Method Name Confusion
**Problem**: `aiAnalysisService.analyzeScreenshots()` doesn't exist
**Correct Methods**: `analyzeScreenshot()` (single) and `analyzeMultipleScreenshots()` (array)
**Pattern**: Always verify method names in TypeScript service files
**Debugging**: Use container exec to test imports and list available methods

### 🔄 Real vs Mock Data Integration Validation
**Critical Insight**: APIs can appear to work but return fake data due to failed imports
**Validation Techniques**:
- Response timing: Real analysis takes 30-180 seconds, mock returns in <5 seconds
- Confidence variance: Real analysis varies 60-90%, mock often fixed at 75%
- Technical detail depth: Real includes specific indicators, entry/exit levels
- Layout analysis: Real mentions "AI Layout" and "DIY Layout" comparison

### 🐛 Container Development Debugging Workflow
**Key Learning**: Always verify what's actually running in the container
**Essential Commands**:
```bash
# Check file sync between host and container
sha256sum app/api/file.js
docker compose exec app bash -c "sha256sum /app/app/api/file.js"

# Clear Next.js compilation cache when behavior doesn't match source
docker compose exec app bash -c "rm -rf /app/.next"

# Test imports directly in container
docker compose exec app bash -c "node -e \"import('./lib/service').then(console.log)\""
```

### 🎯 API Integration Chain Dependencies
**Critical Understanding**: Failure in one link breaks entire automation
**Chain**: GET SIGNAL → ai-analysis/latest → enhanced-screenshot → EnhancedScreenshotService → AI Analysis
**Testing Strategy**: Test each link individually, then validate end-to-end
**Monitoring**: Watch for "Failed to get real screenshot analysis" errors

---

**These patterns represent the most common and critical issues encountered during real data integration. Understanding these will prevent weeks of debugging in future development.**

---

**Follow these patterns to maintain system stability and avoid the complex debugging issues that were resolved in this session.**