trading_bot_v3/.github/copilot-instructions.md

# GitHub Copilot Instructions for Trading Bot Development

## 🎯 Project Context & Architecture

This is an AI-powered trading automation system with advanced learning capabilities built with Next.js 15 App Router, TypeScript, Tailwind CSS, and integrated with Drift Protocol and Jupiter DEX for automated trading execution.

### Core System Components
1. **Superior Parallel Screenshot System** - 60% faster than sequential capture (71s vs 180s)
2. **AI Learning System** - Adapts trading decisions based on outcomes with pattern recognition  
3. **Orphaned Order Cleanup** - Automatic cleanup when positions close via position monitor
4. **Position Monitoring** - Frequent checks with integrated cleanup triggers
5. **Dual-Session Screenshot Automation** - AI and DIY layouts with session persistence
6. **Robust Cleanup System** - Prevents Chromium process accumulation

### Critical File Relationships
```
app/api/automation/position-monitor/route.js → Monitors positions + triggers cleanup
lib/simplified-stop-loss-learner.js → AI learning core with pattern recognition
lib/superior-screenshot-service.ts → Parallel screenshot capture system
lib/enhanced-autonomous-risk-manager.js → Risk management with AI integration
lib/enhanced-screenshot-robust.ts → Guaranteed cleanup with finally blocks
lib/automated-cleanup-service.ts → Background process monitoring
```

## 🚀 Development Environment (Critical)

### Docker Container Development (Required)
**All development happens inside Docker containers** using Docker Compose v2. Browser automation requires specific system dependencies only available in containerized environment.

**IMPORTANT: Use Docker Compose v2 syntax** - All commands use `docker compose` (with space) instead of `docker-compose` (with hyphen).

```bash
# Development environment - Docker Compose v2 dev setup
npm run docker:dev        # Port 9001:3000, hot reload, debug mode
# Direct v2 command: docker compose -f docker-compose.dev.yml up --build

# Production environment  
npm run docker:up         # Port 9000:3000, optimized build
# Direct v2 command: docker compose -f docker-compose.prod.yml up --build

# Debugging commands
npm run docker:logs       # View container logs
npm run docker:exec       # Shell access for debugging inside container
```

**Port Configuration:**
- **Development**: External port `9001` → Internal port `3000` (http://localhost:9001)
- **Production**: External port `9000` → Internal port `3000` (http://localhost:9000)

### Container-First Development Workflow
**Common Issue**: File edits not reflecting in container due to volume mount sync issues.

**Solution - Container Development Workflow:**
```bash
# 1. Access running container for immediate edits
docker compose -f docker-compose.dev.yml exec app bash

# 2. Edit files directly in container (immediate effect)
nano /app/lib/enhanced-screenshot.ts
echo "console.log('Debug: immediate test');" >> /app/debug.js

# 3. Test changes immediately (no rebuild needed)
# Changes take effect instantly for hot reload

# 4. Once everything works, copy changes back to host
docker cp container_name:/app/modified-file.js ./modified-file.js

# 5. Commit successful changes to git BEFORE rebuilding
git add .
git commit -m "feat: implement working solution for [specific feature]"
git push origin development

# 6. Rebuild container for persistence
docker compose -f docker-compose.dev.yml down
docker compose -f docker-compose.dev.yml up --build -d

# 7. Final validation and commit completion
curl http://localhost:9001  # Verify functionality
git add . && git commit -m "chore: confirm container persistence" && git push
```

### Git Branch Strategy (Required)
**Primary development workflow:**
- **`development` branch**: Use for all active development and feature work
- **`main` branch**: Stable, production-ready code only
- **Workflow**: Develop on `development` → test thoroughly → commit progress → merge to `main` when stable

```bash
# Standard development workflow with frequent commits
git checkout development        # Always start here
git pull origin development     # Get latest changes

# Make your changes and test in container...

# Commit working progress BEFORE rebuilding container
git add .
git commit -m "feat: [specific achievement] - tested and working"
git push origin development

# After successful container rebuild and validation
git add .
git commit -m "chore: confirm [feature] persistence after rebuild"
git push origin development

# Only merge to main when features are stable and tested
git checkout main
git merge development          # When ready for production
git push origin main
```

## 🏗️ System Architecture

### Dual-Session Screenshot Automation
- **AI Layout**: `Z1TzpUrf` - RSI (top), EMAs, MACD (bottom)
- **DIY Layout**: `vWVvjLhP` - Stochastic RSI (top), VWAP, OBV (bottom)
- Parallel browser sessions for multi-layout capture in `lib/enhanced-screenshot.ts`
- TradingView automation with session persistence in `lib/tradingview-automation.ts`
- Session data stored in `.tradingview-session/` volume mount to avoid captchas

### AI Analysis Pipeline  
- OpenAI GPT-4o mini for cost-effective chart analysis (~$0.006 per analysis)
- Multi-layout comparison and consensus detection in `lib/ai-analysis.ts`
- Professional trading setups with exact entry/exit levels and risk management
- Layout-specific indicator analysis (RSI vs Stochastic RSI, MACD vs OBV)

### Trading Integration
- **Drift Protocol**: Perpetual futures trading via `@drift-labs/sdk`
- **Jupiter DEX**: Spot trading on Solana
- Position management and P&L tracking in `lib/drift-trading-final.ts`
- Real-time account balance and collateral monitoring

### Browser Process Management & Cleanup System
**Critical Issue**: Chromium processes accumulate during automated trading, consuming system resources over time.

**Robust Cleanup Implementation:**
1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`)
   - Guaranteed cleanup via `finally` blocks in all browser operations
   - Active session tracking to prevent orphaned browsers
   - Session cleanup tasks array for systematic teardown

2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`)
   - Background monitoring service for orphaned processes
   - Multiple kill strategies: graceful → force → system cleanup
   - Periodic cleanup of temporary files and browser data

3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`)
   - System-level process killing for stubborn Chromium processes
   - Port cleanup and temporary directory management
   - Emergency cleanup functions for resource recovery

**Implementation Patterns:**
```typescript
// Always use finally blocks for guaranteed cleanup
try {
  const browser = await puppeteer.launch(options);
  // ... browser operations
} finally {
  // Guaranteed cleanup regardless of success/failure
  await ensureBrowserCleanup(browser, sessionId);
  await cleanupSessionTasks(sessionId);
}

// Background monitoring for long-running operations
const cleanupService = new AutomatedCleanupService();
cleanupService.startPeriodicCleanup(); // Every 10 minutes
```


### API Route Structure
All core functionality exposed via Next.js API routes:
```typescript
// Enhanced screenshot with progress tracking and robust cleanup
POST /api/enhanced-screenshot
{
  symbol: "SOLUSD", 
  timeframe: "240", 
  layouts: ["ai", "diy"],
  analyze: true
}
// Returns: { screenshots, analysis, sessionId }
// Note: Includes automatic Chromium process cleanup via finally blocks

// Drift trading endpoints
GET /api/balance          # Account balance/collateral
POST /api/trading         # Execute trades
GET /api/status          # Trading status
GET /api/automation/position-monitor  # Position monitoring with orphaned cleanup
POST /api/drift/cleanup-orders        # Manual order cleanup
```

### Progress Tracking System
Real-time operation tracking for long-running tasks:
- `lib/progress-tracker.ts` manages EventEmitter-based progress
- SessionId-based tracking for multi-step operations
- Steps: init → auth → navigation → loading → capture → analysis
- Stream endpoint: `/api/progress/[sessionId]/stream`

### Page Structure & Multi-Timeframe Implementation
- `app/analysis/page.js` - Original analysis page with multi-timeframe functionality
- `app/automation/page.js` - Original automation page (legacy, may have issues)
- `app/automation-v2/page.js` - **NEW**: Clean automation page with full multi-timeframe support
- `app/automation/page-v2.js` - Alternative implementation, same functionality as automation-v2

**Multi-Timeframe Architecture Pattern:**
```javascript
// Standard timeframes array - use this exact format
const timeframes = ['5m', '15m', '30m', '1h', '2h', '4h', '1d'];

// State management for multi-timeframe selection
const [selectedTimeframes, setSelectedTimeframes] = useState(['1h', '4h']);

// Toggle function with proper array handling
const toggleTimeframe = (tf) => {
  setSelectedTimeframes(prev => 
    prev.includes(tf) 
      ? prev.filter(t => t !== tf)  // Remove if selected
      : [...prev, tf]                // Add if not selected
  );
};

// Preset configurations for trading styles
const presets = {
  scalping: ['5m', '15m', '1h'],
  day: ['1h', '4h', '1d'],
  swing: ['4h', '1d']
};
```

### Component Architecture
- `app/layout.js` - Root layout with gradient styling and navigation
- `components/Navigation.tsx` - Multi-page navigation system
- `components/AIAnalysisPanel.tsx` - Multi-timeframe analysis interface
- `components/Dashboard.tsx` - Main trading dashboard with real Drift positions
- `components/AdvancedTradingPanel.tsx` - Drift Protocol trading interface

Critical timeframe handling to avoid TradingView confusion:
```typescript
// ALWAYS use minute values first, then alternatives
'4h': ['240', '240m', '4h', '4H'] // 240 minutes FIRST
'1h': ['60', '60m', '1h', '1H']   // 60 minutes FIRST
'15m': ['15', '15m']
```

Layout URL mappings for direct navigation:
```typescript
const LAYOUT_URLS = {
  'ai': 'Z1TzpUrf',    // RSI + EMAs + MACD
  'diy': 'vWVvjLhP'    // Stochastic RSI + VWAP + OBV
}
```

## 🧠 AI Learning System Patterns
```javascript
async generateLearningReport() {
  // Return comprehensive learning status
  return {
    summary: { totalDecisions, systemConfidence, successRate },
    insights: { thresholds, confidenceLevel },
    recommendations: []
  };
}

async getSmartRecommendation(requestData) {
  // Analyze patterns and return AI recommendation
  const { distanceFromSL, symbol, marketConditions } = requestData;
  // Return: { action, confidence, reasoning }
}

async recordDecision(decisionData) {
  // Log decision for learning with unique ID
  const id = `decision_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
  // Store in database for pattern analysis
}

async assessDecisionOutcome(outcomeData) {
  // Update decision with actual result for learning
  // Calculate if decision was correct based on outcome
}
```

### Database Operations Best Practices:
```javascript
// ALWAYS provide unique IDs for Prisma records
await prisma.ai_learning_data.create({
  data: {
    id: `${prefix}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,
    // ... other fields
  }
});

// Use correct import path
const { getDB } = require('./db');  // NOT './database-util'
```

### Always Include These Functions in Learning Classes:
```javascript
async generateLearningReport() {
  // Return comprehensive learning status
  return {
    summary: { totalDecisions, systemConfidence, successRate },
    insights: { thresholds, confidenceLevel },
    recommendations: []
  };
}

async getSmartRecommendation(requestData) {
  // Analyze patterns and return AI recommendation
  const { distanceFromSL, symbol, marketConditions } = requestData;
  // Return: { action, confidence, reasoning }
}

async recordDecision(decisionData) {
  // Log decision for learning with unique ID
  const id = `decision_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
  // Store in database for pattern analysis
}

async assessDecisionOutcome(outcomeData) {
  // Update decision with actual result for learning
  // Calculate if decision was correct based on outcome
}
```

### Database Operations Best Practices:
```javascript
// ALWAYS provide unique IDs for Prisma records
await prisma.ai_learning_data.create({
  data: {
    id: `${prefix}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,
    // ... other fields
  }
});

// Use correct import path
const { getDB } = require('./db');  // NOT './database-util'
```

## 🔧 Error Handling Patterns

## 🔧 Error Handling Patterns

### Function Existence Checks:
```javascript
// Always check if functions exist before calling
if (typeof this.learner.generateLearningReport === 'function') {
  const report = await this.learner.generateLearningReport();
} else {
  // Fallback to alternative method
  const status = await this.learner.getLearningStatus();
}
```

### Comprehensive Try-Catch:
```javascript
try {
  const result = await aiFunction();
  return result;
} catch (error) {
  await this.log(`❌ AI function error: ${error.message}`);
  return fallbackResult(); // Always provide fallback
}
```

## 📊 Integration Patterns
```javascript
// Always check if functions exist before calling
if (typeof this.learner.generateLearningReport === 'function') {
  const report = await this.learner.generateLearningReport();
} else {
  // Fallback to alternative method
  const status = await this.learner.getLearningStatus();
}
```

### Comprehensive Try-Catch:
```javascript
try {
  const result = await aiFunction();
  return result;
} catch (error) {
  await this.log(`❌ AI function error: ${error.message}`);
  return fallbackResult(); // Always provide fallback
}
```

## 📊 Integration Patterns

### Position Monitor Integration:
```javascript
// When no position detected, check for orphaned orders
if (!result.hasPosition) {
  console.log('📋 No active positions detected - checking for orphaned orders...');
  
  try {
    const ordersResponse = await fetch(`${baseUrl}/api/drift/orders`);
    if (ordersResponse.ok) {
      const ordersData = await ordersResponse.json();
      if (ordersData.orders?.length > 0) {
        // Trigger cleanup
        const cleanupResponse = await fetch(`${baseUrl}/api/drift/cleanup-orders`, {
          method: 'POST'
        });
        // Handle cleanup result
      }
    }
  } catch (error) {
    // Handle error gracefully
  }
}
```

### Parallel Processing for Screenshots:
```javascript
// Use Promise.allSettled for parallel processing
const promises = timeframes.map(timeframe => 
  captureTimeframe(timeframe, symbol, layoutType)
);
const results = await Promise.allSettled(promises);

// Process results with error isolation
results.forEach((result, index) => {
  if (result.status === 'fulfilled') {
    // Handle success
  } else {
    // Handle individual failure without breaking others
  }
});
```

## 🎯 Performance Optimization Rules

### Screenshot Capture:
- Always use parallel processing for multiple timeframes
- Reuse browser sessions to avoid login/captcha
- Isolate errors so one failure doesn't break others
- Prefer `Promise.allSettled` over `Promise.all`

### Database Queries:
- Use indexed fields for frequent searches (symbol, createdAt)
- Batch operations when possible
- Include proper error handling for connection issues

### Container Optimization:
- Check syntax before deployment: `node -c filename.js`
- Use health checks for monitoring
- Implement graceful shutdown handling

## 🧪 Testing Requirements

### Always Include These Tests:
```javascript
// Test AI learning functions
const learner = new SimplifiedStopLossLearner();
const report = await learner.generateLearningReport();
console.log('Learning report:', report.summary);

// Test API endpoints
const response = await fetch('/api/automation/position-monitor');
const result = await response.json();
console.log('Position monitor working:', result.success);

// Test error scenarios
try {
  await riskyFunction();
} catch (error) {
  console.log('Error handling working:', error.message);
}
```

## 🎨 UI/UX Patterns

### Preset Configuration:
```javascript
// Frontend presets MUST match backend exactly
const TRADING_PRESETS = {
  scalp: ['5m', '15m', '30m'],    // NOT ['5m', '15m', '1h']
  day: ['1h', '2h'],              // NOT ['1h', '4h', '1d']  
  swing: ['4h', '1D'],
  extended: ['1m', '3m', '5m', '15m', '30m', '1h', '4h', '1D']
};
```

### Status Display:
```javascript
// Always provide detailed feedback
return {
  success: true,
  monitor: {
    hasPosition: false,
    orphanedOrderCleanup: {
      triggered: true,
      success: true,
      message: 'Cleaned up 2 orphaned orders',
      summary: { totalCanceled: 2 }
    }
  }
};
```

## 🔍 Debugging Strategies

### Container Issues:
```bash
# Check for syntax errors
find . -name "*.js" -exec node -c {} \;

# Monitor logs for patterns
docker logs trader_dev --since="1m" | grep -E "(Error|unhandled|crash)"

# Test specific components
node test-learning-system.js
```

### Integration Issues:
```bash
# Test API endpoints individually
curl -s http://localhost:9001/api/automation/position-monitor | jq .

# Verify database connectivity
node -e "const {getDB} = require('./lib/db'); getDB().then(() => console.log('DB OK'));"
```

## 🚨 Critical Anti-Patterns to Avoid

### ❌ Don't Do This:
```javascript
// Missing error handling
const report = await this.learner.generateLearningReport(); // Will crash if function missing

// Redundant polling
setInterval(checkOrders, 60000); // When position monitor already runs frequently

// Frontend/backend preset mismatch  
backend: ['5m', '15m', '1h']
frontend: ['5m', '15m', '30m'] // Will cause confusion

// Missing unique IDs
await prisma.create({ data: { symbol, timeframe } }); // Will fail validation
```

### ✅ Do This Instead:
```javascript
// Defensive programming
if (typeof this.learner.generateLearningReport === 'function') {
  try {
    const report = await this.learner.generateLearningReport();
  } catch (error) {
    await this.log(`Report generation failed: ${error.message}`);
  }
}

// Leverage existing infrastructure
// Add cleanup to existing position monitor instead of new polling

// Ensure consistency
const PRESETS = { scalp: ['5m', '15m', '30m'] }; // Same in frontend and backend

// Always provide unique IDs
const id = `${type}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
```

## 🎯 Configuration Standards

### Environment Variables:
```javascript
// Always provide fallbacks
const apiKey = process.env.OPENAI_API_KEY || '';
if (!apiKey) {
  throw new Error('OPENAI_API_KEY is required');
}
```

### Next.js Configuration:
```typescript
// Use new format, not deprecated
const nextConfig: NextConfig = {
  serverExternalPackages: ['puppeteer-core'], // NOT experimental.serverComponentsExternalPackages
  transpilePackages: ['next-font'],
  typescript: { ignoreBuildErrors: true },
  eslint: { ignoreDuringBuilds: true }
};
```

## 📈 Enhancement Guidelines

When adding new features:

1. **Check Existing Infrastructure** - Can it be integrated vs creating new?
2. **Add Comprehensive Error Handling** - Assume functions may not exist
3. **Include Fallback Mechanisms** - System should work without AI/learning
4. **Test in Isolation** - Create test scripts for new components
5. **Document Integration Points** - How does it connect to existing systems?
6. **Maintain Consistency** - Frontend and backend must match exactly
7. **Use Defensive Programming** - Check before calling, handle gracefully

## 📚 Documentation References

### Technical Documentation
- **`ADVANCED_SYSTEM_KNOWLEDGE.md`** - Deep technical architecture, session management, cleanup systems
- **`README.md`** - Main project overview with current feature status and setup
- **`AI_LEARNING_EXPLAINED.md`** - AI learning system implementation details
- **`DRIFT_FEEDBACK_LOOP_COMPLETE.md`** - Complete Drift trading integration
- **`ROBUST_CLEANUP_IMPLEMENTATION.md`** - Browser process cleanup system details

### Implementation Guides
- **`MULTI_LAYOUT_IMPLEMENTATION.md`** - Dual-session screenshot system
- **`SESSION_PERSISTENCE.md`** - TradingView session management
- **`DOCKER_AUTOMATION.md`** - Container development workflow
- **`DEVELOPMENT_GUIDE.md`** - Complete development setup instructions

### Analysis & Troubleshooting
- **`MULTI_LAYOUT_TROUBLESHOOTING.md`** - Screenshot automation debugging
- **`CLEANUP_IMPROVEMENTS.md`** - Process management enhancements
- **`SCREENSHOT_PATH_FIXES.md`** - Screenshot capture issue resolution

---

**Follow these patterns to maintain system stability and avoid the complex debugging issues that were resolved in this session.**