trading_bot_v3/.github/copilot-instructions.instructions.md

# AI-Powered Trading Bot Dashboard

This is a Next.js 15 App Router application with TypeScript, Tailwind CSS, and API routes. It's a production-ready trading bot with AI analysis, automated screenshot capture, and real-time trading execution via Drift Protocol and Jupiter DEX.

**Prerequisites:**
- Docker and Docker Compose v2 (uses `docker compose` command syntax)
- All development must be done inside Docker containers for browser automation compatibility

## Core Architecture

### Dual-Session Screenshot Automation
- **AI Layout**: `Z1TzpUrf` - RSI (top), EMAs, MACD (bottom)
- **DIY Layout**: `vWVvjLhP` - Stochastic RSI (top), VWAP, OBV (bottom)
- Parallel browser sessions for multi-layout capture in `lib/enhanced-screenshot.ts`
- TradingView automation with session persistence in `lib/tradingview-automation.ts`
- Session data stored in `.tradingview-session/` volume mount to avoid captchas

### AI Analysis Pipeline  
- OpenAI GPT-4o mini for cost-effective chart analysis (~$0.006 per analysis)
- Multi-layout comparison and consensus detection in `lib/ai-analysis.ts`
- Professional trading setups with exact entry/exit levels and risk management
- Layout-specific indicator analysis (RSI vs Stochastic RSI, MACD vs OBV)

### Trading Integration
- **Drift Protocol**: Perpetual futures trading via `@drift-labs/sdk`
- **Jupiter DEX**: Spot trading on Solana
- Position management and P&L tracking in `lib/drift-trading-final.ts`
- Real-time account balance and collateral monitoring

## Critical Development Patterns

### Docker Container Development (Required)
**All development happens inside Docker containers** using Docker Compose v2. Browser automation requires specific system dependencies that are only available in the containerized environment:

**IMPORTANT: Use Docker Compose v2 syntax** - All commands use `docker compose` (with space) instead of `docker-compose` (with hyphen).

```bash
# Development environment - Docker Compose v2 dev setup
npm run docker:dev        # Port 9001:3000, hot reload, debug mode
# Direct v2 command: docker compose -f docker-compose.dev.yml up --build

# Production environment  
npm run docker:up         # Port 9000:3000, optimized build
# Direct v2 command: docker compose -f docker-compose.prod.yml up --build

# Debugging commands
npm run docker:logs       # View container logs
# Direct v2 command: docker compose -f docker-compose.dev.yml logs -f

npm run docker:exec       # Shell access for debugging inside container
# Direct v2 command: docker compose -f docker-compose.dev.yml exec app bash
```

**Port Configuration:**
- **Development**: External port `9001` → Internal port `3000` (http://localhost:9001)
- **Production**: External port `9000` → Internal port `3000` (http://localhost:9000)

### Docker Volume Mount Troubleshooting & Direct Container Development
**Common Issue**: File edits not reflecting in container due to volume mount sync issues.

**Container-First Development Workflow:**
For immediate results and faster iteration, edit files directly inside the running container first, then rebuild for persistence:

```bash
# 1. Access running container for immediate edits
docker compose -f docker-compose.dev.yml exec app bash

# 2. Edit files directly in container (immediate effect)
# Use nano, vi, or echo for quick changes
nano /app/lib/enhanced-screenshot.ts
echo "console.log('Debug: immediate test');" >> /app/debug.js

# 3. Test changes immediately (no rebuild needed)
# Changes take effect instantly for hot reload

# 4. Once everything works, copy changes back to host
docker cp container_name:/app/modified-file.js ./modified-file.js

# 5. Commit successful changes to git BEFORE rebuilding
git add .
git commit -m "feat: implement working solution for [specific feature]"
git push origin development

# 6. Rebuild container for persistence
docker compose -f docker-compose.dev.yml down
docker compose -f docker-compose.dev.yml up --build -d

# 7. Final validation and commit completion
# Test that changes persist after rebuild
curl http://localhost:9001  # Verify functionality
git add . && git commit -m "chore: confirm container persistence" && git push
```

**Alternative Solutions:**
1. **Fresh Implementation Approach**: When modifying existing files fails, create new files (e.g., `page-v2.js`) instead of editing corrupted files
2. **Container Restart**: `docker compose -f docker-compose.dev.yml restart app`
3. **Full Rebuild**: `docker compose -f docker-compose.dev.yml down && docker compose -f docker-compose.dev.yml up --build`
4. **Manual Copy**: Use `docker cp` to copy files directly into container for immediate testing
5. **Avoid sed/awk**: Direct text manipulation commands often corrupt JSX syntax - prefer file replacement

**Volume Mount Verification:**
```bash
# Test volume mount sync
echo "test-$(date)" > test-volume-mount.txt
docker compose -f docker-compose.dev.yml exec app cat test-volume-mount.txt
```

**Container Development Best Practices:**
- **Speed**: Direct container edits = immediate testing
- **Persistence**: Always rebuild container after successful tests
- **Backup**: Use `docker cp` to extract working changes before rebuild
- **Debugging**: Use container shell for real-time log inspection and debugging

### Multi-Timeframe Feature Copy Pattern
When copying multi-timeframe functionality between pages:

**Step 1: Identify Source Implementation**
```bash
# Search for existing timeframe patterns
grep -r "timeframes.*=.*\[" app/ --include="*.js" --include="*.jsx"
grep -r "selectedTimeframes" app/ --include="*.js" --include="*.jsx"
```

**Step 2: Copy Core State Management**
```javascript
// Always include these state hooks
const [selectedTimeframes, setSelectedTimeframes] = useState(['1h', '4h']);
const [balance, setBalance] = useState({ balance: 0, collateral: 0 });

// Essential toggle function
const toggleTimeframe = (tf) => {
  setSelectedTimeframes(prev => 
    prev.includes(tf) ? prev.filter(t => t !== tf) : [...prev, tf]
  );
};
```

**Step 3: Copy UI Components**
- Timeframe checkbox grid
- Preset buttons (Scalping, Day Trading, Swing Trading)
- Auto-sizing position calculator
- Formatted balance display

**Step 4: Avoid Docker Issues**
- Create new file instead of editing existing if volume mount issues persist
- Use fresh filename like `page-v2.js` or `automation-v2/page.js`
- Test in container before committing

### API Route Structure
All core functionality exposed via Next.js API routes:
```typescript
// Enhanced screenshot with progress tracking and robust cleanup
POST /api/enhanced-screenshot
{
  symbol: "SOLUSD", 
  timeframe: "240", 
  layouts: ["ai", "diy"],
  analyze: true
}
// Returns: { screenshots, analysis, sessionId }
// Note: Includes automatic Chromium process cleanup via finally blocks

// Drift trading endpoints
GET /api/balance          # Account balance/collateral
POST /api/trading         # Execute trades
GET /api/status          # Trading status
```

**API Development Tips:**
- All browser automation APIs include guaranteed cleanup via finally blocks
- Use session tracking for long-running operations
- Test API endpoints directly with curl before UI integration
- Monitor Chromium processes during API testing: `pgrep -f chrome | wc -l`

### Progress Tracking System
Real-time operation tracking for long-running tasks:
- `lib/progress-tracker.ts` manages EventEmitter-based progress
- SessionId-based tracking for multi-step operations
- Steps: init → auth → navigation → loading → capture → analysis
- Stream endpoint: `/api/progress/[sessionId]/stream`

### Browser Process Management & Cleanup System
**Critical Issue**: Chromium processes accumulate during automated trading, consuming system resources over time.

**Robust Cleanup Implementation:**
The trading bot includes a comprehensive cleanup system to prevent Chromium process accumulation:

**Core Cleanup Components:**
1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`)
   - Guaranteed cleanup via `finally` blocks in all browser operations
   - Active session tracking to prevent orphaned browsers
   - Session cleanup tasks array for systematic teardown

2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`)
   - Background monitoring service for orphaned processes
   - Multiple kill strategies: graceful → force → system cleanup
   - Periodic cleanup of temporary files and browser data

3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`)
   - System-level process killing for stubborn Chromium processes
   - Port cleanup and temporary directory management
   - Emergency cleanup functions for resource recovery

**Implementation Patterns:**
```typescript
// Always use finally blocks for guaranteed cleanup
try {
  const browser = await puppeteer.launch(options);
  // ... browser operations
} finally {
  // Guaranteed cleanup regardless of success/failure
  await ensureBrowserCleanup(browser, sessionId);
  await cleanupSessionTasks(sessionId);
}

// Background monitoring for long-running operations
const cleanupService = new AutomatedCleanupService();
cleanupService.startPeriodicCleanup(); // Every 10 minutes
```

**Cleanup Testing:**
```bash
# Test cleanup system functionality
node test-cleanup-system.js

# Monitor Chromium processes during automation
watch 'pgrep -f "chrome|chromium" | wc -l'

# Manual cleanup if needed
node -e "require('./lib/aggressive-cleanup.ts').performAggressiveCleanup()"
```

**Prevention Strategies:**
- Use session tracking for all browser instances
- Implement timeout protection for long-running operations
- Monitor resource usage during extended automation cycles
- Restart containers periodically for fresh environment
Critical timeframe handling to avoid TradingView confusion:
```typescript
// ALWAYS use minute values first, then alternatives
'4h': ['240', '240m', '4h', '4H'] // 240 minutes FIRST
'1h': ['60', '60m', '1h', '1H']   // 60 minutes FIRST
'15m': ['15', '15m']
```

Layout URL mappings for direct navigation:
```typescript
const LAYOUT_URLS = {
  'ai': 'Z1TzpUrf',    // RSI + EMAs + MACD
  'diy': 'vWVvjLhP'    // Stochastic RSI + VWAP + OBV
}
```

### Component Architecture
- `app/layout.js` - Root layout with gradient styling and navigation
- `components/Navigation.tsx` - Multi-page navigation system
- `components/AIAnalysisPanel.tsx` - Multi-timeframe analysis interface
- `components/Dashboard.tsx` - Main trading dashboard with real Drift positions
- `components/AdvancedTradingPanel.tsx` - Drift Protocol trading interface

### Cleanup System Architecture
**Critical Production Issue**: Chromium processes accumulate during automated trading, leading to resource exhaustion after several hours of operation.

**Solution Components:**
1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`)
   - Replaces original screenshot service with guaranteed cleanup
   - Uses `finally` blocks to ensure browser cleanup regardless of success/failure
   - Active session tracking with cleanup task arrays
   - Force-kill functionality for stubborn processes

2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`)
   - Background monitoring service that runs every 10 minutes
   - Multiple cleanup strategies: graceful → force → system-level cleanup
   - Temporary file cleanup and browser data directory management
   - Orphaned process detection and elimination

3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`)
   - Emergency cleanup functions for critical resource recovery
   - System-level process management with multiple kill strategies
   - Port cleanup and zombie process elimination
   - Used by both automated service and manual intervention

**Integration Pattern:**
```typescript
// In API routes - always use finally blocks
app/api/enhanced-screenshot/route.js:
try {
  const result = await enhancedScreenshot.captureAndAnalyze(...);
  return NextResponse.json(result);
} finally {
  // Guaranteed cleanup execution
  await enhancedScreenshot.cleanup();
}

// Background monitoring
lib/automated-cleanup-service.ts:
setInterval(async () => {
  await this.performCleanup();
}, 10 * 60 * 1000); // Every 10 minutes
```

**Testing Cleanup System:**
```bash
# Monitor process count during operation
watch 'pgrep -f "chrome|chromium" | wc -l'

# Test cleanup functionality
node test-cleanup-system.js

# Manual cleanup if needed
docker compose exec app node -e "require('./lib/aggressive-cleanup.ts').forceKillAllChromium()"
```

### Page Structure & Multi-Timeframe Implementation
- `app/analysis/page.js` - Original analysis page with multi-timeframe functionality
- `app/automation/page.js` - Original automation page (legacy, may have issues)
- `app/automation-v2/page.js` - **NEW**: Clean automation page with full multi-timeframe support
- `app/automation/page-v2.js` - Alternative implementation, same functionality as automation-v2

**Multi-Timeframe Architecture Pattern:**
```javascript
// Standard timeframes array - use this exact format
const timeframes = ['5m', '15m', '30m', '1h', '2h', '4h', '1d'];

// State management for multi-timeframe selection
const [selectedTimeframes, setSelectedTimeframes] = useState(['1h', '4h']);

// Toggle function with proper array handling
const toggleTimeframe = (tf) => {
  setSelectedTimeframes(prev => 
    prev.includes(tf) 
      ? prev.filter(t => t !== tf)  // Remove if selected
      : [...prev, tf]                // Add if not selected
  );
};

// Preset configurations for trading styles
const presets = {
  scalping: ['5m', '15m', '1h'],
  day: ['1h', '4h', '1d'],
  swing: ['4h', '1d']
};
```

**UI Pattern for Timeframe Selection:**
```jsx
// Checkbox grid layout with visual feedback
<div className="grid grid-cols-4 gap-2 mb-4">
  {timeframes.map(tf => (
    <button
      key={tf}
      onClick={() => toggleTimeframe(tf)}
      className={`p-2 rounded border transition-all ${
        selectedTimeframes.includes(tf)
          ? 'bg-blue-600 border-blue-500 text-white'
          : 'bg-gray-700 border-gray-600 text-gray-300 hover:bg-gray-600'
      }`}
    >
      {tf}
    </button>
  ))}
</div>

// Preset buttons for quick selection
<div className="flex gap-2 mb-4">
  {Object.entries(presets).map(([name, tfs]) => (
    <button
      key={name}
      onClick={() => setSelectedTimeframes(tfs)}
      className="px-3 py-1 bg-purple-600 hover:bg-purple-700 rounded text-sm"
    >
      {name.charAt(0).toUpperCase() + name.slice(1)}
    </button>
  ))}
</div>
```

## Environment Variables
```bash
# AI Analysis (Required)
OPENAI_API_KEY=sk-...     # OpenAI API key for chart analysis

# TradingView Automation (Required)
TRADINGVIEW_EMAIL=        # TradingView account email
TRADINGVIEW_PASSWORD=     # TradingView account password

# Trading Integration (Optional)
SOLANA_RPC_URL=https://api.mainnet-beta.solana.com
DRIFT_PRIVATE_KEY=        # Base58 encoded Solana private key
SOLANA_PRIVATE_KEY=       # JSON array format for Jupiter DEX

# Docker Environment Detection
DOCKER_ENV=true           # Auto-set in containers
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium
```

## Testing & Debugging Workflow
Test files follow specific patterns - use them to validate changes:
```bash
# Test dual-session screenshot capture
node test-enhanced-screenshot.js

# Test robust cleanup system
node test-cleanup-system.js

# Test Docker environment (requires Docker Compose v2)
./test-docker-comprehensive.sh

# Test API endpoints directly
node test-analysis-api.js

# Test Drift trading integration
node test-drift-trading.js

# Monitor resource usage during automation
watch 'pgrep -f "chrome|chromium" | wc -l'
```

**Container-First Development Workflow:**
```bash
# 1. Start development container
npm run docker:dev  # Port 9001

# 2. For immediate testing, edit directly in container
docker compose -f docker-compose.dev.yml exec app bash
# Edit files in /app/ directory for instant results

# 3. Test changes in real-time (hot reload active)
curl http://localhost:9001/api/enhanced-screenshot

# 4. Once working, commit progress to git
git add .
git commit -m "feat: implement [feature] - tested and working"
git push origin development

# 5. Rebuild container for persistence
docker compose -f docker-compose.dev.yml down
docker compose -f docker-compose.dev.yml up --build -d

# 6. Validate persistent changes and final commit
curl http://localhost:9001  # Should show updated functionality
git add . && git commit -m "chore: confirm persistence after container rebuild" && git push
```

Browser automation debugging:
- Screenshots automatically saved to `screenshots/` with timestamps
- Debug screenshots: `takeDebugScreenshot('prefix')` 
- Session persistence prevents repeated logins/captchas
- Use `npm run docker:logs` to view real-time automation logs
- All Docker commands use v2 syntax: `docker compose` (not `docker-compose`)
- Monitor Chromium processes: `docker compose exec app pgrep -f chrome`

## Code Style & Architecture Patterns
- **Client Components**: Use `"use client"` for state/effects, server components by default
- **Styling**: Tailwind with gradient backgrounds (`bg-gradient-to-br from-gray-900 via-blue-900 to-purple-900`)
- **Error Handling**: Detailed logging for browser automation with fallbacks
- **File Structure**: Mixed `.js`/`.tsx` - components in TypeScript, API routes in JavaScript
- **Database**: Prisma with SQLite (`DATABASE_URL=file:./prisma/dev.db`)

## Key Integration Points
- **Session Persistence**: `.tradingview-session/` directory volume-mounted
- **Screenshots**: `screenshots/` directory for chart captures
- **Progress Tracking**: EventEmitter-based real-time updates via SSE
- **Multi-Stage Docker**: Development vs production builds with browser optimization
- **CAPTCHA Handling**: Manual CAPTCHA mode with X11 forwarding (`ALLOW_MANUAL_CAPTCHA=true`)
- **Process Management**: Robust cleanup system prevents Chromium accumulation
- **Container Development**: Direct in-container editing for immediate testing, rebuild for persistence

## Development vs Production Modes
- **Development**: Port 9001:3000, hot reload, debug logging, headless: false option
- **Production**: Port 9000:3000, optimized build, minimal logging, always headless

**Development Container Features:**
- **Hot Reload**: File changes reflect immediately (when volume mounts work)
- **Process Monitoring**: Real-time Chromium process tracking
- **Debug Access**: Shell access via `docker compose exec app bash`
- **Immediate Testing**: Edit files in `/app/` directory for instant results
- **Resource Cleanup**: Automated cleanup services running in background

### Git Branch Strategy (Required)
**Primary development workflow:**
- **`development` branch**: Use for all active development and feature work
- **`main` branch**: Stable, production-ready code only
- **Workflow**: Develop on `development` → test thoroughly → commit progress → merge to `main` when stable

```bash
# Standard development workflow with frequent commits
git checkout development        # Always start here
git pull origin development     # Get latest changes

# Make your changes and test in container...

# Commit working progress BEFORE rebuilding container
git add .
git commit -m "feat: [specific achievement] - tested and working"
git push origin development

# After successful container rebuild and validation
git add .
git commit -m "chore: confirm [feature] persistence after rebuild"
git push origin development

# Only merge to main when features are stable and tested and you have asked the user to merge to main
git checkout main
git merge development          # When ready for production
git push origin main
```

**Git Commit Best Practices:**
- **Commit Early**: Save working progress before container rebuilds
- **Commit Often**: After each successful test or implementation step
- **Descriptive Messages**: Include what was accomplished and tested
- **Final Commits**: Always commit after confirming container persistence

**Container Persistence & Git Strategy:**
- Git changes only persist after container rebuild with `--build` flag
- **CRITICAL**: Commit working changes BEFORE rebuilding container
- Test changes in container first, then commit and rebuild
- Use descriptive commit messages for cleanup system improvements
- Example commits from robust cleanup implementation:
  - `feat: implement robust cleanup system with finally blocks - tested in container`
  - `fix: restore automation-v2 page with balance slider - confirmed working`
  - `chore: confirm cleanup system persistence after container rebuild`
  - `docs: update instructions with container development workflow`

When working with this codebase, prioritize Docker consistency, understand the dual-session architecture, leverage the comprehensive test suite to validate changes, and always implement proper cleanup patterns for browser automation to prevent resource exhaustion.

## Common Issues & Troubleshooting

### Chromium Process Accumulation
**Symptoms**: System becomes slow after hours of automation, high CPU/memory usage, many chrome processes running
**Diagnosis**: `pgrep -f "chrome|chromium" | wc -l` shows increasing process count
**Solutions**:
1. Ensure all browser automation uses finally blocks for cleanup
2. Restart automated cleanup service: `docker compose exec app node -e "require('./lib/automated-cleanup-service.ts').startPeriodicCleanup()"`
3. Manual cleanup: `docker compose exec app node test-cleanup-system.js`
4. Container restart: `docker compose restart app`

### Volume Mount Sync Issues
**Symptoms**: File changes not reflecting in running container
**Diagnosis**: Edit test file and check if visible in container
**Solutions**:
1. **Quick Fix**: Edit directly in container for immediate testing
2. **Commit Progress**: Save working changes to git before rebuilding
3. **Persistence**: Always rebuild container after confirming changes work
4. **Final Commit**: Validate and commit after successful rebuild
5. **Manual Copy**: Use `docker cp` to transfer working files
6. **Fresh Start**: Create new files instead of editing problematic ones

### Automation Page Restoration
**Symptoms**: Automation page shows old version after container rebuild
**Issue**: Git history not properly maintained in container
**Solutions**:
1. Check current branch: `git branch`
2. Restore from git: `git checkout HEAD -- app/automation-v2/page.js`
3. Verify features: Check for balance slider and multi-timeframe selection
4. Commit restoration: `git add . && git commit -m "fix: restore automation-v2 functionality" && git push`
5. Rebuild container to persist restoration

### Successful Implementation Workflow
**After completing any feature or fix:**
```bash
# 1. Test functionality thoroughly
curl http://localhost:9001/api/test-endpoint

# 2. Commit successful implementation
git add .
git commit -m "feat: [specific achievement] - fully tested and working"
git push origin development

# 3. Rebuild container for persistence  
docker compose down && docker compose up --build -d

# 4. Final validation and completion commit
curl http://localhost:9001  # Verify persistent functionality
git add . && git commit -m "chore: confirm [feature] persistence - implementation complete" && git push

# 5. Consider merge to main if ready for production
# (Ask user first before merging to main branch)
```

### API Endpoint Not Responding
**Symptoms**: API calls timeout or return errors
**Diagnosis**: Check container logs: `docker compose logs -f app`
**Solutions**:
1. Verify container is running: `docker ps`
2. Check port mapping: Development=9001:3000, Production=9000:3000
3. Test direct access: `curl localhost:9001/api/status`
4. Restart if needed: `docker compose restart app`