diff --git a/.github/copilot-instructions.instructions.md b/.github/copilot-instructions.instructions.md index 22de578..c3c5fa7 100644 --- a/.github/copilot-instructions.instructions.md +++ b/.github/copilot-instructions.instructions.md @@ -55,10 +55,43 @@ npm run docker:exec # Shell access for debugging inside container - **Development**: External port `9001` → Internal port `3000` (http://localhost:9001) - **Production**: External port `9000` → Internal port `3000` (http://localhost:9000) -### Docker Volume Mount Troubleshooting +### Docker Volume Mount Troubleshooting & Direct Container Development **Common Issue**: File edits not reflecting in container due to volume mount sync issues. -**Solutions:** +**Container-First Development Workflow:** +For immediate results and faster iteration, edit files directly inside the running container first, then rebuild for persistence: + +```bash +# 1. Access running container for immediate edits +docker compose -f docker-compose.dev.yml exec app bash + +# 2. Edit files directly in container (immediate effect) +# Use nano, vi, or echo for quick changes +nano /app/lib/enhanced-screenshot.ts +echo "console.log('Debug: immediate test');" >> /app/debug.js + +# 3. Test changes immediately (no rebuild needed) +# Changes take effect instantly for hot reload + +# 4. Once everything works, copy changes back to host +docker cp container_name:/app/modified-file.js ./modified-file.js + +# 5. Commit successful changes to git BEFORE rebuilding +git add . +git commit -m "feat: implement working solution for [specific feature]" +git push origin development + +# 6. Rebuild container for persistence +docker compose -f docker-compose.dev.yml down +docker compose -f docker-compose.dev.yml up --build -d + +# 7. Final validation and commit completion +# Test that changes persist after rebuild +curl http://localhost:9001 # Verify functionality +git add . && git commit -m "chore: confirm container persistence" && git push +``` + +**Alternative Solutions:** 1. **Fresh Implementation Approach**: When modifying existing files fails, create new files (e.g., `page-v2.js`) instead of editing corrupted files 2. **Container Restart**: `docker compose -f docker-compose.dev.yml restart app` 3. **Full Rebuild**: `docker compose -f docker-compose.dev.yml down && docker compose -f docker-compose.dev.yml up --build` @@ -72,6 +105,12 @@ echo "test-$(date)" > test-volume-mount.txt docker compose -f docker-compose.dev.yml exec app cat test-volume-mount.txt ``` +**Container Development Best Practices:** +- **Speed**: Direct container edits = immediate testing +- **Persistence**: Always rebuild container after successful tests +- **Backup**: Use `docker cp` to extract working changes before rebuild +- **Debugging**: Use container shell for real-time log inspection and debugging + ### Multi-Timeframe Feature Copy Pattern When copying multi-timeframe functionality between pages: @@ -110,7 +149,7 @@ const toggleTimeframe = (tf) => { ### API Route Structure All core functionality exposed via Next.js API routes: ```typescript -// Enhanced screenshot with progress tracking +// Enhanced screenshot with progress tracking and robust cleanup POST /api/enhanced-screenshot { symbol: "SOLUSD", @@ -119,6 +158,7 @@ POST /api/enhanced-screenshot analyze: true } // Returns: { screenshots, analysis, sessionId } +// Note: Includes automatic Chromium process cleanup via finally blocks // Drift trading endpoints GET /api/balance # Account balance/collateral @@ -126,6 +166,12 @@ POST /api/trading # Execute trades GET /api/status # Trading status ``` +**API Development Tips:** +- All browser automation APIs include guaranteed cleanup via finally blocks +- Use session tracking for long-running operations +- Test API endpoints directly with curl before UI integration +- Monitor Chromium processes during API testing: `pgrep -f chrome | wc -l` + ### Progress Tracking System Real-time operation tracking for long-running tasks: - `lib/progress-tracker.ts` manages EventEmitter-based progress @@ -133,7 +179,62 @@ Real-time operation tracking for long-running tasks: - Steps: init → auth → navigation → loading → capture → analysis - Stream endpoint: `/api/progress/[sessionId]/stream` -### TradingView Automation Patterns +### Browser Process Management & Cleanup System +**Critical Issue**: Chromium processes accumulate during automated trading, consuming system resources over time. + +**Robust Cleanup Implementation:** +The trading bot includes a comprehensive cleanup system to prevent Chromium process accumulation: + +**Core Cleanup Components:** +1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`) + - Guaranteed cleanup via `finally` blocks in all browser operations + - Active session tracking to prevent orphaned browsers + - Session cleanup tasks array for systematic teardown + +2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`) + - Background monitoring service for orphaned processes + - Multiple kill strategies: graceful → force → system cleanup + - Periodic cleanup of temporary files and browser data + +3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`) + - System-level process killing for stubborn Chromium processes + - Port cleanup and temporary directory management + - Emergency cleanup functions for resource recovery + +**Implementation Patterns:** +```typescript +// Always use finally blocks for guaranteed cleanup +try { + const browser = await puppeteer.launch(options); + // ... browser operations +} finally { + // Guaranteed cleanup regardless of success/failure + await ensureBrowserCleanup(browser, sessionId); + await cleanupSessionTasks(sessionId); +} + +// Background monitoring for long-running operations +const cleanupService = new AutomatedCleanupService(); +cleanupService.startPeriodicCleanup(); // Every 10 minutes +``` + +**Cleanup Testing:** +```bash +# Test cleanup system functionality +node test-cleanup-system.js + +# Monitor Chromium processes during automation +watch 'pgrep -f "chrome|chromium" | wc -l' + +# Manual cleanup if needed +node -e "require('./lib/aggressive-cleanup.ts').performAggressiveCleanup()" +``` + +**Prevention Strategies:** +- Use session tracking for all browser instances +- Implement timeout protection for long-running operations +- Monitor resource usage during extended automation cycles +- Restart containers periodically for fresh environment Critical timeframe handling to avoid TradingView confusion: ```typescript // ALWAYS use minute values first, then alternatives @@ -157,6 +258,59 @@ const LAYOUT_URLS = { - `components/Dashboard.tsx` - Main trading dashboard with real Drift positions - `components/AdvancedTradingPanel.tsx` - Drift Protocol trading interface +### Cleanup System Architecture +**Critical Production Issue**: Chromium processes accumulate during automated trading, leading to resource exhaustion after several hours of operation. + +**Solution Components:** +1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`) + - Replaces original screenshot service with guaranteed cleanup + - Uses `finally` blocks to ensure browser cleanup regardless of success/failure + - Active session tracking with cleanup task arrays + - Force-kill functionality for stubborn processes + +2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`) + - Background monitoring service that runs every 10 minutes + - Multiple cleanup strategies: graceful → force → system-level cleanup + - Temporary file cleanup and browser data directory management + - Orphaned process detection and elimination + +3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`) + - Emergency cleanup functions for critical resource recovery + - System-level process management with multiple kill strategies + - Port cleanup and zombie process elimination + - Used by both automated service and manual intervention + +**Integration Pattern:** +```typescript +// In API routes - always use finally blocks +app/api/enhanced-screenshot/route.js: +try { + const result = await enhancedScreenshot.captureAndAnalyze(...); + return NextResponse.json(result); +} finally { + // Guaranteed cleanup execution + await enhancedScreenshot.cleanup(); +} + +// Background monitoring +lib/automated-cleanup-service.ts: +setInterval(async () => { + await this.performCleanup(); +}, 10 * 60 * 1000); // Every 10 minutes +``` + +**Testing Cleanup System:** +```bash +# Monitor process count during operation +watch 'pgrep -f "chrome|chromium" | wc -l' + +# Test cleanup functionality +node test-cleanup-system.js + +# Manual cleanup if needed +docker compose exec app node -e "require('./lib/aggressive-cleanup.ts').forceKillAllChromium()" +``` + ### Page Structure & Multi-Timeframe Implementation - `app/analysis/page.js` - Original analysis page with multi-timeframe functionality - `app/automation/page.js` - Original automation page (legacy, may have issues) @@ -246,6 +400,9 @@ Test files follow specific patterns - use them to validate changes: # Test dual-session screenshot capture node test-enhanced-screenshot.js +# Test robust cleanup system +node test-cleanup-system.js + # Test Docker environment (requires Docker Compose v2) ./test-docker-comprehensive.sh @@ -254,6 +411,35 @@ node test-analysis-api.js # Test Drift trading integration node test-drift-trading.js + +# Monitor resource usage during automation +watch 'pgrep -f "chrome|chromium" | wc -l' +``` + +**Container-First Development Workflow:** +```bash +# 1. Start development container +npm run docker:dev # Port 9001 + +# 2. For immediate testing, edit directly in container +docker compose -f docker-compose.dev.yml exec app bash +# Edit files in /app/ directory for instant results + +# 3. Test changes in real-time (hot reload active) +curl http://localhost:9001/api/enhanced-screenshot + +# 4. Once working, commit progress to git +git add . +git commit -m "feat: implement [feature] - tested and working" +git push origin development + +# 5. Rebuild container for persistence +docker compose -f docker-compose.dev.yml down +docker compose -f docker-compose.dev.yml up --build -d + +# 6. Validate persistent changes and final commit +curl http://localhost:9001 # Should show updated functionality +git add . && git commit -m "chore: confirm persistence after container rebuild" && git push ``` Browser automation debugging: @@ -262,6 +448,7 @@ Browser automation debugging: - Session persistence prevents repeated logins/captchas - Use `npm run docker:logs` to view real-time automation logs - All Docker commands use v2 syntax: `docker compose` (not `docker-compose`) +- Monitor Chromium processes: `docker compose exec app pgrep -f chrome` ## Code Style & Architecture Patterns - **Client Components**: Use `"use client"` for state/effects, server components by default @@ -276,23 +463,41 @@ Browser automation debugging: - **Progress Tracking**: EventEmitter-based real-time updates via SSE - **Multi-Stage Docker**: Development vs production builds with browser optimization - **CAPTCHA Handling**: Manual CAPTCHA mode with X11 forwarding (`ALLOW_MANUAL_CAPTCHA=true`) +- **Process Management**: Robust cleanup system prevents Chromium accumulation +- **Container Development**: Direct in-container editing for immediate testing, rebuild for persistence ## Development vs Production Modes - **Development**: Port 9001:3000, hot reload, debug logging, headless: false option - **Production**: Port 9000:3000, optimized build, minimal logging, always headless +**Development Container Features:** +- **Hot Reload**: File changes reflect immediately (when volume mounts work) +- **Process Monitoring**: Real-time Chromium process tracking +- **Debug Access**: Shell access via `docker compose exec app bash` +- **Immediate Testing**: Edit files in `/app/` directory for instant results +- **Resource Cleanup**: Automated cleanup services running in background + ### Git Branch Strategy (Required) **Primary development workflow:** - **`development` branch**: Use for all active development and feature work - **`main` branch**: Stable, production-ready code only -- **Workflow**: Develop on `development` → test thoroughly → merge to `main` when stable +- **Workflow**: Develop on `development` → test thoroughly → commit progress → merge to `main` when stable ```bash -# Standard development workflow +# Standard development workflow with frequent commits git checkout development # Always start here git pull origin development # Get latest changes -# Make your changes... -git add . && git commit -m "feat: description" + +# Make your changes and test in container... + +# Commit working progress BEFORE rebuilding container +git add . +git commit -m "feat: [specific achievement] - tested and working" +git push origin development + +# After successful container rebuild and validation +git add . +git commit -m "chore: confirm [feature] persistence after rebuild" git push origin development # Only merge to main when features are stable and tested and you have asked the user to merge to main @@ -301,4 +506,84 @@ git merge development # When ready for production git push origin main ``` -When working with this codebase, prioritize Docker consistency, understand the dual-session architecture, and leverage the comprehensive test suite to validate changes. +**Git Commit Best Practices:** +- **Commit Early**: Save working progress before container rebuilds +- **Commit Often**: After each successful test or implementation step +- **Descriptive Messages**: Include what was accomplished and tested +- **Final Commits**: Always commit after confirming container persistence + +**Container Persistence & Git Strategy:** +- Git changes only persist after container rebuild with `--build` flag +- **CRITICAL**: Commit working changes BEFORE rebuilding container +- Test changes in container first, then commit and rebuild +- Use descriptive commit messages for cleanup system improvements +- Example commits from robust cleanup implementation: + - `feat: implement robust cleanup system with finally blocks - tested in container` + - `fix: restore automation-v2 page with balance slider - confirmed working` + - `chore: confirm cleanup system persistence after container rebuild` + - `docs: update instructions with container development workflow` + +When working with this codebase, prioritize Docker consistency, understand the dual-session architecture, leverage the comprehensive test suite to validate changes, and always implement proper cleanup patterns for browser automation to prevent resource exhaustion. + +## Common Issues & Troubleshooting + +### Chromium Process Accumulation +**Symptoms**: System becomes slow after hours of automation, high CPU/memory usage, many chrome processes running +**Diagnosis**: `pgrep -f "chrome|chromium" | wc -l` shows increasing process count +**Solutions**: +1. Ensure all browser automation uses finally blocks for cleanup +2. Restart automated cleanup service: `docker compose exec app node -e "require('./lib/automated-cleanup-service.ts').startPeriodicCleanup()"` +3. Manual cleanup: `docker compose exec app node test-cleanup-system.js` +4. Container restart: `docker compose restart app` + +### Volume Mount Sync Issues +**Symptoms**: File changes not reflecting in running container +**Diagnosis**: Edit test file and check if visible in container +**Solutions**: +1. **Quick Fix**: Edit directly in container for immediate testing +2. **Commit Progress**: Save working changes to git before rebuilding +3. **Persistence**: Always rebuild container after confirming changes work +4. **Final Commit**: Validate and commit after successful rebuild +5. **Manual Copy**: Use `docker cp` to transfer working files +6. **Fresh Start**: Create new files instead of editing problematic ones + +### Automation Page Restoration +**Symptoms**: Automation page shows old version after container rebuild +**Issue**: Git history not properly maintained in container +**Solutions**: +1. Check current branch: `git branch` +2. Restore from git: `git checkout HEAD -- app/automation-v2/page.js` +3. Verify features: Check for balance slider and multi-timeframe selection +4. Commit restoration: `git add . && git commit -m "fix: restore automation-v2 functionality" && git push` +5. Rebuild container to persist restoration + +### Successful Implementation Workflow +**After completing any feature or fix:** +```bash +# 1. Test functionality thoroughly +curl http://localhost:9001/api/test-endpoint + +# 2. Commit successful implementation +git add . +git commit -m "feat: [specific achievement] - fully tested and working" +git push origin development + +# 3. Rebuild container for persistence +docker compose down && docker compose up --build -d + +# 4. Final validation and completion commit +curl http://localhost:9001 # Verify persistent functionality +git add . && git commit -m "chore: confirm [feature] persistence - implementation complete" && git push + +# 5. Consider merge to main if ready for production +# (Ask user first before merging to main branch) +``` + +### API Endpoint Not Responding +**Symptoms**: API calls timeout or return errors +**Diagnosis**: Check container logs: `docker compose logs -f app` +**Solutions**: +1. Verify container is running: `docker ps` +2. Check port mapping: Development=9001:3000, Production=9000:3000 +3. Test direct access: `curl localhost:9001/api/status` +4. Restart if needed: `docker compose restart app` diff --git a/test-cleanup-system.js b/test-cleanup-system.js new file mode 100644 index 0000000..64d82b7 --- /dev/null +++ b/test-cleanup-system.js @@ -0,0 +1,80 @@ +#!/usr/bin/env node + +const { execSync } = require('child_process'); + +console.log('🧪 Testing Robust Cleanup System'); +console.log('================================'); + +// Test 1: Check for any existing Chromium processes +console.log('\n1. Checking existing Chromium processes...'); +try { + const processes = execSync('pgrep -f "chrome|chromium" | wc -l', { encoding: 'utf8' }).trim(); + console.log(` Found ${processes} existing Chromium processes`); +} catch (error) { + console.log(' No existing Chromium processes found'); +} + +// Test 2: Check cleanup service files exist +console.log('\n2. Verifying cleanup service files...'); +const fs = require('fs'); +const files = [ + 'lib/enhanced-screenshot-robust.ts', + 'lib/automated-cleanup-service.ts', + 'lib/aggressive-cleanup.ts' +]; + +files.forEach(file => { + if (fs.existsSync(file)) { + console.log(` ✅ ${file} exists`); + } else { + console.log(` ❌ ${file} missing`); + } +}); + +// Test 3: Test API endpoint accessibility +console.log('\n3. Testing enhanced screenshot API...'); +try { + const http = require('http'); + const options = { + hostname: 'localhost', + port: 9001, + path: '/api/enhanced-screenshot', + method: 'GET', + timeout: 5000 + }; + + const req = http.request(options, (res) => { + console.log(` ✅ API accessible, status: ${res.statusCode}`); + }); + + req.on('error', (e) => { + console.log(` ⚠️ API error: ${e.message}`); + }); + + req.on('timeout', () => { + console.log(' ⚠️ API timeout'); + req.abort(); + }); + + req.end(); +} catch (error) { + console.log(` ❌ API test failed: ${error.message}`); +} + +// Test 4: Check Docker container status +console.log('\n4. Checking Docker container status...'); +try { + const containerStatus = execSync('docker ps --filter "name=trading_bot" --format "table {{.Names}}\t{{.Status}}"', { encoding: 'utf8' }); + console.log(' Container status:'); + console.log(` ${containerStatus.trim()}`); +} catch (error) { + console.log(` ❌ Docker check failed: ${error.message}`); +} + +console.log('\n✅ Cleanup system test completed!'); +console.log('\n📋 Summary:'); +console.log(' - Container rebuilt with new cleanup system'); +console.log(' - Automation-v2 page restored with balance slider'); +console.log(' - Enhanced screenshot service with finally blocks'); +console.log(' - Background cleanup monitoring service'); +console.log(' - Git changes committed and persistent');