root/trading_bot_v3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: consolidate copilot instructions into single comprehensive file

Browse Source 
- Merged duplicate .github/copilot-instructions.instructions.md into main copilot-instructions.md
- Combined development patterns, architecture details, and AI learning system docs
- Added comprehensive references to all technical documentation files
- Single source of truth for GitHub Copilot development guidance
- Includes Docker workflow, cleanup systems, error handling patterns
This commit is contained in:
mindesbunister
2025-07-26 15:19:36 +02:00
parent 71694ca660
commit 11aec95d47
2 changed files with 320 additions and 593 deletions
Download Patch File Download Diff File Expand all files Collapse all files

589
.github/copilot-instructions.instructions.md vendored
View File

@@ -1,589 +0,0 @@
# AI-Powered Trading Bot Dashboard
This is a Next.js 15 App Router application with TypeScript, Tailwind CSS, and API routes. It's a production-ready trading bot with AI analysis, automated screenshot capture, and real-time trading execution via Drift Protocol and Jupiter DEX.
**Prerequisites:**
- Docker and Docker Compose v2 (uses `docker compose` command syntax)
- All development must be done inside Docker containers for browser automation compatibility
## Core Architecture
### Dual-Session Screenshot Automation
- **AI Layout**: `Z1TzpUrf` - RSI (top), EMAs, MACD (bottom)
- **DIY Layout**: `vWVvjLhP` - Stochastic RSI (top), VWAP, OBV (bottom)
- Parallel browser sessions for multi-layout capture in `lib/enhanced-screenshot.ts`
- TradingView automation with session persistence in `lib/tradingview-automation.ts`
- Session data stored in `.tradingview-session/` volume mount to avoid captchas
### AI Analysis Pipeline
- OpenAI GPT-4o mini for cost-effective chart analysis (~$0.006 per analysis)
- Multi-layout comparison and consensus detection in `lib/ai-analysis.ts`
- Professional trading setups with exact entry/exit levels and risk management
- Layout-specific indicator analysis (RSI vs Stochastic RSI, MACD vs OBV)
### Trading Integration
- **Drift Protocol**: Perpetual futures trading via `@drift-labs/sdk`
- **Jupiter DEX**: Spot trading on Solana
- Position management and P&L tracking in `lib/drift-trading-final.ts`
- Real-time account balance and collateral monitoring
## Critical Development Patterns
### Docker Container Development (Required)
**All development happens inside Docker containers** using Docker Compose v2. Browser automation requires specific system dependencies that are only available in the containerized environment:
**IMPORTANT: Use Docker Compose v2 syntax** - All commands use `docker compose` (with space) instead of `docker-compose` (with hyphen).
```bash
# Development environment - Docker Compose v2 dev setup
npm run docker:dev # Port 9001:3000, hot reload, debug mode
# Direct v2 command: docker compose -f docker-compose.dev.yml up --build
# Production environment
npm run docker:up # Port 9000:3000, optimized build
# Direct v2 command: docker compose -f docker-compose.prod.yml up --build
# Debugging commands
npm run docker:logs # View container logs
# Direct v2 command: docker compose -f docker-compose.dev.yml logs -f
npm run docker:exec # Shell access for debugging inside container
# Direct v2 command: docker compose -f docker-compose.dev.yml exec app bash
```
**Port Configuration:**
- **Development**: External port `9001` → Internal port `3000` (http://localhost:9001)
- **Production**: External port `9000` → Internal port `3000` (http://localhost:9000)
### Docker Volume Mount Troubleshooting & Direct Container Development
**Common Issue**: File edits not reflecting in container due to volume mount sync issues.
**Container-First Development Workflow:**
For immediate results and faster iteration, edit files directly inside the running container first, then rebuild for persistence:
```bash
# 1. Access running container for immediate edits
docker compose -f docker-compose.dev.yml exec app bash
# 2. Edit files directly in container (immediate effect)
# Use nano, vi, or echo for quick changes
nano /app/lib/enhanced-screenshot.ts
echo "console.log('Debug: immediate test');" >> /app/debug.js
# 3. Test changes immediately (no rebuild needed)
# Changes take effect instantly for hot reload
# 4. Once everything works, copy changes back to host
docker cp container_name:/app/modified-file.js ./modified-file.js
# 5. Commit successful changes to git BEFORE rebuilding
git add .
git commit -m "feat: implement working solution for [specific feature]"
git push origin development
# 6. Rebuild container for persistence
docker compose -f docker-compose.dev.yml down
docker compose -f docker-compose.dev.yml up --build -d
# 7. Final validation and commit completion
# Test that changes persist after rebuild
curl http://localhost:9001 # Verify functionality
git add . && git commit -m "chore: confirm container persistence" && git push
```
**Alternative Solutions:**
1. **Fresh Implementation Approach**: When modifying existing files fails, create new files (e.g., `page-v2.js`) instead of editing corrupted files
2. **Container Restart**: `docker compose -f docker-compose.dev.yml restart app`
3. **Full Rebuild**: `docker compose -f docker-compose.dev.yml down && docker compose -f docker-compose.dev.yml up --build`
4. **Manual Copy**: Use `docker cp` to copy files directly into container for immediate testing
5. **Avoid sed/awk**: Direct text manipulation commands often corrupt JSX syntax - prefer file replacement
**Volume Mount Verification:**
```bash
# Test volume mount sync
echo "test-$(date)" > test-volume-mount.txt
docker compose -f docker-compose.dev.yml exec app cat test-volume-mount.txt
```
**Container Development Best Practices:**
- **Speed**: Direct container edits = immediate testing
- **Persistence**: Always rebuild container after successful tests
- **Backup**: Use `docker cp` to extract working changes before rebuild
- **Debugging**: Use container shell for real-time log inspection and debugging
### Multi-Timeframe Feature Copy Pattern
When copying multi-timeframe functionality between pages:
**Step 1: Identify Source Implementation**
```bash
# Search for existing timeframe patterns
grep -r "timeframes.*=.*\[" app/ --include="*.js" --include="*.jsx"
grep -r "selectedTimeframes" app/ --include="*.js" --include="*.jsx"
```
**Step 2: Copy Core State Management**
```javascript
// Always include these state hooks
const [selectedTimeframes, setSelectedTimeframes] = useState(['1h', '4h']);
const [balance, setBalance] = useState({ balance: 0, collateral: 0 });
// Essential toggle function
const toggleTimeframe = (tf) => {
setSelectedTimeframes(prev =>
prev.includes(tf) ? prev.filter(t => t !== tf) : [...prev, tf]
);
};
```
**Step 3: Copy UI Components**
- Timeframe checkbox grid
- Preset buttons (Scalping, Day Trading, Swing Trading)
- Auto-sizing position calculator
- Formatted balance display
**Step 4: Avoid Docker Issues**
- Create new file instead of editing existing if volume mount issues persist
- Use fresh filename like `page-v2.js` or `automation-v2/page.js`
- Test in container before committing
### API Route Structure
All core functionality exposed via Next.js API routes:
```typescript
// Enhanced screenshot with progress tracking and robust cleanup
POST /api/enhanced-screenshot
{
symbol: "SOLUSD",
timeframe: "240",
layouts: ["ai", "diy"],
analyze: true
}
// Returns: { screenshots, analysis, sessionId }
// Note: Includes automatic Chromium process cleanup via finally blocks
// Drift trading endpoints
GET /api/balance # Account balance/collateral
POST /api/trading # Execute trades
GET /api/status # Trading status
```
**API Development Tips:**
- All browser automation APIs include guaranteed cleanup via finally blocks
- Use session tracking for long-running operations
- Test API endpoints directly with curl before UI integration
- Monitor Chromium processes during API testing: `pgrep -f chrome | wc -l`
### Progress Tracking System
Real-time operation tracking for long-running tasks:
- `lib/progress-tracker.ts` manages EventEmitter-based progress
- SessionId-based tracking for multi-step operations
- Steps: init → auth → navigation → loading → capture → analysis
- Stream endpoint: `/api/progress/[sessionId]/stream`
### Browser Process Management & Cleanup System
**Critical Issue**: Chromium processes accumulate during automated trading, consuming system resources over time.
**Robust Cleanup Implementation:**
The trading bot includes a comprehensive cleanup system to prevent Chromium process accumulation:
**Core Cleanup Components:**
1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`)
- Guaranteed cleanup via `finally` blocks in all browser operations
- Active session tracking to prevent orphaned browsers
- Session cleanup tasks array for systematic teardown
2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`)
- Background monitoring service for orphaned processes
- Multiple kill strategies: graceful → force → system cleanup
- Periodic cleanup of temporary files and browser data
3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`)
- System-level process killing for stubborn Chromium processes
- Port cleanup and temporary directory management
- Emergency cleanup functions for resource recovery
**Implementation Patterns:**
```typescript
// Always use finally blocks for guaranteed cleanup
try {
const browser = await puppeteer.launch(options);
// ... browser operations
} finally {
// Guaranteed cleanup regardless of success/failure
await ensureBrowserCleanup(browser, sessionId);
await cleanupSessionTasks(sessionId);
}
// Background monitoring for long-running operations
const cleanupService = new AutomatedCleanupService();
cleanupService.startPeriodicCleanup(); // Every 10 minutes
```
**Cleanup Testing:**
```bash
# Test cleanup system functionality
node test-cleanup-system.js
# Monitor Chromium processes during automation
watch 'pgrep -f "chrome|chromium" | wc -l'
# Manual cleanup if needed
node -e "require('./lib/aggressive-cleanup.ts').performAggressiveCleanup()"
```
**Prevention Strategies:**
- Use session tracking for all browser instances
- Implement timeout protection for long-running operations
- Monitor resource usage during extended automation cycles
- Restart containers periodically for fresh environment
Critical timeframe handling to avoid TradingView confusion:
```typescript
// ALWAYS use minute values first, then alternatives
'4h': ['240', '240m', '4h', '4H'] // 240 minutes FIRST
'1h': ['60', '60m', '1h', '1H'] // 60 minutes FIRST
'15m': ['15', '15m']
```
Layout URL mappings for direct navigation:
```typescript
const LAYOUT_URLS = {
'ai': 'Z1TzpUrf', // RSI + EMAs + MACD
'diy': 'vWVvjLhP' // Stochastic RSI + VWAP + OBV
}
```
### Component Architecture
- `app/layout.js` - Root layout with gradient styling and navigation
- `components/Navigation.tsx` - Multi-page navigation system
- `components/AIAnalysisPanel.tsx` - Multi-timeframe analysis interface
- `components/Dashboard.tsx` - Main trading dashboard with real Drift positions
- `components/AdvancedTradingPanel.tsx` - Drift Protocol trading interface
### Cleanup System Architecture
**Critical Production Issue**: Chromium processes accumulate during automated trading, leading to resource exhaustion after several hours of operation.
**Solution Components:**
1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`)
- Replaces original screenshot service with guaranteed cleanup
- Uses `finally` blocks to ensure browser cleanup regardless of success/failure
- Active session tracking with cleanup task arrays
- Force-kill functionality for stubborn processes
2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`)
- Background monitoring service that runs every 10 minutes
- Multiple cleanup strategies: graceful → force → system-level cleanup
- Temporary file cleanup and browser data directory management
- Orphaned process detection and elimination
3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`)
- Emergency cleanup functions for critical resource recovery
- System-level process management with multiple kill strategies
- Port cleanup and zombie process elimination
- Used by both automated service and manual intervention
**Integration Pattern:**
```typescript
// In API routes - always use finally blocks
app/api/enhanced-screenshot/route.js:
try {
const result = await enhancedScreenshot.captureAndAnalyze(...);
return NextResponse.json(result);
} finally {
// Guaranteed cleanup execution
await enhancedScreenshot.cleanup();
}
// Background monitoring
lib/automated-cleanup-service.ts:
setInterval(async () => {
await this.performCleanup();
}, 10 * 60 * 1000); // Every 10 minutes
```
**Testing Cleanup System:**
```bash
# Monitor process count during operation
watch 'pgrep -f "chrome|chromium" | wc -l'
# Test cleanup functionality
node test-cleanup-system.js
# Manual cleanup if needed
docker compose exec app node -e "require('./lib/aggressive-cleanup.ts').forceKillAllChromium()"
```
### Page Structure & Multi-Timeframe Implementation
- `app/analysis/page.js` - Original analysis page with multi-timeframe functionality
- `app/automation/page.js` - Original automation page (legacy, may have issues)
- `app/automation-v2/page.js` - **NEW**: Clean automation page with full multi-timeframe support
- `app/automation/page-v2.js` - Alternative implementation, same functionality as automation-v2
**Multi-Timeframe Architecture Pattern:**
```javascript
// Standard timeframes array - use this exact format
const timeframes = ['5m', '15m', '30m', '1h', '2h', '4h', '1d'];
// State management for multi-timeframe selection
const [selectedTimeframes, setSelectedTimeframes] = useState(['1h', '4h']);
// Toggle function with proper array handling
const toggleTimeframe = (tf) => {
setSelectedTimeframes(prev =>
prev.includes(tf)
? prev.filter(t => t !== tf) // Remove if selected
: [...prev, tf] // Add if not selected
);
};
// Preset configurations for trading styles
const presets = {
scalping: ['5m', '15m', '1h'],
day: ['1h', '4h', '1d'],
swing: ['4h', '1d']
};
```
**UI Pattern for Timeframe Selection:**
```jsx
// Checkbox grid layout with visual feedback
<div className="grid grid-cols-4 gap-2 mb-4">
{timeframes.map(tf => (
<button
key={tf}
onClick={() => toggleTimeframe(tf)}
className={`p-2 rounded border transition-all ${
selectedTimeframes.includes(tf)
? 'bg-blue-600 border-blue-500 text-white'
: 'bg-gray-700 border-gray-600 text-gray-300 hover:bg-gray-600'
}`}
>
{tf}
</button>
))}
</div>
// Preset buttons for quick selection
<div className="flex gap-2 mb-4">
{Object.entries(presets).map(([name, tfs]) => (
<button
key={name}
onClick={() => setSelectedTimeframes(tfs)}
className="px-3 py-1 bg-purple-600 hover:bg-purple-700 rounded text-sm"
>
{name.charAt(0).toUpperCase() + name.slice(1)}
</button>
))}
</div>
```
## Environment Variables
```bash
# AI Analysis (Required)
OPENAI_API_KEY=sk-... # OpenAI API key for chart analysis
# TradingView Automation (Required)
TRADINGVIEW_EMAIL= # TradingView account email
TRADINGVIEW_PASSWORD= # TradingView account password
# Trading Integration (Optional)
SOLANA_RPC_URL=https://api.mainnet-beta.solana.com
DRIFT_PRIVATE_KEY= # Base58 encoded Solana private key
SOLANA_PRIVATE_KEY= # JSON array format for Jupiter DEX
# Docker Environment Detection
DOCKER_ENV=true # Auto-set in containers
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium
```
## Testing & Debugging Workflow
Test files follow specific patterns - use them to validate changes:
```bash
# Test dual-session screenshot capture
node test-enhanced-screenshot.js
# Test robust cleanup system
node test-cleanup-system.js
# Test Docker environment (requires Docker Compose v2)
./test-docker-comprehensive.sh
# Test API endpoints directly
node test-analysis-api.js
# Test Drift trading integration
node test-drift-trading.js
# Monitor resource usage during automation
watch 'pgrep -f "chrome|chromium" | wc -l'
```
**Container-First Development Workflow:**
```bash
# 1. Start development container
npm run docker:dev # Port 9001
# 2. For immediate testing, edit directly in container
docker compose -f docker-compose.dev.yml exec app bash
# Edit files in /app/ directory for instant results
# 3. Test changes in real-time (hot reload active)
curl http://localhost:9001/api/enhanced-screenshot
# 4. Once working, commit progress to git
git add .
git commit -m "feat: implement [feature] - tested and working"
git push origin development
# 5. Rebuild container for persistence
docker compose -f docker-compose.dev.yml down
docker compose -f docker-compose.dev.yml up --build -d
# 6. Validate persistent changes and final commit
curl http://localhost:9001 # Should show updated functionality
git add . && git commit -m "chore: confirm persistence after container rebuild" && git push
```
Browser automation debugging:
- Screenshots automatically saved to `screenshots/` with timestamps
- Debug screenshots: `takeDebugScreenshot('prefix')`
- Session persistence prevents repeated logins/captchas
- Use `npm run docker:logs` to view real-time automation logs
- All Docker commands use v2 syntax: `docker compose` (not `docker-compose`)
- Monitor Chromium processes: `docker compose exec app pgrep -f chrome`
## Code Style & Architecture Patterns
- **Client Components**: Use `"use client"` for state/effects, server components by default
- **Styling**: Tailwind with gradient backgrounds (`bg-gradient-to-br from-gray-900 via-blue-900 to-purple-900`)
- **Error Handling**: Detailed logging for browser automation with fallbacks
- **File Structure**: Mixed `.js`/`.tsx` - components in TypeScript, API routes in JavaScript
- **Database**: Prisma with SQLite (`DATABASE_URL=file:./prisma/dev.db`)
## Key Integration Points
- **Session Persistence**: `.tradingview-session/` directory volume-mounted
- **Screenshots**: `screenshots/` directory for chart captures
- **Progress Tracking**: EventEmitter-based real-time updates via SSE
- **Multi-Stage Docker**: Development vs production builds with browser optimization
- **CAPTCHA Handling**: Manual CAPTCHA mode with X11 forwarding (`ALLOW_MANUAL_CAPTCHA=true`)
- **Process Management**: Robust cleanup system prevents Chromium accumulation
- **Container Development**: Direct in-container editing for immediate testing, rebuild for persistence
## Development vs Production Modes
- **Development**: Port 9001:3000, hot reload, debug logging, headless: false option
- **Production**: Port 9000:3000, optimized build, minimal logging, always headless
**Development Container Features:**
- **Hot Reload**: File changes reflect immediately (when volume mounts work)
- **Process Monitoring**: Real-time Chromium process tracking
- **Debug Access**: Shell access via `docker compose exec app bash`
- **Immediate Testing**: Edit files in `/app/` directory for instant results
- **Resource Cleanup**: Automated cleanup services running in background
### Git Branch Strategy (Required)
**Primary development workflow:**
- **`development` branch**: Use for all active development and feature work
- **`main` branch**: Stable, production-ready code only
- **Workflow**: Develop on `development` → test thoroughly → commit progress → merge to `main` when stable
```bash
# Standard development workflow with frequent commits
git checkout development # Always start here
git pull origin development # Get latest changes
# Make your changes and test in container...
# Commit working progress BEFORE rebuilding container
git add .
git commit -m "feat: [specific achievement] - tested and working"
git push origin development
# After successful container rebuild and validation
git add .
git commit -m "chore: confirm [feature] persistence after rebuild"
git push origin development
# Only merge to main when features are stable and tested and you have asked the user to merge to main
git checkout main
git merge development # When ready for production
git push origin main
```
**Git Commit Best Practices:**
- **Commit Early**: Save working progress before container rebuilds
- **Commit Often**: After each successful test or implementation step
- **Descriptive Messages**: Include what was accomplished and tested
- **Final Commits**: Always commit after confirming container persistence
**Container Persistence & Git Strategy:**
- Git changes only persist after container rebuild with `--build` flag
- **CRITICAL**: Commit working changes BEFORE rebuilding container
- Test changes in container first, then commit and rebuild
- Use descriptive commit messages for cleanup system improvements
- Example commits from robust cleanup implementation:
- `feat: implement robust cleanup system with finally blocks - tested in container`
- `fix: restore automation-v2 page with balance slider - confirmed working`
- `chore: confirm cleanup system persistence after container rebuild`
- `docs: update instructions with container development workflow`
When working with this codebase, prioritize Docker consistency, understand the dual-session architecture, leverage the comprehensive test suite to validate changes, and always implement proper cleanup patterns for browser automation to prevent resource exhaustion.
## Common Issues & Troubleshooting
### Chromium Process Accumulation
**Symptoms**: System becomes slow after hours of automation, high CPU/memory usage, many chrome processes running
**Diagnosis**: `pgrep -f "chrome|chromium" | wc -l` shows increasing process count
**Solutions**:
1. Ensure all browser automation uses finally blocks for cleanup
2. Restart automated cleanup service: `docker compose exec app node -e "require('./lib/automated-cleanup-service.ts').startPeriodicCleanup()"`
3. Manual cleanup: `docker compose exec app node test-cleanup-system.js`
4. Container restart: `docker compose restart app`
### Volume Mount Sync Issues
**Symptoms**: File changes not reflecting in running container
**Diagnosis**: Edit test file and check if visible in container
**Solutions**:
1. **Quick Fix**: Edit directly in container for immediate testing
2. **Commit Progress**: Save working changes to git before rebuilding
3. **Persistence**: Always rebuild container after confirming changes work
4. **Final Commit**: Validate and commit after successful rebuild
5. **Manual Copy**: Use `docker cp` to transfer working files
6. **Fresh Start**: Create new files instead of editing problematic ones
### Automation Page Restoration
**Symptoms**: Automation page shows old version after container rebuild
**Issue**: Git history not properly maintained in container
**Solutions**:
1. Check current branch: `git branch`
2. Restore from git: `git checkout HEAD -- app/automation-v2/page.js`
3. Verify features: Check for balance slider and multi-timeframe selection
4. Commit restoration: `git add . && git commit -m "fix: restore automation-v2 functionality" && git push`
5. Rebuild container to persist restoration
### Successful Implementation Workflow
**After completing any feature or fix:**
```bash
# 1. Test functionality thoroughly
curl http://localhost:9001/api/test-endpoint
# 2. Commit successful implementation
git add .
git commit -m "feat: [specific achievement] - fully tested and working"
git push origin development
# 3. Rebuild container for persistence
docker compose down && docker compose up --build -d
# 4. Final validation and completion commit
curl http://localhost:9001 # Verify persistent functionality
git add . && git commit -m "chore: confirm [feature] persistence - implementation complete" && git push
# 5. Consider merge to main if ready for production
# (Ask user first before merging to main branch)
```
### API Endpoint Not Responding
**Symptoms**: API calls timeout or return errors
**Diagnosis**: Check container logs: `docker compose logs -f app`
**Solutions**:
1. Verify container is running: `docker ps`
2. Check port mapping: Development=9001:3000, Production=9000:3000
3. Test direct access: `curl localhost:9001/api/status`
4. Restart if needed: `docker compose restart app`

324
.github/copilot-instructions.md vendored
View File

@@ -2,13 +2,15 @@
## 🎯 Project Context & Architecture ## 🎯 Project Context & Architecture
This is an AI-powered trading automation system with advanced learning capabilities. When working on this codebase: This is an AI-powered trading automation system with advanced learning capabilities built with Next.js 15 App Router, TypeScript, Tailwind CSS, and integrated with Drift Protocol and Jupiter DEX for automated trading execution.
### Core System Components ### Core System Components
1. **Superior Parallel Screenshot System** - 60% faster than sequential capture 1. **Superior Parallel Screenshot System** - 60% faster than sequential capture (71s vs 180s)
2. **AI Learning System** - Adapts trading decisions based on outcomes 2. **AI Learning System** - Adapts trading decisions based on outcomes with pattern recognition
3. **Orphaned Order Cleanup** - Automatic cleanup when positions close 3. **Orphaned Order Cleanup** - Automatic cleanup when positions close via position monitor
4. **Position Monitoring** - Frequent checks with integrated cleanup triggers 4. **Position Monitoring** - Frequent checks with integrated cleanup triggers
5. **Dual-Session Screenshot Automation** - AI and DIY layouts with session persistence
6. **Robust Cleanup System** - Prevents Chromium process accumulation
### Critical File Relationships ### Critical File Relationships
``` ```
@@ -16,9 +18,279 @@ app/api/automation/position-monitor/route.js → Monitors positions + triggers c
lib/simplified-stop-loss-learner.js → AI learning core with pattern recognition lib/simplified-stop-loss-learner.js → AI learning core with pattern recognition
lib/superior-screenshot-service.ts → Parallel screenshot capture system lib/superior-screenshot-service.ts → Parallel screenshot capture system
lib/enhanced-autonomous-risk-manager.js → Risk management with AI integration lib/enhanced-autonomous-risk-manager.js → Risk management with AI integration
lib/enhanced-screenshot-robust.ts → Guaranteed cleanup with finally blocks
lib/automated-cleanup-service.ts → Background process monitoring
```
## 🚀 Development Environment (Critical)
### Docker Container Development (Required)
**All development happens inside Docker containers** using Docker Compose v2. Browser automation requires specific system dependencies only available in containerized environment.
**IMPORTANT: Use Docker Compose v2 syntax** - All commands use `docker compose` (with space) instead of `docker-compose` (with hyphen).
```bash
# Development environment - Docker Compose v2 dev setup
npm run docker:dev # Port 9001:3000, hot reload, debug mode
# Direct v2 command: docker compose -f docker-compose.dev.yml up --build
# Production environment
npm run docker:up # Port 9000:3000, optimized build
# Direct v2 command: docker compose -f docker-compose.prod.yml up --build
# Debugging commands
npm run docker:logs # View container logs
npm run docker:exec # Shell access for debugging inside container
```
**Port Configuration:**
- **Development**: External port `9001` → Internal port `3000` (http://localhost:9001)
- **Production**: External port `9000` → Internal port `3000` (http://localhost:9000)
### Container-First Development Workflow
**Common Issue**: File edits not reflecting in container due to volume mount sync issues.
**Solution - Container Development Workflow:**
```bash
# 1. Access running container for immediate edits
docker compose -f docker-compose.dev.yml exec app bash
# 2. Edit files directly in container (immediate effect)
nano /app/lib/enhanced-screenshot.ts
echo "console.log('Debug: immediate test');" >> /app/debug.js
# 3. Test changes immediately (no rebuild needed)
# Changes take effect instantly for hot reload
# 4. Once everything works, copy changes back to host
docker cp container_name:/app/modified-file.js ./modified-file.js
# 5. Commit successful changes to git BEFORE rebuilding
git add .
git commit -m "feat: implement working solution for [specific feature]"
git push origin development
# 6. Rebuild container for persistence
docker compose -f docker-compose.dev.yml down
docker compose -f docker-compose.dev.yml up --build -d
# 7. Final validation and commit completion
curl http://localhost:9001 # Verify functionality
git add . && git commit -m "chore: confirm container persistence" && git push
```
### Git Branch Strategy (Required)
**Primary development workflow:**
- **`development` branch**: Use for all active development and feature work
- **`main` branch**: Stable, production-ready code only
- **Workflow**: Develop on `development` → test thoroughly → commit progress → merge to `main` when stable
```bash
# Standard development workflow with frequent commits
git checkout development # Always start here
git pull origin development # Get latest changes
# Make your changes and test in container...
# Commit working progress BEFORE rebuilding container
git add .
git commit -m "feat: [specific achievement] - tested and working"
git push origin development
# After successful container rebuild and validation
git add .
git commit -m "chore: confirm [feature] persistence after rebuild"
git push origin development
# Only merge to main when features are stable and tested
git checkout main
git merge development # When ready for production
git push origin main
```
## 🏗️ System Architecture
### Dual-Session Screenshot Automation
- **AI Layout**: `Z1TzpUrf` - RSI (top), EMAs, MACD (bottom)
- **DIY Layout**: `vWVvjLhP` - Stochastic RSI (top), VWAP, OBV (bottom)
- Parallel browser sessions for multi-layout capture in `lib/enhanced-screenshot.ts`
- TradingView automation with session persistence in `lib/tradingview-automation.ts`
- Session data stored in `.tradingview-session/` volume mount to avoid captchas
### AI Analysis Pipeline
- OpenAI GPT-4o mini for cost-effective chart analysis (~$0.006 per analysis)
- Multi-layout comparison and consensus detection in `lib/ai-analysis.ts`
- Professional trading setups with exact entry/exit levels and risk management
- Layout-specific indicator analysis (RSI vs Stochastic RSI, MACD vs OBV)
### Trading Integration
- **Drift Protocol**: Perpetual futures trading via `@drift-labs/sdk`
- **Jupiter DEX**: Spot trading on Solana
- Position management and P&L tracking in `lib/drift-trading-final.ts`
- Real-time account balance and collateral monitoring
### Browser Process Management & Cleanup System
**Critical Issue**: Chromium processes accumulate during automated trading, consuming system resources over time.
**Robust Cleanup Implementation:**
1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`)
- Guaranteed cleanup via `finally` blocks in all browser operations
- Active session tracking to prevent orphaned browsers
- Session cleanup tasks array for systematic teardown
2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`)
- Background monitoring service for orphaned processes
- Multiple kill strategies: graceful → force → system cleanup
- Periodic cleanup of temporary files and browser data
3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`)
- System-level process killing for stubborn Chromium processes
- Port cleanup and temporary directory management
- Emergency cleanup functions for resource recovery
**Implementation Patterns:**
```typescript
// Always use finally blocks for guaranteed cleanup
try {
const browser = await puppeteer.launch(options);
// ... browser operations
} finally {
// Guaranteed cleanup regardless of success/failure
await ensureBrowserCleanup(browser, sessionId);
await cleanupSessionTasks(sessionId);
}
// Background monitoring for long-running operations
const cleanupService = new AutomatedCleanupService();
cleanupService.startPeriodicCleanup(); // Every 10 minutes
```
### API Route Structure
All core functionality exposed via Next.js API routes:
```typescript
// Enhanced screenshot with progress tracking and robust cleanup
POST /api/enhanced-screenshot
{
symbol: "SOLUSD",
timeframe: "240",
layouts: ["ai", "diy"],
analyze: true
}
// Returns: { screenshots, analysis, sessionId }
// Note: Includes automatic Chromium process cleanup via finally blocks
// Drift trading endpoints
GET /api/balance # Account balance/collateral
POST /api/trading # Execute trades
GET /api/status # Trading status
GET /api/automation/position-monitor # Position monitoring with orphaned cleanup
POST /api/drift/cleanup-orders # Manual order cleanup
```
### Progress Tracking System
Real-time operation tracking for long-running tasks:
- `lib/progress-tracker.ts` manages EventEmitter-based progress
- SessionId-based tracking for multi-step operations
- Steps: init → auth → navigation → loading → capture → analysis
- Stream endpoint: `/api/progress/[sessionId]/stream`
### Page Structure & Multi-Timeframe Implementation
- `app/analysis/page.js` - Original analysis page with multi-timeframe functionality
- `app/automation/page.js` - Original automation page (legacy, may have issues)
- `app/automation-v2/page.js` - **NEW**: Clean automation page with full multi-timeframe support
- `app/automation/page-v2.js` - Alternative implementation, same functionality as automation-v2
**Multi-Timeframe Architecture Pattern:**
```javascript
// Standard timeframes array - use this exact format
const timeframes = ['5m', '15m', '30m', '1h', '2h', '4h', '1d'];
// State management for multi-timeframe selection
const [selectedTimeframes, setSelectedTimeframes] = useState(['1h', '4h']);
// Toggle function with proper array handling
const toggleTimeframe = (tf) => {
setSelectedTimeframes(prev =>
prev.includes(tf)
? prev.filter(t => t !== tf) // Remove if selected
: [...prev, tf] // Add if not selected
);
};
// Preset configurations for trading styles
const presets = {
scalping: ['5m', '15m', '1h'],
day: ['1h', '4h', '1d'],
swing: ['4h', '1d']
};
```
### Component Architecture
- `app/layout.js` - Root layout with gradient styling and navigation
- `components/Navigation.tsx` - Multi-page navigation system
- `components/AIAnalysisPanel.tsx` - Multi-timeframe analysis interface
- `components/Dashboard.tsx` - Main trading dashboard with real Drift positions
- `components/AdvancedTradingPanel.tsx` - Drift Protocol trading interface
Critical timeframe handling to avoid TradingView confusion:
```typescript
// ALWAYS use minute values first, then alternatives
'4h': ['240', '240m', '4h', '4H'] // 240 minutes FIRST
'1h': ['60', '60m', '1h', '1H'] // 60 minutes FIRST
'15m': ['15', '15m']
```
Layout URL mappings for direct navigation:
```typescript
const LAYOUT_URLS = {
'ai': 'Z1TzpUrf', // RSI + EMAs + MACD
'diy': 'vWVvjLhP' // Stochastic RSI + VWAP + OBV
}
``` ```
## 🧠 AI Learning System Patterns ## 🧠 AI Learning System Patterns
```javascript
async generateLearningReport() {
// Return comprehensive learning status
return {
summary: { totalDecisions, systemConfidence, successRate },
insights: { thresholds, confidenceLevel },
recommendations: []
};
}
async getSmartRecommendation(requestData) {
// Analyze patterns and return AI recommendation
const { distanceFromSL, symbol, marketConditions } = requestData;
// Return: { action, confidence, reasoning }
}
async recordDecision(decisionData) {
// Log decision for learning with unique ID
const id = `decision_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
// Store in database for pattern analysis
}
async assessDecisionOutcome(outcomeData) {
// Update decision with actual result for learning
// Calculate if decision was correct based on outcome
}
```
### Database Operations Best Practices:
```javascript
// ALWAYS provide unique IDs for Prisma records
await prisma.ai_learning_data.create({
data: {
id: `${prefix}_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,
// ... other fields
}
});
// Use correct import path
const { getDB } = require('./db'); // NOT './database-util'
```
### Always Include These Functions in Learning Classes: ### Always Include These Functions in Learning Classes:
```javascript ```javascript
@@ -65,6 +337,8 @@ const { getDB } = require('./db'); // NOT './database-util'
## 🔧 Error Handling Patterns ## 🔧 Error Handling Patterns
## 🔧 Error Handling Patterns
### Function Existence Checks: ### Function Existence Checks:
```javascript ```javascript
// Always check if functions exist before calling // Always check if functions exist before calling
@@ -87,6 +361,28 @@ try {
} }
``` ```
## 📊 Integration Patterns
```javascript
// Always check if functions exist before calling
if (typeof this.learner.generateLearningReport === 'function') {
const report = await this.learner.generateLearningReport();
} else {
// Fallback to alternative method
const status = await this.learner.getLearningStatus();
}
```
### Comprehensive Try-Catch:
```javascript
try {
const result = await aiFunction();
return result;
} catch (error) {
await this.log(`❌ AI function error: ${error.message}`);
return fallbackResult(); // Always provide fallback
}
```
## 📊 Integration Patterns ## 📊 Integration Patterns
### Position Monitor Integration: ### Position Monitor Integration:
@@ -297,6 +593,26 @@ When adding new features:
6. **Maintain Consistency** - Frontend and backend must match exactly 6. **Maintain Consistency** - Frontend and backend must match exactly
7. **Use Defensive Programming** - Check before calling, handle gracefully 7. **Use Defensive Programming** - Check before calling, handle gracefully
## 📚 Documentation References
### Technical Documentation
- **`ADVANCED_SYSTEM_KNOWLEDGE.md`** - Deep technical architecture, session management, cleanup systems
- **`README.md`** - Main project overview with current feature status and setup
- **`AI_LEARNING_EXPLAINED.md`** - AI learning system implementation details
- **`DRIFT_FEEDBACK_LOOP_COMPLETE.md`** - Complete Drift trading integration
- **`ROBUST_CLEANUP_IMPLEMENTATION.md`** - Browser process cleanup system details
### Implementation Guides
- **`MULTI_LAYOUT_IMPLEMENTATION.md`** - Dual-session screenshot system
- **`SESSION_PERSISTENCE.md`** - TradingView session management
- **`DOCKER_AUTOMATION.md`** - Container development workflow
- **`DEVELOPMENT_GUIDE.md`** - Complete development setup instructions
### Analysis & Troubleshooting
- **`MULTI_LAYOUT_TROUBLESHOOTING.md`** - Screenshot automation debugging
- **`CLEANUP_IMPROVEMENTS.md`** - Process management enhancements
- **`SCREENSHOT_PATH_FIXES.md`** - Screenshot capture issue resolution
--- ---
**Follow these patterns to maintain system stability and avoid the complex debugging issues that were resolved in this session.** **Follow these patterns to maintain system stability and avoid the complex debugging issues that were resolved in this session.**