docs: enhance copilot instructions with container development workflow and git best practices
- Add direct container editing workflow for immediate testing - Document robust cleanup system architecture and implementation - Include comprehensive troubleshooting section with common issues - Add git commit patterns for progress tracking and persistence - Update testing procedures with process monitoring - Enhance API documentation with cleanup integration - Add successful implementation workflow with validation steps
This commit is contained in:
mindesbunister
303
.github/copilot-instructions.instructions.md
vendored
303
.github/copilot-instructions.instructions.md
vendored
@@ -55,10 +55,43 @@ npm run docker:exec # Shell access for debugging inside container
|
||||
- **Development**: External port `9001` → Internal port `3000` (http://localhost:9001)
|
||||
- **Production**: External port `9000` → Internal port `3000` (http://localhost:9000)
|
||||
|
||||
### Docker Volume Mount Troubleshooting
|
||||
### Docker Volume Mount Troubleshooting & Direct Container Development
|
||||
**Common Issue**: File edits not reflecting in container due to volume mount sync issues.
|
||||
|
||||
**Solutions:**
|
||||
**Container-First Development Workflow:**
|
||||
For immediate results and faster iteration, edit files directly inside the running container first, then rebuild for persistence:
|
||||
|
||||
```bash
|
||||
# 1. Access running container for immediate edits
|
||||
docker compose -f docker-compose.dev.yml exec app bash
|
||||
|
||||
# 2. Edit files directly in container (immediate effect)
|
||||
# Use nano, vi, or echo for quick changes
|
||||
nano /app/lib/enhanced-screenshot.ts
|
||||
echo "console.log('Debug: immediate test');" >> /app/debug.js
|
||||
|
||||
# 3. Test changes immediately (no rebuild needed)
|
||||
# Changes take effect instantly for hot reload
|
||||
|
||||
# 4. Once everything works, copy changes back to host
|
||||
docker cp container_name:/app/modified-file.js ./modified-file.js
|
||||
|
||||
# 5. Commit successful changes to git BEFORE rebuilding
|
||||
git add .
|
||||
git commit -m "feat: implement working solution for [specific feature]"
|
||||
git push origin development
|
||||
|
||||
# 6. Rebuild container for persistence
|
||||
docker compose -f docker-compose.dev.yml down
|
||||
docker compose -f docker-compose.dev.yml up --build -d
|
||||
|
||||
# 7. Final validation and commit completion
|
||||
# Test that changes persist after rebuild
|
||||
curl http://localhost:9001 # Verify functionality
|
||||
git add . && git commit -m "chore: confirm container persistence" && git push
|
||||
```
|
||||
|
||||
**Alternative Solutions:**
|
||||
1. **Fresh Implementation Approach**: When modifying existing files fails, create new files (e.g., `page-v2.js`) instead of editing corrupted files
|
||||
2. **Container Restart**: `docker compose -f docker-compose.dev.yml restart app`
|
||||
3. **Full Rebuild**: `docker compose -f docker-compose.dev.yml down && docker compose -f docker-compose.dev.yml up --build`
|
||||
@@ -72,6 +105,12 @@ echo "test-$(date)" > test-volume-mount.txt
|
||||
docker compose -f docker-compose.dev.yml exec app cat test-volume-mount.txt
|
||||
```
|
||||
|
||||
**Container Development Best Practices:**
|
||||
- **Speed**: Direct container edits = immediate testing
|
||||
- **Persistence**: Always rebuild container after successful tests
|
||||
- **Backup**: Use `docker cp` to extract working changes before rebuild
|
||||
- **Debugging**: Use container shell for real-time log inspection and debugging
|
||||
|
||||
### Multi-Timeframe Feature Copy Pattern
|
||||
When copying multi-timeframe functionality between pages:
|
||||
|
||||
@@ -110,7 +149,7 @@ const toggleTimeframe = (tf) => {
|
||||
### API Route Structure
|
||||
All core functionality exposed via Next.js API routes:
|
||||
```typescript
|
||||
// Enhanced screenshot with progress tracking
|
||||
// Enhanced screenshot with progress tracking and robust cleanup
|
||||
POST /api/enhanced-screenshot
|
||||
{
|
||||
symbol: "SOLUSD",
|
||||
@@ -119,6 +158,7 @@ POST /api/enhanced-screenshot
|
||||
analyze: true
|
||||
}
|
||||
// Returns: { screenshots, analysis, sessionId }
|
||||
// Note: Includes automatic Chromium process cleanup via finally blocks
|
||||
|
||||
// Drift trading endpoints
|
||||
GET /api/balance # Account balance/collateral
|
||||
@@ -126,6 +166,12 @@ POST /api/trading # Execute trades
|
||||
GET /api/status # Trading status
|
||||
```
|
||||
|
||||
**API Development Tips:**
|
||||
- All browser automation APIs include guaranteed cleanup via finally blocks
|
||||
- Use session tracking for long-running operations
|
||||
- Test API endpoints directly with curl before UI integration
|
||||
- Monitor Chromium processes during API testing: `pgrep -f chrome | wc -l`
|
||||
|
||||
### Progress Tracking System
|
||||
Real-time operation tracking for long-running tasks:
|
||||
- `lib/progress-tracker.ts` manages EventEmitter-based progress
|
||||
@@ -133,7 +179,62 @@ Real-time operation tracking for long-running tasks:
|
||||
- Steps: init → auth → navigation → loading → capture → analysis
|
||||
- Stream endpoint: `/api/progress/[sessionId]/stream`
|
||||
|
||||
### TradingView Automation Patterns
|
||||
### Browser Process Management & Cleanup System
|
||||
**Critical Issue**: Chromium processes accumulate during automated trading, consuming system resources over time.
|
||||
|
||||
**Robust Cleanup Implementation:**
|
||||
The trading bot includes a comprehensive cleanup system to prevent Chromium process accumulation:
|
||||
|
||||
**Core Cleanup Components:**
|
||||
1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`)
|
||||
- Guaranteed cleanup via `finally` blocks in all browser operations
|
||||
- Active session tracking to prevent orphaned browsers
|
||||
- Session cleanup tasks array for systematic teardown
|
||||
|
||||
2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`)
|
||||
- Background monitoring service for orphaned processes
|
||||
- Multiple kill strategies: graceful → force → system cleanup
|
||||
- Periodic cleanup of temporary files and browser data
|
||||
|
||||
3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`)
|
||||
- System-level process killing for stubborn Chromium processes
|
||||
- Port cleanup and temporary directory management
|
||||
- Emergency cleanup functions for resource recovery
|
||||
|
||||
**Implementation Patterns:**
|
||||
```typescript
|
||||
// Always use finally blocks for guaranteed cleanup
|
||||
try {
|
||||
const browser = await puppeteer.launch(options);
|
||||
// ... browser operations
|
||||
} finally {
|
||||
// Guaranteed cleanup regardless of success/failure
|
||||
await ensureBrowserCleanup(browser, sessionId);
|
||||
await cleanupSessionTasks(sessionId);
|
||||
}
|
||||
|
||||
// Background monitoring for long-running operations
|
||||
const cleanupService = new AutomatedCleanupService();
|
||||
cleanupService.startPeriodicCleanup(); // Every 10 minutes
|
||||
```
|
||||
|
||||
**Cleanup Testing:**
|
||||
```bash
|
||||
# Test cleanup system functionality
|
||||
node test-cleanup-system.js
|
||||
|
||||
# Monitor Chromium processes during automation
|
||||
watch 'pgrep -f "chrome|chromium" | wc -l'
|
||||
|
||||
# Manual cleanup if needed
|
||||
node -e "require('./lib/aggressive-cleanup.ts').performAggressiveCleanup()"
|
||||
```
|
||||
|
||||
**Prevention Strategies:**
|
||||
- Use session tracking for all browser instances
|
||||
- Implement timeout protection for long-running operations
|
||||
- Monitor resource usage during extended automation cycles
|
||||
- Restart containers periodically for fresh environment
|
||||
Critical timeframe handling to avoid TradingView confusion:
|
||||
```typescript
|
||||
// ALWAYS use minute values first, then alternatives
|
||||
@@ -157,6 +258,59 @@ const LAYOUT_URLS = {
|
||||
- `components/Dashboard.tsx` - Main trading dashboard with real Drift positions
|
||||
- `components/AdvancedTradingPanel.tsx` - Drift Protocol trading interface
|
||||
|
||||
### Cleanup System Architecture
|
||||
**Critical Production Issue**: Chromium processes accumulate during automated trading, leading to resource exhaustion after several hours of operation.
|
||||
|
||||
**Solution Components:**
|
||||
1. **Enhanced Screenshot Service** (`lib/enhanced-screenshot-robust.ts`)
|
||||
- Replaces original screenshot service with guaranteed cleanup
|
||||
- Uses `finally` blocks to ensure browser cleanup regardless of success/failure
|
||||
- Active session tracking with cleanup task arrays
|
||||
- Force-kill functionality for stubborn processes
|
||||
|
||||
2. **Automated Cleanup Service** (`lib/automated-cleanup-service.ts`)
|
||||
- Background monitoring service that runs every 10 minutes
|
||||
- Multiple cleanup strategies: graceful → force → system-level cleanup
|
||||
- Temporary file cleanup and browser data directory management
|
||||
- Orphaned process detection and elimination
|
||||
|
||||
3. **Aggressive Cleanup Utilities** (`lib/aggressive-cleanup.ts`)
|
||||
- Emergency cleanup functions for critical resource recovery
|
||||
- System-level process management with multiple kill strategies
|
||||
- Port cleanup and zombie process elimination
|
||||
- Used by both automated service and manual intervention
|
||||
|
||||
**Integration Pattern:**
|
||||
```typescript
|
||||
// In API routes - always use finally blocks
|
||||
app/api/enhanced-screenshot/route.js:
|
||||
try {
|
||||
const result = await enhancedScreenshot.captureAndAnalyze(...);
|
||||
return NextResponse.json(result);
|
||||
} finally {
|
||||
// Guaranteed cleanup execution
|
||||
await enhancedScreenshot.cleanup();
|
||||
}
|
||||
|
||||
// Background monitoring
|
||||
lib/automated-cleanup-service.ts:
|
||||
setInterval(async () => {
|
||||
await this.performCleanup();
|
||||
}, 10 * 60 * 1000); // Every 10 minutes
|
||||
```
|
||||
|
||||
**Testing Cleanup System:**
|
||||
```bash
|
||||
# Monitor process count during operation
|
||||
watch 'pgrep -f "chrome|chromium" | wc -l'
|
||||
|
||||
# Test cleanup functionality
|
||||
node test-cleanup-system.js
|
||||
|
||||
# Manual cleanup if needed
|
||||
docker compose exec app node -e "require('./lib/aggressive-cleanup.ts').forceKillAllChromium()"
|
||||
```
|
||||
|
||||
### Page Structure & Multi-Timeframe Implementation
|
||||
- `app/analysis/page.js` - Original analysis page with multi-timeframe functionality
|
||||
- `app/automation/page.js` - Original automation page (legacy, may have issues)
|
||||
@@ -246,6 +400,9 @@ Test files follow specific patterns - use them to validate changes:
|
||||
# Test dual-session screenshot capture
|
||||
node test-enhanced-screenshot.js
|
||||
|
||||
# Test robust cleanup system
|
||||
node test-cleanup-system.js
|
||||
|
||||
# Test Docker environment (requires Docker Compose v2)
|
||||
./test-docker-comprehensive.sh
|
||||
|
||||
@@ -254,6 +411,35 @@ node test-analysis-api.js
|
||||
|
||||
# Test Drift trading integration
|
||||
node test-drift-trading.js
|
||||
|
||||
# Monitor resource usage during automation
|
||||
watch 'pgrep -f "chrome|chromium" | wc -l'
|
||||
```
|
||||
|
||||
**Container-First Development Workflow:**
|
||||
```bash
|
||||
# 1. Start development container
|
||||
npm run docker:dev # Port 9001
|
||||
|
||||
# 2. For immediate testing, edit directly in container
|
||||
docker compose -f docker-compose.dev.yml exec app bash
|
||||
# Edit files in /app/ directory for instant results
|
||||
|
||||
# 3. Test changes in real-time (hot reload active)
|
||||
curl http://localhost:9001/api/enhanced-screenshot
|
||||
|
||||
# 4. Once working, commit progress to git
|
||||
git add .
|
||||
git commit -m "feat: implement [feature] - tested and working"
|
||||
git push origin development
|
||||
|
||||
# 5. Rebuild container for persistence
|
||||
docker compose -f docker-compose.dev.yml down
|
||||
docker compose -f docker-compose.dev.yml up --build -d
|
||||
|
||||
# 6. Validate persistent changes and final commit
|
||||
curl http://localhost:9001 # Should show updated functionality
|
||||
git add . && git commit -m "chore: confirm persistence after container rebuild" && git push
|
||||
```
|
||||
|
||||
Browser automation debugging:
|
||||
@@ -262,6 +448,7 @@ Browser automation debugging:
|
||||
- Session persistence prevents repeated logins/captchas
|
||||
- Use `npm run docker:logs` to view real-time automation logs
|
||||
- All Docker commands use v2 syntax: `docker compose` (not `docker-compose`)
|
||||
- Monitor Chromium processes: `docker compose exec app pgrep -f chrome`
|
||||
|
||||
## Code Style & Architecture Patterns
|
||||
- **Client Components**: Use `"use client"` for state/effects, server components by default
|
||||
@@ -276,23 +463,41 @@ Browser automation debugging:
|
||||
- **Progress Tracking**: EventEmitter-based real-time updates via SSE
|
||||
- **Multi-Stage Docker**: Development vs production builds with browser optimization
|
||||
- **CAPTCHA Handling**: Manual CAPTCHA mode with X11 forwarding (`ALLOW_MANUAL_CAPTCHA=true`)
|
||||
- **Process Management**: Robust cleanup system prevents Chromium accumulation
|
||||
- **Container Development**: Direct in-container editing for immediate testing, rebuild for persistence
|
||||
|
||||
## Development vs Production Modes
|
||||
- **Development**: Port 9001:3000, hot reload, debug logging, headless: false option
|
||||
- **Production**: Port 9000:3000, optimized build, minimal logging, always headless
|
||||
|
||||
**Development Container Features:**
|
||||
- **Hot Reload**: File changes reflect immediately (when volume mounts work)
|
||||
- **Process Monitoring**: Real-time Chromium process tracking
|
||||
- **Debug Access**: Shell access via `docker compose exec app bash`
|
||||
- **Immediate Testing**: Edit files in `/app/` directory for instant results
|
||||
- **Resource Cleanup**: Automated cleanup services running in background
|
||||
|
||||
### Git Branch Strategy (Required)
|
||||
**Primary development workflow:**
|
||||
- **`development` branch**: Use for all active development and feature work
|
||||
- **`main` branch**: Stable, production-ready code only
|
||||
- **Workflow**: Develop on `development` → test thoroughly → merge to `main` when stable
|
||||
- **Workflow**: Develop on `development` → test thoroughly → commit progress → merge to `main` when stable
|
||||
|
||||
```bash
|
||||
# Standard development workflow
|
||||
# Standard development workflow with frequent commits
|
||||
git checkout development # Always start here
|
||||
git pull origin development # Get latest changes
|
||||
# Make your changes...
|
||||
git add . && git commit -m "feat: description"
|
||||
|
||||
# Make your changes and test in container...
|
||||
|
||||
# Commit working progress BEFORE rebuilding container
|
||||
git add .
|
||||
git commit -m "feat: [specific achievement] - tested and working"
|
||||
git push origin development
|
||||
|
||||
# After successful container rebuild and validation
|
||||
git add .
|
||||
git commit -m "chore: confirm [feature] persistence after rebuild"
|
||||
git push origin development
|
||||
|
||||
# Only merge to main when features are stable and tested and you have asked the user to merge to main
|
||||
@@ -301,4 +506,84 @@ git merge development # When ready for production
|
||||
git push origin main
|
||||
```
|
||||
|
||||
When working with this codebase, prioritize Docker consistency, understand the dual-session architecture, and leverage the comprehensive test suite to validate changes.
|
||||
**Git Commit Best Practices:**
|
||||
- **Commit Early**: Save working progress before container rebuilds
|
||||
- **Commit Often**: After each successful test or implementation step
|
||||
- **Descriptive Messages**: Include what was accomplished and tested
|
||||
- **Final Commits**: Always commit after confirming container persistence
|
||||
|
||||
**Container Persistence & Git Strategy:**
|
||||
- Git changes only persist after container rebuild with `--build` flag
|
||||
- **CRITICAL**: Commit working changes BEFORE rebuilding container
|
||||
- Test changes in container first, then commit and rebuild
|
||||
- Use descriptive commit messages for cleanup system improvements
|
||||
- Example commits from robust cleanup implementation:
|
||||
- `feat: implement robust cleanup system with finally blocks - tested in container`
|
||||
- `fix: restore automation-v2 page with balance slider - confirmed working`
|
||||
- `chore: confirm cleanup system persistence after container rebuild`
|
||||
- `docs: update instructions with container development workflow`
|
||||
|
||||
When working with this codebase, prioritize Docker consistency, understand the dual-session architecture, leverage the comprehensive test suite to validate changes, and always implement proper cleanup patterns for browser automation to prevent resource exhaustion.
|
||||
|
||||
## Common Issues & Troubleshooting
|
||||
|
||||
### Chromium Process Accumulation
|
||||
**Symptoms**: System becomes slow after hours of automation, high CPU/memory usage, many chrome processes running
|
||||
**Diagnosis**: `pgrep -f "chrome|chromium" | wc -l` shows increasing process count
|
||||
**Solutions**:
|
||||
1. Ensure all browser automation uses finally blocks for cleanup
|
||||
2. Restart automated cleanup service: `docker compose exec app node -e "require('./lib/automated-cleanup-service.ts').startPeriodicCleanup()"`
|
||||
3. Manual cleanup: `docker compose exec app node test-cleanup-system.js`
|
||||
4. Container restart: `docker compose restart app`
|
||||
|
||||
### Volume Mount Sync Issues
|
||||
**Symptoms**: File changes not reflecting in running container
|
||||
**Diagnosis**: Edit test file and check if visible in container
|
||||
**Solutions**:
|
||||
1. **Quick Fix**: Edit directly in container for immediate testing
|
||||
2. **Commit Progress**: Save working changes to git before rebuilding
|
||||
3. **Persistence**: Always rebuild container after confirming changes work
|
||||
4. **Final Commit**: Validate and commit after successful rebuild
|
||||
5. **Manual Copy**: Use `docker cp` to transfer working files
|
||||
6. **Fresh Start**: Create new files instead of editing problematic ones
|
||||
|
||||
### Automation Page Restoration
|
||||
**Symptoms**: Automation page shows old version after container rebuild
|
||||
**Issue**: Git history not properly maintained in container
|
||||
**Solutions**:
|
||||
1. Check current branch: `git branch`
|
||||
2. Restore from git: `git checkout HEAD -- app/automation-v2/page.js`
|
||||
3. Verify features: Check for balance slider and multi-timeframe selection
|
||||
4. Commit restoration: `git add . && git commit -m "fix: restore automation-v2 functionality" && git push`
|
||||
5. Rebuild container to persist restoration
|
||||
|
||||
### Successful Implementation Workflow
|
||||
**After completing any feature or fix:**
|
||||
```bash
|
||||
# 1. Test functionality thoroughly
|
||||
curl http://localhost:9001/api/test-endpoint
|
||||
|
||||
# 2. Commit successful implementation
|
||||
git add .
|
||||
git commit -m "feat: [specific achievement] - fully tested and working"
|
||||
git push origin development
|
||||
|
||||
# 3. Rebuild container for persistence
|
||||
docker compose down && docker compose up --build -d
|
||||
|
||||
# 4. Final validation and completion commit
|
||||
curl http://localhost:9001 # Verify persistent functionality
|
||||
git add . && git commit -m "chore: confirm [feature] persistence - implementation complete" && git push
|
||||
|
||||
# 5. Consider merge to main if ready for production
|
||||
# (Ask user first before merging to main branch)
|
||||
```
|
||||
|
||||
### API Endpoint Not Responding
|
||||
**Symptoms**: API calls timeout or return errors
|
||||
**Diagnosis**: Check container logs: `docker compose logs -f app`
|
||||
**Solutions**:
|
||||
1. Verify container is running: `docker ps`
|
||||
2. Check port mapping: Development=9001:3000, Production=9000:3000
|
||||
3. Test direct access: `curl localhost:9001/api/status`
|
||||
4. Restart if needed: `docker compose restart app`
|
||||
|
||||
Reference in New Issue
Block a user