docs: Add Docker Optimization & Build Cache Management section
DOCUMENTATION UPDATE (Nov 26, 2025): User quote: "ok. dont forget the documentation" Added comprehensive Docker Optimization section covering: 1. MULTI-STAGE BUILDS (already implemented): - Verified Dockerfile uses builder → runner pattern - Benefits: Smaller images, faster builds, better layer reuse 2. BUILDKIT AUTO-CLEANUP (just configured): - Updated /etc/docker/daemon.json with 20GB threshold - Auto garbage collection when cache exceeds limit - Docker restarted, BuildKit v0.14.1 active - Current baseline: 11.13GB cache (healthy) 3. AUTOMATED CLEANUP SCRIPT (ready to use): - Script: /home/icke/traderv4/cleanup_trading_bot.sh (94 lines) - Features: Keeps last 2 images, prunes cache, protects volumes - Usage: Manual (after builds) or automated (cron daily) - Typical savings: 40-50 GB per run WHY THIS MATTERS: - User previously hit 40GB cache accumulation - BuildKit auto-cleanup provides 20GB safety net - Manual script gives on-demand control - Documented process for team reference IMPLEMENTATION STATUS: Multi-stage builds confirmed in Dockerfile BuildKit configured in daemon.json (20GB threshold) Cleanup script exists and executable Docker daemon restarted with new config Current disk usage healthy (11.13GB < 20GB) Files documented: - /etc/docker/daemon.json (BuildKit config) - /home/icke/traderv4/cleanup_trading_bot.sh (manual cleanup) - Dockerfile (multi-stage builds) Added monitoring commands, usage recommendations, safety measures, and typical space savings data for team reference.
This commit is contained in:
123
.github/copilot-instructions.md
vendored
123
.github/copilot-instructions.md
vendored
@@ -614,6 +614,129 @@ docker system df
|
|||||||
- Active containers
|
- Active containers
|
||||||
- Tagged images currently in use
|
- Tagged images currently in use
|
||||||
|
|
||||||
|
### Docker Optimization & Build Cache Management (Nov 26, 2025)
|
||||||
|
|
||||||
|
**Purpose:** Prevent Docker cache accumulation (40+ GB) through automated cleanup and BuildKit optimizations
|
||||||
|
|
||||||
|
**Three-Layer Optimization Strategy:**
|
||||||
|
|
||||||
|
**1. Multi-Stage Builds (ALREADY IMPLEMENTED)**
|
||||||
|
```dockerfile
|
||||||
|
# Dockerfile already uses multi-stage pattern:
|
||||||
|
FROM node:20-alpine AS deps # Install dependencies
|
||||||
|
FROM node:20-alpine AS builder # Build application
|
||||||
|
FROM node:20-alpine AS runner # Final minimal image
|
||||||
|
|
||||||
|
# Benefits:
|
||||||
|
# - Smaller final images (only runtime dependencies)
|
||||||
|
# - Faster builds (caches each stage independently)
|
||||||
|
# - Better layer reuse
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. BuildKit Auto-Cleanup (Nov 26, 2025)**
|
||||||
|
```bash
|
||||||
|
# /etc/docker/daemon.json configuration:
|
||||||
|
{
|
||||||
|
"features": {
|
||||||
|
"buildkit": true
|
||||||
|
},
|
||||||
|
"builder": {
|
||||||
|
"gc": {
|
||||||
|
"enabled": true,
|
||||||
|
"defaultKeepStorage": "20GB"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Restart Docker to apply:
|
||||||
|
sudo systemctl restart docker
|
||||||
|
|
||||||
|
# Verify BuildKit active:
|
||||||
|
docker buildx version # Should show v0.14.1+
|
||||||
|
```
|
||||||
|
|
||||||
|
**Auto-Cleanup Behavior:**
|
||||||
|
- **Threshold:** 20GB build cache limit
|
||||||
|
- **Action:** Automatically garbage collects when exceeded
|
||||||
|
- **Safety:** Keeps recent layers for build speed
|
||||||
|
- **Monitoring:** Check current usage: `docker system df`
|
||||||
|
|
||||||
|
**Current Disk Usage Baseline (Nov 26, 2025):**
|
||||||
|
- Build Cache: 11.13GB (healthy, under 20GB threshold)
|
||||||
|
- Images: 59.2GB (33.3GB reclaimable)
|
||||||
|
- Volumes: 8.5GB (7.9GB reclaimable)
|
||||||
|
- Containers: 232.9MB
|
||||||
|
|
||||||
|
**3. Automated Cleanup Script (READY TO USE)**
|
||||||
|
```bash
|
||||||
|
# Script: /home/icke/traderv4/cleanup_trading_bot.sh (94 lines)
|
||||||
|
# Executable: -rwxr-xr-x (already set)
|
||||||
|
|
||||||
|
# Features:
|
||||||
|
# - Step 1: Keeps last 2 trading-bot images (rollback safety)
|
||||||
|
# - Step 2: Removes dangling images (untagged layers)
|
||||||
|
# - Step 3: Prunes build cache (biggest space saver)
|
||||||
|
# - Step 4: Safe volume handling (protects postgres)
|
||||||
|
# - Reporting: Shows disk space before/after
|
||||||
|
|
||||||
|
# Manual usage (recommended after builds):
|
||||||
|
cd /home/icke/traderv4
|
||||||
|
docker compose build trading-bot && ./cleanup_trading_bot.sh
|
||||||
|
|
||||||
|
# Automated usage (daily cleanup at 2 AM):
|
||||||
|
# Add to crontab: crontab -e
|
||||||
|
0 2 * * * /home/icke/traderv4/cleanup_trading_bot.sh
|
||||||
|
|
||||||
|
# Check current disk usage:
|
||||||
|
docker system df
|
||||||
|
```
|
||||||
|
|
||||||
|
**Script Safety Measures:**
|
||||||
|
- **Never removes:** Named volumes (trading-bot-postgres, etc.)
|
||||||
|
- **Never removes:** Running containers
|
||||||
|
- **Never removes:** Tagged images currently in use
|
||||||
|
- **Keeps:** Last 2 trading-bot images for quick rollback
|
||||||
|
- **Reports:** Space freed after cleanup (typical: 40-50 GB)
|
||||||
|
|
||||||
|
**When to Run Cleanup:**
|
||||||
|
1. **After builds:** Most effective, immediate cleanup
|
||||||
|
2. **Weekly:** If building frequently during development
|
||||||
|
3. **On demand:** When disk space warnings appear
|
||||||
|
4. **Before deployments:** Clean slate for major updates
|
||||||
|
|
||||||
|
**Typical Space Savings:**
|
||||||
|
- Manual script run: 40-50 GB (build cache + dangling images)
|
||||||
|
- BuildKit auto-cleanup: Maintains 20GB cap automatically
|
||||||
|
- Combined approach: Prevents accumulation entirely
|
||||||
|
|
||||||
|
**Monitoring Commands:**
|
||||||
|
```bash
|
||||||
|
# Check current disk usage
|
||||||
|
docker system df
|
||||||
|
|
||||||
|
# Detailed breakdown
|
||||||
|
docker system df -v
|
||||||
|
|
||||||
|
# Check BuildKit cache
|
||||||
|
docker buildx du
|
||||||
|
|
||||||
|
# Verify auto-cleanup threshold
|
||||||
|
grep -A10 "builder" /etc/docker/daemon.json
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why This Matters:**
|
||||||
|
- **Problem:** User previously hit 40GB cache accumulation
|
||||||
|
- **Solution:** BuildKit auto-cleanup (20GB cap) + manual script (on-demand)
|
||||||
|
- **Result:** System self-maintains, prevents disk full scenarios
|
||||||
|
- **Team benefit:** Documented process for all developers
|
||||||
|
|
||||||
|
**Implementation Status:**
|
||||||
|
- ✅ Multi-stage builds: Already present in Dockerfile (builder → runner)
|
||||||
|
- ✅ BuildKit auto-cleanup: Configured in daemon.json (20GB threshold)
|
||||||
|
- ✅ Cleanup script: Exists and ready (/home/icke/traderv4/cleanup_trading_bot.sh)
|
||||||
|
- ✅ Docker daemon: Restarted with new config (BuildKit v0.14.1 active)
|
||||||
|
- ✅ Current state: Healthy (11.13GB cache, under threshold)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Multi-Timeframe Price Tracking System (Nov 19, 2025)
|
## Multi-Timeframe Price Tracking System (Nov 19, 2025)
|
||||||
|
|||||||
Reference in New Issue
Block a user