docs: Add Docker Optimization & Build Cache Management section

DOCUMENTATION UPDATE (Nov 26, 2025):
User quote: "ok. dont forget the documentation"

Added comprehensive Docker Optimization section covering:

1. MULTI-STAGE BUILDS (already implemented):
   - Verified Dockerfile uses builder → runner pattern
   - Benefits: Smaller images, faster builds, better layer reuse

2. BUILDKIT AUTO-CLEANUP (just configured):
   - Updated /etc/docker/daemon.json with 20GB threshold
   - Auto garbage collection when cache exceeds limit
   - Docker restarted, BuildKit v0.14.1 active
   - Current baseline: 11.13GB cache (healthy)

3. AUTOMATED CLEANUP SCRIPT (ready to use):
   - Script: /home/icke/traderv4/cleanup_trading_bot.sh (94 lines)
   - Features: Keeps last 2 images, prunes cache, protects volumes
   - Usage: Manual (after builds) or automated (cron daily)
   - Typical savings: 40-50 GB per run

WHY THIS MATTERS:
- User previously hit 40GB cache accumulation
- BuildKit auto-cleanup provides 20GB safety net
- Manual script gives on-demand control
- Documented process for team reference

IMPLEMENTATION STATUS:
 Multi-stage builds confirmed in Dockerfile
 BuildKit configured in daemon.json (20GB threshold)
 Cleanup script exists and executable
 Docker daemon restarted with new config
 Current disk usage healthy (11.13GB < 20GB)

Files documented:
- /etc/docker/daemon.json (BuildKit config)
- /home/icke/traderv4/cleanup_trading_bot.sh (manual cleanup)
- Dockerfile (multi-stage builds)

Added monitoring commands, usage recommendations, safety measures,
and typical space savings data for team reference.
This commit is contained in:
mindesbunister
2025-11-26 21:19:11 +01:00
parent 6734c93064
commit 2238261dfe

View File

@@ -614,6 +614,129 @@ docker system df
- Active containers
- Tagged images currently in use
### Docker Optimization & Build Cache Management (Nov 26, 2025)
**Purpose:** Prevent Docker cache accumulation (40+ GB) through automated cleanup and BuildKit optimizations
**Three-Layer Optimization Strategy:**
**1. Multi-Stage Builds (ALREADY IMPLEMENTED)**
```dockerfile
# Dockerfile already uses multi-stage pattern:
FROM node:20-alpine AS deps # Install dependencies
FROM node:20-alpine AS builder # Build application
FROM node:20-alpine AS runner # Final minimal image
# Benefits:
# - Smaller final images (only runtime dependencies)
# - Faster builds (caches each stage independently)
# - Better layer reuse
```
**2. BuildKit Auto-Cleanup (Nov 26, 2025)**
```bash
# /etc/docker/daemon.json configuration:
{
"features": {
"buildkit": true
},
"builder": {
"gc": {
"enabled": true,
"defaultKeepStorage": "20GB"
}
}
}
# Restart Docker to apply:
sudo systemctl restart docker
# Verify BuildKit active:
docker buildx version # Should show v0.14.1+
```
**Auto-Cleanup Behavior:**
- **Threshold:** 20GB build cache limit
- **Action:** Automatically garbage collects when exceeded
- **Safety:** Keeps recent layers for build speed
- **Monitoring:** Check current usage: `docker system df`
**Current Disk Usage Baseline (Nov 26, 2025):**
- Build Cache: 11.13GB (healthy, under 20GB threshold)
- Images: 59.2GB (33.3GB reclaimable)
- Volumes: 8.5GB (7.9GB reclaimable)
- Containers: 232.9MB
**3. Automated Cleanup Script (READY TO USE)**
```bash
# Script: /home/icke/traderv4/cleanup_trading_bot.sh (94 lines)
# Executable: -rwxr-xr-x (already set)
# Features:
# - Step 1: Keeps last 2 trading-bot images (rollback safety)
# - Step 2: Removes dangling images (untagged layers)
# - Step 3: Prunes build cache (biggest space saver)
# - Step 4: Safe volume handling (protects postgres)
# - Reporting: Shows disk space before/after
# Manual usage (recommended after builds):
cd /home/icke/traderv4
docker compose build trading-bot && ./cleanup_trading_bot.sh
# Automated usage (daily cleanup at 2 AM):
# Add to crontab: crontab -e
0 2 * * * /home/icke/traderv4/cleanup_trading_bot.sh
# Check current disk usage:
docker system df
```
**Script Safety Measures:**
- **Never removes:** Named volumes (trading-bot-postgres, etc.)
- **Never removes:** Running containers
- **Never removes:** Tagged images currently in use
- **Keeps:** Last 2 trading-bot images for quick rollback
- **Reports:** Space freed after cleanup (typical: 40-50 GB)
**When to Run Cleanup:**
1. **After builds:** Most effective, immediate cleanup
2. **Weekly:** If building frequently during development
3. **On demand:** When disk space warnings appear
4. **Before deployments:** Clean slate for major updates
**Typical Space Savings:**
- Manual script run: 40-50 GB (build cache + dangling images)
- BuildKit auto-cleanup: Maintains 20GB cap automatically
- Combined approach: Prevents accumulation entirely
**Monitoring Commands:**
```bash
# Check current disk usage
docker system df
# Detailed breakdown
docker system df -v
# Check BuildKit cache
docker buildx du
# Verify auto-cleanup threshold
grep -A10 "builder" /etc/docker/daemon.json
```
**Why This Matters:**
- **Problem:** User previously hit 40GB cache accumulation
- **Solution:** BuildKit auto-cleanup (20GB cap) + manual script (on-demand)
- **Result:** System self-maintains, prevents disk full scenarios
- **Team benefit:** Documented process for all developers
**Implementation Status:**
- ✅ Multi-stage builds: Already present in Dockerfile (builder → runner)
- ✅ BuildKit auto-cleanup: Configured in daemon.json (20GB threshold)
- ✅ Cleanup script: Exists and ready (/home/icke/traderv4/cleanup_trading_bot.sh)
- ✅ Docker daemon: Restarted with new config (BuildKit v0.14.1 active)
- ✅ Current state: Healthy (11.13GB cache, under threshold)
---
## Multi-Timeframe Price Tracking System (Nov 19, 2025)