298 lines
8.9 KiB
Markdown
298 lines
8.9 KiB
Markdown
# 1-Year Retention Deployment - Dec 2, 2025
|
||
|
||
## ✅ DEPLOYMENT COMPLETE
|
||
|
||
**Date:** December 2, 2025
|
||
**Status:** Fully deployed and verified
|
||
**Git Commit:** 5773d7d
|
||
|
||
---
|
||
|
||
## Changes Made
|
||
|
||
### 1. Code Update: lib/maintenance/data-cleanup.ts
|
||
|
||
**Previous state:** 4-week retention (28 days)
|
||
**New state:** 1-year retention (365 days)
|
||
|
||
**Key changes:**
|
||
- Updated retention period: `setDate(-28)` → `setDate(-365)`
|
||
- Variable renamed: `fourWeeksAgo` → `oneYearAgo`
|
||
- Documentation updated with storage impact (~251 MB/year)
|
||
|
||
---
|
||
|
||
## Storage Analysis
|
||
|
||
### Row Size Measurement
|
||
```sql
|
||
SELECT pg_column_size(row(m.*)) as row_size_bytes
|
||
FROM "MarketData" m LIMIT 1;
|
||
```
|
||
**Result:** 152 bytes per record
|
||
|
||
### Storage Calculations
|
||
|
||
| Timeframe | Records | Storage |
|
||
|-----------|---------|---------|
|
||
| 1 hour | 180 | 27.4 KB |
|
||
| 1 day | 4,320 | 0.63 MB |
|
||
| 1 week | 30,240 | 4.4 MB |
|
||
| 28 days | 120,960 | 17.5 MB |
|
||
| **365 days** | **1,576,800** | **228.5 MB** |
|
||
| **With 10% index overhead** | | **251 MB/year** |
|
||
|
||
**Data collection rate:** 3 records/minute (1/min × 3 symbols: SOL, ETH, BTC)
|
||
|
||
---
|
||
|
||
## Deployment Verification
|
||
|
||
### Container Status
|
||
```bash
|
||
docker compose up -d --force-recreate trading-bot
|
||
# Container trading-bot-v4 Started in 19.0s ✅
|
||
```
|
||
|
||
### Startup Logs Confirmed
|
||
```
|
||
🎯 Server starting - initializing services...
|
||
🧹 Starting data cleanup service...
|
||
✅ Data cleanup scheduled for 3 AM (in 15 hours)
|
||
✅ Data cleanup complete: Deleted 0 old market data rows (older than 2024-12-02) in 5ms
|
||
```
|
||
|
||
### Cutoff Date Verification
|
||
```sql
|
||
SELECT NOW() - INTERVAL '365 days' as one_year_cutoff;
|
||
```
|
||
**Result:** 2024-12-02 (1 year ago from deployment date) ✅
|
||
|
||
**Previous cutoff (4 weeks):** Nov 4, 2025
|
||
**New cutoff (1 year):** Dec 2, 2024
|
||
|
||
**Impact:** Records from Dec 2, 2024 onwards will be retained (vs only since Nov 4, 2025)
|
||
|
||
---
|
||
|
||
## Benefits of 1-Year Retention
|
||
|
||
### Comparison: 4 Weeks vs 1 Year
|
||
|
||
| Metric | 4 Weeks | 1 Year | Increase |
|
||
|--------|---------|--------|----------|
|
||
| **Storage** | 18 MB | 251 MB | 14× |
|
||
| **Records** | 120,960 | 1,576,800 | 13× |
|
||
| **Blocked signals** | 20-30 | 260-390 | 13× |
|
||
| **Analysis value** | Limited | Comprehensive | Massive |
|
||
|
||
### Key Advantages
|
||
|
||
1. **13× more historical data** for pattern analysis
|
||
2. **Seasonal trend detection** (summer vs winter volatility)
|
||
3. **Better statistical significance** for threshold decisions
|
||
4. **No risk of losing valuable blocked signal data**
|
||
5. **More complete picture** of indicator behavior over time
|
||
6. **Storage cost negligible** (0.25 GB vs user likely has TB+ available)
|
||
|
||
### Blocked Signal Analysis Benefits
|
||
|
||
**With 4-week retention:**
|
||
- ~20-30 blocked signals per month
|
||
- Limited timeframe for pattern detection
|
||
- Risk of losing valuable historical data
|
||
|
||
**With 1-year retention:**
|
||
- ~260-390 blocked signals per year
|
||
- Can analyze across different market conditions
|
||
- Discover patterns like: "Quality 80 + ADX rising 17→22 = avg 180min to TP1"
|
||
|
||
---
|
||
|
||
## Current Data Status
|
||
|
||
### Database Check (Dec 2, 2025 10:55)
|
||
```sql
|
||
SELECT symbol, COUNT(*) as rows,
|
||
MIN(TO_CHAR(timestamp, 'MM-DD HH24:MI')) as oldest,
|
||
MAX(TO_CHAR(timestamp, 'MM-DD HH24:MI')) as newest
|
||
FROM "MarketData" GROUP BY symbol;
|
||
```
|
||
|
||
**Result:**
|
||
```
|
||
symbol | rows | oldest | newest
|
||
----------+------+-------------+-------------
|
||
SOL-PERP | 1 | 12-02 10:25 | 12-02 10:25
|
||
```
|
||
|
||
**Status:** Test record confirmed, awaiting live TradingView 1-minute alerts
|
||
|
||
---
|
||
|
||
## Next Steps
|
||
|
||
### Immediate (Next 24 hours)
|
||
1. ✅ Monitor container stability - No crashes detected
|
||
2. ⏳ Watch for live 1-minute data from TradingView alerts
|
||
3. ⏳ Verify row growth: Should increase by ~180 rows/hour (3 symbols × 60 min)
|
||
4. ⏳ Check at 3 AM: Cleanup should run with 1-year cutoff
|
||
|
||
### Short Term (Week 1)
|
||
5. Monitor database size growth (~4.4 MB expected)
|
||
6. Verify no gaps in data collection
|
||
7. Confirm all 8 indicator fields populated (not NULL)
|
||
8. Validate cleanup runs daily without errors
|
||
|
||
### Medium Term (Months 1-3)
|
||
9. Collect 65-100 blocked signals with 8-hour 1-minute history
|
||
10. Monitor database size (18-55 MB)
|
||
11. Validate data quality (no gaps, all indicators present)
|
||
12. Begin preliminary pattern analysis
|
||
|
||
### Long Term (Months 4-12)
|
||
13. Continue data collection to 260-390 blocked signals
|
||
14. Refactor BlockedSignalTracker to query MarketData table
|
||
15. Add precise timing fields: tp1HitTime, minutesToTP1, adxAtTP1, rsiAtTP1
|
||
16. Comprehensive pattern analysis with full year of data
|
||
17. Make data-driven threshold decisions (lower to 85/80 or keep 90/80)
|
||
|
||
---
|
||
|
||
## Monitoring Commands
|
||
|
||
### Check Data Collection
|
||
```bash
|
||
# View current row counts
|
||
docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c \
|
||
"SELECT symbol, COUNT(*) as rows FROM \"MarketData\" GROUP BY symbol;"
|
||
|
||
# View recent data
|
||
docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c \
|
||
"SELECT symbol, price, adx, atr, TO_CHAR(timestamp, 'MM-DD HH24:MI:SS') \
|
||
FROM \"MarketData\" ORDER BY timestamp DESC LIMIT 10;"
|
||
```
|
||
|
||
### Check Database Size
|
||
```bash
|
||
docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c \
|
||
"SELECT pg_size_pretty(pg_total_relation_size('\"MarketData\"')) as table_size;"
|
||
```
|
||
|
||
### Check Cleanup Schedule
|
||
```bash
|
||
docker logs trading-bot-v4 | grep "cleanup"
|
||
```
|
||
|
||
---
|
||
|
||
## Technical Details
|
||
|
||
### MarketData Model (8 Fields)
|
||
```typescript
|
||
{
|
||
id: String (cuid)
|
||
createdAt: DateTime
|
||
|
||
symbol: String // "SOL-PERP", "ETH-PERP", "BTC-PERP"
|
||
timeframe: String // "1" for 1-minute
|
||
price: Float // Close price
|
||
|
||
// Full indicator suite (ALL CONFIRMED SAVING):
|
||
atr: Float // Volatility %
|
||
adx: Float // Trend strength
|
||
rsi: Float // Momentum
|
||
volumeRatio: Float // Volume vs average
|
||
pricePosition: Float // Position in range (%)
|
||
maGap: Float // MA50-MA200 gap
|
||
volume: Float // Raw volume
|
||
|
||
timestamp: DateTime // Exact candle close time
|
||
}
|
||
```
|
||
|
||
### Cleanup Service Configuration
|
||
- **File:** lib/maintenance/data-cleanup.ts
|
||
- **Schedule:** Daily at 3 AM (cron: `0 3 * * *`)
|
||
- **Retention:** 365 days (1 year)
|
||
- **Action:** Deletes records where `createdAt < NOW() - INTERVAL '365 days'`
|
||
- **Integration:** Started automatically via lib/startup/init-position-manager.ts
|
||
|
||
### Test Data Validation
|
||
```
|
||
ID: cmiofn61g0000t407ilf019cy
|
||
Symbol: SOL-PERP ✅
|
||
Timeframe: 1 ✅
|
||
Price: $127.85 ✅
|
||
ATR: 2.8 ✅
|
||
ADX: 21.5 ✅
|
||
RSI: 62.1 ✅
|
||
Volume Ratio: 1.5 ✅
|
||
Price Position: 55.2% ✅
|
||
MA Gap: 0.3 ✅
|
||
Volume: 18500 ✅
|
||
Timestamp: Dec 2, 10:25:55 ✅
|
||
```
|
||
|
||
---
|
||
|
||
## Git History
|
||
|
||
### Commit: 5773d7d
|
||
```
|
||
feat: Extend 1-minute data retention from 4 weeks to 1 year
|
||
|
||
- Updated lib/maintenance/data-cleanup.ts retention period: 28 days → 365 days
|
||
- Storage requirements validated: 251 MB/year (negligible)
|
||
- Rationale: 13× more historical data for better pattern analysis
|
||
- Benefits: 260-390 blocked signals/year vs 20-30/month
|
||
- Cleanup cutoff: Now Dec 2, 2024 (vs Nov 4, 2025 previously)
|
||
- Deployment verified: Container restarted, cleanup scheduled for 3 AM daily
|
||
```
|
||
|
||
**Files changed:** 11 files, 1191 insertions, 7 deletions
|
||
**Branch:** master
|
||
**Remote:** Pushed successfully
|
||
|
||
---
|
||
|
||
## Success Criteria
|
||
|
||
| Criterion | Status |
|
||
|-----------|--------|
|
||
| Code updated with 1-year retention | ✅ COMPLETE |
|
||
| Docker image rebuilt | ✅ COMPLETE |
|
||
| Container restarted | ✅ COMPLETE |
|
||
| Startup logs verified | ✅ COMPLETE |
|
||
| Cleanup cutoff date confirmed (Dec 2, 2024) | ✅ COMPLETE |
|
||
| Cleanup scheduled for 3 AM daily | ✅ COMPLETE |
|
||
| Git commit created | ✅ COMPLETE |
|
||
| Changes pushed to remote | ✅ COMPLETE |
|
||
| Documentation created | ✅ COMPLETE |
|
||
| Test data validated (all 8 fields) | ✅ COMPLETE |
|
||
| Storage requirements calculated (251 MB/year) | ✅ COMPLETE |
|
||
|
||
---
|
||
|
||
## User's Original Question
|
||
|
||
**Question:** "please calculate how much mb the one month storage of the 1 minute datapoints will consume. maybe we can extend this to 1 year. i think it will not take much storage."
|
||
|
||
**Answer:**
|
||
- **1 month (28 days):** 17.5 MB (~18 MB)
|
||
- **1 year (365 days):** 251 MB
|
||
|
||
**User's intuition:** "i think it will not take much storage" → **CORRECT!** ✅
|
||
|
||
**Decision:** Extended retention from 4 weeks to 1 year based on minimal storage requirements and massive analytical benefits.
|
||
|
||
---
|
||
|
||
## Conclusion
|
||
|
||
✅ **DEPLOYMENT SUCCESSFUL**
|
||
|
||
The 1-minute data retention period has been successfully extended from 4 weeks to 1 year. Storage requirements are negligible (251 MB/year), while analytical benefits are massive (13× more historical data). System is now configured to collect and retain a full year of continuous 1-minute market data across all indicators, providing comprehensive historical context for future blocked signal analysis and threshold optimization decisions.
|
||
|
||
**Next milestone:** Begin collecting live TradingView 1-minute alerts and monitor data accumulation over the next 24 hours.
|