docs: Update minimum quality score from 60 to 81 across documentation
- Updated .github/copilot-instructions.md key constraints and signal quality system description - Updated config/trading.ts minimum score from 60 to 81 with v8 performance rationale - Updated SIGNAL_QUALITY_SETUP_GUIDE.md intro to reflect 81 threshold - Updated SIGNAL_QUALITY_OPTIMIZATION_ROADMAP.md current system section - Updated BLOCKED_SIGNALS_TRACKING.md quality score requirements Context: After v8 Money Line indicator deployed with 0.6% flip threshold, system achieving 66.7% win rate with average quality score 94.2. Raised minimum threshold from 60 to 81 to maintain exceptional selectivity. Current v8 stats: 6 trades, 4 wins, $649.32 profit, 94.2 avg quality Account growth: $540 → $1,134.92 (110% gain in 2-3 days)
This commit is contained in:
mindesbunister
229
.github/copilot-instructions.md
vendored
229
.github/copilot-instructions.md
vendored
@@ -24,7 +24,7 @@
|
||||
**Key Constraints:**
|
||||
- Can't afford extended drawdowns (limited capital)
|
||||
- Must maintain 60%+ win rate to compound effectively
|
||||
- Quality over quantity - only trade 60+ signal quality scores (lowered from 65 on Nov 12, 2025)
|
||||
- Quality over quantity - only trade 81+ signal quality scores (raised from 60 on Nov 21, 2025 after v8 success)
|
||||
- After 3 consecutive losses, STOP and review system
|
||||
|
||||
## Architecture Overview
|
||||
@@ -78,7 +78,7 @@
|
||||
- BTC and other symbols fall back to global settings (`MAX_POSITION_SIZE_USD`, `LEVERAGE`)
|
||||
- **Priority:** Per-symbol ENV → Market config → Global ENV → Defaults
|
||||
|
||||
**Signal Quality System:** Filters trades based on 5 metrics (ATR, ADX, RSI, volumeRatio, pricePosition) scored 0-100. Only trades scoring 60+ are executed (lowered from 65 after data analysis showed 60-64 tier outperformed higher scores). Scores stored in database for future optimization.
|
||||
**Signal Quality System:** Filters trades based on 5 metrics (ATR, ADX, RSI, volumeRatio, pricePosition) scored 0-100. Only trades scoring 81+ are executed (raised from 60 on Nov 21, 2025 after v8 proving 94.2 avg quality with 66.7% win rate). Scores stored in database for future optimization.
|
||||
|
||||
**Timeframe-Aware Scoring:** Signal quality thresholds adjust based on timeframe (5min vs daily):
|
||||
- 5min: ADX 12+ trending (vs 18+ for daily), ATR 0.2-0.7% healthy (vs 0.4%+ for daily)
|
||||
@@ -562,7 +562,47 @@ After sufficient data collected:
|
||||
|
||||
## Critical Components
|
||||
|
||||
### 1. Phantom Trade Auto-Closure System
|
||||
### 1. Persistent Logger System (lib/utils/persistent-logger.ts)
|
||||
**Purpose:** Survive-container-restarts logging for critical errors and trade failures
|
||||
|
||||
**Key features:**
|
||||
- Writes to `/app/logs/errors.log` (Docker volume mounted from host)
|
||||
- Logs survive container restarts, rebuilds, crashes
|
||||
- Daily log rotation with 30-day retention
|
||||
- Structured JSON logging with timestamps, context, stack traces
|
||||
- Used for database save failures, Drift API errors, critical incidents
|
||||
|
||||
**Usage:**
|
||||
```typescript
|
||||
import { persistentLogger } from '../utils/persistent-logger'
|
||||
|
||||
try {
|
||||
await createTrade({...})
|
||||
} catch (error) {
|
||||
persistentLogger.logError('DATABASE_SAVE_FAILED', error, {
|
||||
symbol: 'SOL-PERP',
|
||||
entryPrice: 133.69,
|
||||
transactionSignature: '5Yx2...',
|
||||
// ALL data needed to reconstruct trade
|
||||
})
|
||||
throw error
|
||||
}
|
||||
```
|
||||
|
||||
**Infrastructure:**
|
||||
- Docker volume: `./logs:/app/logs` (docker-compose.yml line 63)
|
||||
- Directory: `/home/icke/traderv4/logs/` with `.gitkeep`
|
||||
- Log format: `{"timestamp":"2025-11-21T00:40:14.123Z","context":"DATABASE_SAVE_FAILED","error":"...","stack":"...","metadata":{...}}`
|
||||
|
||||
**Why it matters:**
|
||||
- Console logs disappear on container restart
|
||||
- Database failures need persistent record for recovery
|
||||
- Enables post-mortem analysis of incidents
|
||||
- Orphan position detection can reference logs to reconstruct trades
|
||||
|
||||
**Implemented:** Nov 21, 2025 as part of 5-layer database protection system
|
||||
|
||||
### 2. Phantom Trade Auto-Closure System
|
||||
**Purpose:** Automatically close positions when size mismatch detected (position opened but wrong size)
|
||||
|
||||
**When triggered:**
|
||||
@@ -1065,7 +1105,7 @@ const driftSymbol = normalizeTradingViewSymbol(body.symbol)
|
||||
### Execute Trade (Production)
|
||||
```
|
||||
TradingView alert → n8n Parse Signal Enhanced (extracts metrics + timeframe)
|
||||
↓ /api/trading/check-risk [validates quality score ≥60, checks duplicates, per-symbol cooldown]
|
||||
↓ /api/trading/check-risk [validates quality score ≥81, checks duplicates, per-symbol cooldown]
|
||||
↓ /api/trading/execute
|
||||
↓ normalize symbol (SOLUSDT → SOL-PERP)
|
||||
↓ getMergedConfig()
|
||||
@@ -3217,6 +3257,185 @@ trade.realizedPnL += actualRealizedPnL // NOT: result.realizedPnL from SDK
|
||||
- **Git commit:** c607a66 "critical: Fix position close verification to prevent ghost positions"
|
||||
- **Lesson:** In DEX trading, always verify state changes actually propagated before updating local state
|
||||
|
||||
58. **5-Layer Database Protection System (IMPLEMENTED - Nov 21, 2025):**
|
||||
- **Purpose:** Bulletproof protection against untracked positions from database failures
|
||||
- **Trigger:** Investigation of potential missed trade (Nov 21, 00:40 CET) - turned out to be false alarm, but protection implemented anyway
|
||||
- **Real incident that sparked this:**
|
||||
* User concerned about missing database record after SL stop
|
||||
* Investigation found trade WAS saved: cmi82qg590001tn079c3qpw4r
|
||||
* SHORT SOL-PERP $133.69 → $134.67, -$89.17 loss
|
||||
* But concern was valid - what if database HAD failed?
|
||||
- **5-Layer Protection Architecture:**
|
||||
```typescript
|
||||
// LAYER 1: Persistent File Logger (lib/utils/persistent-logger.ts)
|
||||
class PersistentLogger {
|
||||
// Survives container restarts
|
||||
private logFile = '/app/logs/errors.log'
|
||||
|
||||
logError(context: string, error: any, metadata?: any): void {
|
||||
const entry = {
|
||||
timestamp: new Date().toISOString(),
|
||||
context,
|
||||
error: error.message,
|
||||
stack: error.stack,
|
||||
metadata
|
||||
}
|
||||
fs.appendFileSync(this.logFile, JSON.stringify(entry) + '\n')
|
||||
}
|
||||
}
|
||||
// Daily log rotation, 30-day retention
|
||||
|
||||
// LAYER 2: Database Save with Retry + Verification (lib/database/trades.ts)
|
||||
export async function createTrade(params: CreateTradeParams): Promise<Trade> {
|
||||
const maxRetries = 3
|
||||
const baseDelay = 1000 // 1s → 2s → 4s exponential backoff
|
||||
|
||||
for (let attempt = 1; attempt <= maxRetries; attempt++) {
|
||||
try {
|
||||
const trade = await prisma.trade.create({ data: tradeData })
|
||||
|
||||
// CRITICAL: Verify trade actually saved
|
||||
const verification = await prisma.trade.findUnique({
|
||||
where: { id: trade.id }
|
||||
})
|
||||
|
||||
if (!verification) {
|
||||
throw new Error('Trade created but verification query returned null')
|
||||
}
|
||||
|
||||
console.log(`✅ Trade saved and verified: ${trade.id}`)
|
||||
return trade
|
||||
} catch (error) {
|
||||
persistentLogger.logError('DATABASE_SAVE_FAILED', error, {
|
||||
attempt,
|
||||
tradeParams: params
|
||||
})
|
||||
|
||||
if (attempt < maxRetries) {
|
||||
const delay = baseDelay * Math.pow(2, attempt - 1)
|
||||
await new Promise(resolve => setTimeout(resolve, delay))
|
||||
continue
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// LAYER 3: Orphan Position Detection (lib/startup/init-position-manager.ts)
|
||||
async function detectOrphanPositions(): Promise<void> {
|
||||
// Runs on EVERY container startup
|
||||
const driftService = await initializeDriftService()
|
||||
const allPositions = await driftService.getAllPositions()
|
||||
|
||||
for (const position of allPositions) {
|
||||
if (Math.abs(position.size) < 0.01) continue
|
||||
|
||||
// Check if position exists in database
|
||||
const existingTrade = await prisma.trade.findFirst({
|
||||
where: {
|
||||
symbol: marketConfig.symbol,
|
||||
exitReason: null,
|
||||
status: { not: 'phantom' }
|
||||
}
|
||||
})
|
||||
|
||||
if (!existingTrade) {
|
||||
console.log(`🚨 ORPHAN POSITION DETECTED: ${marketConfig.symbol}`)
|
||||
|
||||
// Create retroactive database record
|
||||
const trade = await createTrade({
|
||||
symbol: marketConfig.symbol,
|
||||
direction: position.size > 0 ? 'long' : 'short',
|
||||
entryPrice: position.entryPrice,
|
||||
positionSizeUSD: Math.abs(position.size) * currentPrice,
|
||||
// ... other fields
|
||||
})
|
||||
|
||||
// Restore Position Manager monitoring
|
||||
const activeTrade: ActiveTrade = { /* ... */ }
|
||||
await positionManager.addTrade(activeTrade)
|
||||
|
||||
// Alert user via Telegram
|
||||
await sendTelegramAlert(
|
||||
`🚨 ORPHAN POSITION RECOVERED\n\n` +
|
||||
`${marketConfig.symbol} ${direction.toUpperCase()}\n` +
|
||||
`Entry: $${position.entryPrice}\n` +
|
||||
`Size: $${positionSizeUSD.toFixed(2)}\n\n` +
|
||||
`Position found on Drift but missing from database.\n` +
|
||||
`Retroactive record created and monitoring restored.`
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// LAYER 4: Critical Logging in Execute Endpoint (app/api/trading/execute/route.ts)
|
||||
try {
|
||||
await createTrade({...})
|
||||
} catch (dbError) {
|
||||
// Log with FULL trade details for recovery
|
||||
persistentLogger.logError('CRITICAL_DATABASE_FAILURE', dbError, {
|
||||
symbol: driftSymbol,
|
||||
direction: body.direction,
|
||||
entryPrice: openResult.entryPrice,
|
||||
positionSize: size,
|
||||
transactionSignature: openResult.transactionSignature,
|
||||
timestamp: new Date().toISOString(),
|
||||
// ALL data needed to reconstruct trade
|
||||
})
|
||||
|
||||
console.error('❌ CRITICAL: Failed to save trade to database:', dbError)
|
||||
return NextResponse.json({
|
||||
success: false,
|
||||
error: 'Database save failed - position unprotected',
|
||||
message: `Position opened on Drift but database save failed. ` +
|
||||
`ORPHAN DETECTION WILL CATCH THIS ON NEXT RESTART. ` +
|
||||
`Transaction: ${openResult.transactionSignature}`,
|
||||
}, { status: 500 })
|
||||
}
|
||||
|
||||
// LAYER 5: Infrastructure (Docker + File System)
|
||||
// docker-compose.yml:
|
||||
volumes:
|
||||
- ./logs:/app/logs # Persistent across container restarts
|
||||
|
||||
// logs/.gitkeep ensures directory exists in git
|
||||
```
|
||||
- **How it works together:**
|
||||
1. **Primary:** Retry logic catches transient database failures (network blips, brief outages)
|
||||
2. **Verification:** Ensures "success" means data actually persisted, not just query returned
|
||||
3. **Persistent logs:** If all retries fail, full trade details saved to disk (survives restarts)
|
||||
4. **Orphan detection:** On every container startup, queries Drift for untracked positions
|
||||
5. **Auto-recovery:** Creates missing database records, restores monitoring, alerts user
|
||||
- **Real-world validation (Nov 21, 2025):**
|
||||
* Investigated trade from 00:40:14 CET
|
||||
* Found in database: cmi82qg590001tn079c3qpw4r
|
||||
* SHORT SOL-PERP entry $133.69 → exit $134.67 (SL)
|
||||
* Closed 01:17:03 CET (37 minutes duration)
|
||||
* P&L: -$89.17
|
||||
* **No database failure occurred** - system working correctly
|
||||
* But protection now in place for future edge cases
|
||||
- **Files changed:**
|
||||
* `lib/utils/persistent-logger.ts` (NEW - 85 lines)
|
||||
* `lib/database/trades.ts` (retry + verification logic)
|
||||
* `lib/startup/init-position-manager.ts` (orphan detection + recovery)
|
||||
* `app/api/trading/execute/route.ts` (critical logging)
|
||||
* `docker-compose.yml` (logs volume mount)
|
||||
* `logs/.gitkeep` (NEW - ensures directory exists)
|
||||
- **Benefits:**
|
||||
* **Zero data loss:** Even if database fails, logs preserve trade details
|
||||
* **Auto-recovery:** Orphan detection runs on every restart, catches missed trades
|
||||
* **User transparency:** Telegram alerts explain what happened and what was fixed
|
||||
* **Developer visibility:** Persistent logs enable post-mortem analysis
|
||||
* **Confidence:** User can trust system for $816 → $600k journey
|
||||
- **Testing required:**
|
||||
* Manual trade execution to verify retry logic works
|
||||
* Container restart to verify orphan detection runs
|
||||
* Simulate database failure (stop postgres) to test error logging
|
||||
* Verify Telegram alerts send correctly
|
||||
- **Git commit:** "feat: Add comprehensive database save protection system"
|
||||
- **Deployment status:** Code committed and pushed, awaiting container rebuild
|
||||
- **Lesson:** In financial systems, "it worked this time" is not enough. Implement defense-in-depth protection for ALL critical operations, even when no failure has occurred yet. Better to have protection you never need than need protection you don't have.
|
||||
|
||||
## File Conventions
|
||||
|
||||
- **API routes:** `app/api/[feature]/[action]/route.ts` (Next.js 15 App Router)
|
||||
@@ -3498,7 +3717,7 @@ See `POSITION_SCALING_ROADMAP.md` for planned position management optimizations:
|
||||
- **v6:** HalfTrend + BarColor strategy (Nov 12-18) - baseline performance, prone to flip-flops in choppy markets
|
||||
- **v7:** v6 with toggle filters (deprecated - no fundamental improvements, just filter combinations)
|
||||
- **v8:** Money Line Sticky Trend (Nov 18+) - Current production indicator
|
||||
- Flip threshold: 0.8% (price must breach line by 0.8% before signal change)
|
||||
- Flip threshold: 0.6% (price must breach line by 0.6% before signal change)
|
||||
- Momentum confirmation: Tracks consecutive bars beyond threshold (anti-whipsaw)
|
||||
- Confirm bars: 0 (user-tuned, immediate signals with threshold protection)
|
||||
- Entry buffer: 0.2 ATR (filters wick flips)
|
||||
|
||||
Reference in New Issue
Block a user