trading_bot_v4/.github/copilot-instructions.md

# AI Agent Instructions for Trading Bot v4

## Architecture Overview

**Type:** Autonomous cryptocurrency trading bot with Next.js 15 frontend + Solana/Drift Protocol backend

**Data Flow:** TradingView → n8n webhook → Next.js API → Drift Protocol (Solana DEX) → Real-time monitoring → Auto-exit

**Key Design Principle:** Dual-layer redundancy - every trade has both on-chain orders (Drift) AND software monitoring (Position Manager) as backup.

**Exit Strategy:** Three-tier scaling system:
- TP1 at +0.4%: Close 75% (configurable via `TAKE_PROFIT_1_SIZE_PERCENT`)
- TP2 at +0.7%: Close 80% of remaining = 20% total (configurable via `TAKE_PROFIT_2_SIZE_PERCENT`)
- Runner: 5% remaining with 0.3% trailing stop (configurable via `TRAILING_STOP_PERCENT`)

**Per-Symbol Configuration:** SOL and ETH have independent enable/disable toggles and position sizing:
- `SOLANA_ENABLED`, `SOLANA_POSITION_SIZE`, `SOLANA_LEVERAGE` (defaults: true, $210, 10x)
- `ETHEREUM_ENABLED`, `ETHEREUM_POSITION_SIZE`, `ETHEREUM_LEVERAGE` (defaults: true, $4, 1x)
- BTC and other symbols fall back to global settings (`MAX_POSITION_SIZE_USD`, `LEVERAGE`)
- **Priority:** Per-symbol ENV → Market config → Global ENV → Defaults

**Signal Quality System:** Filters trades based on 5 metrics (ATR, ADX, RSI, volumeRatio, pricePosition) scored 0-100. Only trades scoring 60+ are executed. Scores stored in database for future optimization.

**Timeframe-Aware Scoring:** Signal quality thresholds adjust based on timeframe (5min vs daily):
- 5min: ADX 12+ trending (vs 18+ for daily), ATR 0.2-0.7% healthy (vs 0.4%+ for daily)
- Anti-chop filter: -20 points for extreme sideways (ADX <10, ATR <0.25%, Vol <0.9x)
- Pass `timeframe` param to `scoreSignalQuality()` from TradingView alerts (e.g., `timeframe: "5"`)

**MAE/MFE Tracking:** Every trade tracks Maximum Favorable Excursion (best profit %) and Maximum Adverse Excursion (worst loss %) updated every 2s. Used for data-driven optimization of TP/SL levels.

**Manual Trading via Telegram:** Send plain-text messages like `long sol`, `short eth`, `long btc` to open positions instantly (bypasses n8n, calls `/api/trading/execute` directly with preset healthy metrics).

## Critical Components

### 1. Signal Quality Scoring (`lib/trading/signal-quality.ts`)
**Purpose:** Unified quality validation system that scores trading signals 0-100 based on 5 market metrics

**Timeframe-aware thresholds:**
```typescript
scoreSignalQuality({ 
  atr, adx, rsi, volumeRatio, pricePosition, 
  timeframe?: string // "5" for 5min, undefined for higher timeframes
})
```

**5min chart adjustments:**
- ADX healthy range: 12-22 (vs 18-30 for daily)
- ATR healthy range: 0.2-0.7% (vs 0.4%+ for daily)
- Anti-chop filter: -20 points for extreme sideways (ADX <10, ATR <0.25%, Vol <0.9x)

**Price position penalties (all timeframes):**
- Long at 90-95%+ range: -15 to -30 points (chasing highs)
- Short at <5-10% range: -15 to -30 points (chasing lows)
- Prevents flip-flop losses from entering range extremes

**Key behaviors:**
- Returns score 0-100 and detailed breakdown object
- Minimum score 60 required to execute trade
- Called by both `/api/trading/check-risk` and `/api/trading/execute`
- Scores saved to database for post-trade analysis

### 2. Position Manager (`lib/trading/position-manager.ts`)
**Purpose:** Software-based monitoring loop that checks prices every 2 seconds and closes positions via market orders

**Singleton pattern:** Always use `getInitializedPositionManager()` - never instantiate directly
```typescript
const positionManager = await getInitializedPositionManager()
await positionManager.addTrade(activeTrade)
```

**Key behaviors:**
- Tracks `ActiveTrade` objects in a Map
- Three-tier exits: TP1 (75%), TP2 (80% of remaining), Runner (with trailing stop)
- Dynamic SL adjustments: Moves to breakeven after TP1, locks profit at +1.2%
- **On-chain order synchronization:** After TP1 hits, calls `cancelAllOrders()` then `placeExitOrders()` with updated SL price at breakeven
- Trailing stop: Activates after TP2, tracks `peakPrice` and trails by configured %
- Closes positions via `closePosition()` market orders when targets hit
- Acts as backup if on-chain orders don't fill
- State persistence: Saves to database, restores on restart via `configSnapshot.positionManagerState`
- **Grace period for new trades:** Skips "external closure" detection for positions <30 seconds old (Drift positions take 5-10s to propagate)
- **Exit reason detection:** Uses trade state flags (`tp1Hit`, `tp2Hit`) and realized P&L to determine exit reason, NOT current price (avoids misclassification when price moves after order fills)

### 3. Telegram Bot (`telegram_command_bot.py`)
**Purpose:** Python-based Telegram bot for manual trading commands and position status monitoring

**Manual trade commands via plain text:**
```python
# User sends plain text message (not slash commands)
"long sol"   → Opens SOL-PERP long position
"short eth"  → Opens ETH-PERP short position
"long btc"   → Opens BTC-PERP long position
```

**Key behaviors:**
- MessageHandler processes all text messages (not just commands)
- Maps user-friendly symbols (sol, eth, btc) to Drift format (SOL-PERP, etc.)
- Calls `/api/trading/execute` directly with preset healthy metrics (ATR=1.0, ADX=25, RSI=50, volumeRatio=1.2)
- Bypasses n8n workflow and TradingView requirements
- 60-second timeout for API calls
- Responds with trade confirmation or error message

**Status command:**
```python
/status → Returns JSON of open positions from Drift
```

**Implementation details:**
- Uses `python-telegram-bot` library
- Deployed via `docker-compose.telegram-bot.yml`
- Requires `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHANNEL_ID` in .env
- API calls to `http://trading-bot:3000/api/trading/execute`

**Drift client integration:**
- Singleton pattern: Use `initializeDriftService()` and `getDriftService()` - maintains single connection
```typescript
const driftService = await initializeDriftService()
const health = await driftService.getAccountHealth()
```
- Wallet handling: Supports both JSON array `[91,24,...]` and base58 string formats from Phantom wallet

### 4. Order Placement (`lib/drift/orders.ts`)
**Critical functions:** 
- `openPosition()` - Opens market position with transaction confirmation
- `closePosition()` - Closes position with transaction confirmation
- `placeExitOrders()` - Places TP/SL orders on-chain

**CRITICAL: Transaction Confirmation Pattern**
Both `openPosition()` and `closePosition()` MUST confirm transactions on-chain:
```typescript
const txSig = await driftClient.placePerpOrder(orderParams)
console.log('⏳ Confirming transaction on-chain...')
const connection = driftService.getConnection()
const confirmation = await connection.confirmTransaction(txSig, 'confirmed')

if (confirmation.value.err) {
  throw new Error(`Transaction failed: ${JSON.stringify(confirmation.value.err)}`)
}
console.log('✅ Transaction confirmed on-chain')
```
Without this, the SDK returns signatures for transactions that never execute, causing phantom trades/closes.

**Dual Stop System** (USE_DUAL_STOPS=true):
```typescript
// Soft stop: TRIGGER_LIMIT at -1.5% (avoids wicks)
// Hard stop: TRIGGER_MARKET at -2.5% (guarantees exit)
```

**Order types:**
- Entry: MARKET (immediate execution)
- TP1/TP2: LIMIT reduce-only orders
- Soft SL: TRIGGER_LIMIT reduce-only
- Hard SL: TRIGGER_MARKET reduce-only

### 5. Database (`lib/database/trades.ts` + `prisma/schema.prisma`)
**Purpose:** PostgreSQL via Prisma ORM for trade history and analytics

**Models:** Trade, PriceUpdate, SystemEvent, DailyStats

**Singleton pattern:** Use `getPrismaClient()` - never instantiate PrismaClient directly

**Key functions:**
- `createTrade()` - Save trade after execution (includes dual stop TX signatures + signalQualityScore)
- `updateTradeExit()` - Record exit with P&L
- `addPriceUpdate()` - Track price movements (called by Position Manager)
- `getTradeStats()` - Win rate, profit factor, avg win/loss
- `getLastTrade()` - Fetch most recent trade for analytics dashboard

**Important fields:**
- `signalQualityScore` (Int?) - 0-100 score for data-driven optimization
- `maxFavorableExcursion` / `maxAdverseExcursion` - Track best/worst P&L during trade lifetime
- `maxFavorablePrice` / `maxAdversePrice` - Track prices at MFE/MAE points
- `configSnapshot` (Json) - Stores Position Manager state for crash recovery
- `atr`, `adx`, `rsi`, `volumeRatio`, `pricePosition` - Context metrics from TradingView

**Per-symbol functions:**
- `getLastTradeTimeForSymbol(symbol)` - Get last trade time for specific coin (enables per-symbol cooldown)
- Each coin (SOL/ETH/BTC) has independent cooldown timer to avoid missed opportunities

## Configuration System

**Three-layer merge:**
1. `DEFAULT_TRADING_CONFIG` (config/trading.ts)
2. Environment variables (.env) via `getConfigFromEnv()`
3. Runtime overrides via `getMergedConfig(overrides)`

**Always use:** `getMergedConfig()` to get final config - never read env vars directly in business logic

**Per-symbol position sizing:** Use `getPositionSizeForSymbol(symbol, config)` which returns `{ size, leverage, enabled }`
```typescript
const { size, leverage, enabled } = getPositionSizeForSymbol('SOL-PERP', config)
if (!enabled) {
  return NextResponse.json({ success: false, error: 'Symbol trading disabled' }, { status: 400 })
}
```

**Symbol normalization:** TradingView sends "SOLUSDT" → must convert to "SOL-PERP" for Drift
```typescript
const driftSymbol = normalizeTradingViewSymbol(body.symbol)
```

## API Endpoints Architecture

**Authentication:** All `/api/trading/*` endpoints (except `/test`) require `Authorization: Bearer API_SECRET_KEY`

**Pattern:** Each endpoint follows same flow:
1. Auth check
2. Get config via `getMergedConfig()`
3. Initialize Drift service
4. Check account health
5. Execute operation
6. Save to database
7. Add to Position Manager if applicable

**Key endpoints:**
- `/api/trading/execute` - Main entry point from n8n (production, requires auth)
- `/api/trading/check-risk` - Pre-execution validation (duplicate check, quality score, **per-symbol cooldown**, rate limits, **symbol enabled check**)
- `/api/trading/test` - Test trades from settings UI (no auth required, **respects symbol enable/disable**)
- `/api/trading/close` - Manual position closing
- `/api/trading/positions` - Query open positions from Drift
- `/api/settings` - Get/update config (writes to .env file, **includes per-symbol settings**)
- `/api/analytics/last-trade` - Fetch most recent trade details for dashboard (includes quality score)
- `/api/restart` - Create restart flag for watch-restart.sh script

## Critical Workflows

### Execute Trade (Production)
```
TradingView alert → n8n Parse Signal Enhanced (extracts metrics + timeframe)
  ↓ /api/trading/check-risk [validates quality score ≥60, checks duplicates, per-symbol cooldown]
  ↓ /api/trading/execute
  ↓ normalize symbol (SOLUSDT → SOL-PERP)
  ↓ getMergedConfig()
  ↓ getPositionSizeForSymbol() [check if symbol enabled + get sizing]
  ↓ openPosition() [MARKET order]
  ↓ calculate dual stop prices if enabled
  ↓ placeExitOrders() [on-chain TP1/TP2/SL orders]
  ↓ scoreSignalQuality({ ..., timeframe }) [compute 0-100 score with timeframe-aware thresholds]
  ↓ createTrade() [save to database with signalQualityScore]
  ↓ positionManager.addTrade() [start monitoring]
```

### Position Monitoring Loop
```
Position Manager every 2s:
  ↓ Verify on-chain position still exists (detect external closures)
  ↓ getPythPriceMonitor().getLatestPrice()
  ↓ Calculate current P&L and update MAE/MFE metrics
  ↓ Check emergency stop (-2%) → closePosition(100%)
  ↓ Check SL hit → closePosition(100%)
  ↓ Check TP1 hit → closePosition(75%), cancelAllOrders(), placeExitOrders() with SL at breakeven
  ↓ Check profit lock trigger (+1.2%) → move SL to +configured%
  ↓ Check TP2 hit → closePosition(80% of remaining), activate runner
  ↓ Check trailing stop (if runner active) → adjust SL dynamically based on peakPrice
  ↓ addPriceUpdate() [save to database every N checks]
  ↓ saveTradeState() [persist Position Manager state + MAE/MFE for crash recovery]
```

### Settings Update
```
Web UI → /api/settings POST
  ↓ Validate new settings
  ↓ Write to .env file using string replacement
  ↓ Return success
  ↓ User clicks "Restart Bot" → /api/restart
  ↓ Creates /tmp/trading-bot-restart.flag
  ↓ watch-restart.sh detects flag
  ↓ Executes: docker restart trading-bot-v4
```

## Docker Context

**Multi-stage build:** deps → builder → runner (Node 20 Alpine)

**Critical Dockerfile steps:**
1. Install deps with `npm install --production`
2. Copy source and `npx prisma generate` (MUST happen before build)
3. `npm run build` (Next.js standalone output)
4. Runner stage copies standalone + static + node_modules + Prisma client

**Container networking:**
- External: `trading-bot-v4` on port 3001
- Internal: Next.js on port 3000
- Database: `trading-bot-postgres` on 172.28.0.0/16 network

**DATABASE_URL caveat:** Use `trading-bot-postgres` (container name) in .env for runtime, but `localhost:5432` for Prisma CLI migrations from host

## Project-Specific Patterns

### 1. Singleton Services
Never create multiple instances - always use getter functions:
```typescript
const driftService = await initializeDriftService() // NOT: new DriftService()
const positionManager = getPositionManager()        // NOT: new PositionManager()
const prisma = getPrismaClient()                     // NOT: new PrismaClient()
```

### 2. Price Calculations
Direction matters for long vs short:
```typescript
function calculatePrice(entry: number, percent: number, direction: 'long' | 'short') {
  if (direction === 'long') {
    return entry * (1 + percent / 100)  // Long: +1% = higher price
  } else {
    return entry * (1 - percent / 100)  // Short: +1% = lower price
  }
}
```

### 3. Error Handling
Database failures should not fail trades - always wrap in try/catch:
```typescript
try {
  await createTrade(params)
  console.log('💾 Trade saved to database')
} catch (dbError) {
  console.error('❌ Failed to save trade:', dbError)
  // Don't fail the trade if database save fails
}
```

### 4. Reduce-Only Orders
All exit orders MUST be reduce-only (can only close, not open positions):
```typescript
const orderParams = {
  reduceOnly: true,  // CRITICAL for TP/SL orders
  // ... other params
}
```

## Testing Commands

```bash
# Local development
npm run dev

# Build production
npm run build && npm start

# Docker build and restart
docker compose build trading-bot
docker compose up -d --force-recreate trading-bot
docker logs -f trading-bot-v4

# Database operations
npx prisma generate                                    # Generate client
DATABASE_URL="postgresql://...@localhost:5432/..." npx prisma migrate dev
docker exec trading-bot-postgres psql -U postgres -d trading_bot_v4 -c "\dt"

# Test trade from UI
# Go to http://localhost:3001/settings
# Click "Test LONG" or "Test SHORT"
```

## Common Pitfalls

1. **Prisma not generated in Docker:** Must run `npx prisma generate` in Dockerfile BEFORE `npm run build`

2. **Wrong DATABASE_URL:** Container runtime needs `trading-bot-postgres`, Prisma CLI from host needs `localhost:5432`

3. **Symbol format mismatch:** Always normalize with `normalizeTradingViewSymbol()` before calling Drift (applies to ALL endpoints including `/api/trading/close`)

4. **Missing reduce-only flag:** Exit orders without `reduceOnly: true` can accidentally open new positions

5. **Singleton violations:** Creating multiple DriftClient or Position Manager instances causes connection/state issues

6. **Type errors with Prisma:** The Trade type from Prisma is only available AFTER `npx prisma generate` - use explicit types or `// @ts-ignore` carefully

7. **Quality score duplication:** Signal quality calculation exists in BOTH `check-risk` and `execute` endpoints - keep logic synchronized

8. **Runner configuration confusion:** 
   - `TAKE_PROFIT_1_SIZE_PERCENT=75` means "close 75% at TP1" (not "keep 75%")
   - `TAKE_PROFIT_2_SIZE_PERCENT=80` means "close 80% of REMAINING" (not of original)
   - Actual runner size = (100 - TP1%) × (100 - TP2%) / 100 = 5% with defaults

9. **Transaction confirmation CRITICAL:** Both `openPosition()` AND `closePosition()` MUST call `connection.confirmTransaction()` after `placePerpOrder()`. Without this, the SDK returns transaction signatures that aren't confirmed on-chain, causing "phantom trades" or "phantom closes". Always check `confirmation.value.err` before proceeding.

10. **Execution order matters:** When creating trades via API endpoints, the order MUST be:
    1. Open position + place exit orders
    2. Save to database (`createTrade()`)
    3. Add to Position Manager (`positionManager.addTrade()`)
    
    If Position Manager is added before database save, race conditions occur where monitoring checks before the trade exists in DB.

11. **New trade grace period:** Position Manager skips "external closure" detection for trades <30 seconds old because Drift positions take 5-10 seconds to propagate after opening. Without this grace period, new positions are immediately detected as "closed externally" and cancelled.

12. **Drift minimum position sizes:** Actual minimums differ from documentation:
    - SOL-PERP: 0.1 SOL (~$5-15 depending on price)
    - ETH-PERP: 0.01 ETH (~$38-40 at $4000/ETH)
    - BTC-PERP: 0.0001 BTC (~$10-12 at $100k/BTC)
    
    Always calculate: `minOrderSize × currentPrice` must exceed Drift's $4 minimum. Add buffer for price movement.

13. **Exit reason detection bug:** Position Manager was using current price to determine exit reason, but on-chain orders filled at a DIFFERENT price in the past. Now uses `trade.tp1Hit` / `trade.tp2Hit` flags and realized P&L to correctly identify whether TP1, TP2, or SL triggered. Prevents profitable trades being mislabeled as "SL" exits.

14. **Per-symbol cooldown:** Cooldown period is per-symbol, NOT global. ETH trade at 10:00 does NOT block SOL trade at 10:01. Each coin (SOL/ETH/BTC) has independent cooldown timer to avoid missing opportunities on different assets.

15. **Timeframe-aware scoring crucial:** Signal quality thresholds MUST adjust for 5min vs higher timeframes:
    - 5min charts naturally have lower ADX (12-22 healthy) and ATR (0.2-0.7% healthy) than daily charts
    - Without timeframe awareness, valid 5min breakouts get blocked as "low quality"
    - Anti-chop filter applies -20 points for extreme sideways regardless of timeframe
    - Always pass `timeframe` parameter from TradingView alerts to `scoreSignalQuality()`

16. **Price position chasing causes flip-flops:** Opening longs at 90%+ range or shorts at <10% range reliably loses money:
    - Database analysis showed overnight flip-flop losses all had price position 9-94% (chasing extremes)
    - These trades had valid ADX (16-18) but entered at worst possible time
    - Quality scoring now penalizes -15 to -30 points for range extremes
    - Prevents rapid reversals when price is already overextended

17. **TradingView ADX minimum for 5min:** Set ADX filter to 15 (not 20+) in TradingView alerts for 5min charts:
    - Higher timeframes can use ADX 20+ for strong trends
    - 5min charts need lower threshold to catch valid breakouts
    - Bot's quality scoring provides second-layer filtering with context-aware metrics
    - Two-stage filtering (TradingView + bot) prevents both overtrading and missing valid signals

## File Conventions

- **API routes:** `app/api/[feature]/[action]/route.ts` (Next.js 15 App Router)
- **Services:** `lib/[service]/[module].ts` (drift, pyth, trading, database)
- **Config:** Single source in `config/trading.ts` with env merging
- **Types:** Define interfaces in same file as implementation (not separate types directory)
- **Console logs:** Use emojis for visual scanning: 🎯 🚀 ✅ ❌ 💰 📊 🛡️

## Per-Symbol Trading Controls

**Purpose:** Independent enable/disable toggles and position sizing for SOL and ETH to support different trading strategies (e.g., ETH for data collection at minimal size, SOL for profit generation).

**Configuration Priority:**
1. **Per-symbol ENV vars** (highest priority)
   - `SOLANA_ENABLED`, `SOLANA_POSITION_SIZE`, `SOLANA_LEVERAGE`
   - `ETHEREUM_ENABLED`, `ETHEREUM_POSITION_SIZE`, `ETHEREUM_LEVERAGE`
2. **Market-specific config** (from `MARKET_CONFIGS` in config/trading.ts)
3. **Global ENV vars** (fallback for BTC and other symbols)
   - `MAX_POSITION_SIZE_USD`, `LEVERAGE`
4. **Default config** (lowest priority)

**Settings UI:** `app/settings/page.tsx` has dedicated sections:
- 💎 Solana section: Toggle + position size + leverage + risk calculator
- ⚡ Ethereum section: Toggle + position size + leverage + risk calculator
- 💰 Global fallback: For BTC-PERP and future symbols

**Example usage:**
```typescript
// In execute/test endpoints
const { size, leverage, enabled } = getPositionSizeForSymbol(driftSymbol, config)
if (!enabled) {
  return NextResponse.json({
    success: false,
    error: 'Symbol trading disabled'
  }, { status: 400 })
}
```

**Test buttons:** Settings UI has symbol-specific test buttons:
- 💎 Test SOL LONG/SHORT (disabled when `SOLANA_ENABLED=false`)
- ⚡ Test ETH LONG/SHORT (disabled when `ETHEREUM_ENABLED=false`)

## When Making Changes

1. **Adding new config:** Update DEFAULT_TRADING_CONFIG + getConfigFromEnv() + .env file
2. **Adding database fields:** Update prisma/schema.prisma → `npx prisma migrate dev` → `npx prisma generate` → rebuild Docker
3. **Changing order logic:** Test with DRY_RUN=true first, use small position sizes ($10)
4. **API endpoint changes:** Update both endpoint + corresponding n8n workflow JSON (Check Risk and Execute Trade nodes)
5. **Docker changes:** Rebuild with `docker compose build trading-bot` then restart container
6. **Modifying quality score logic:** Update BOTH `/api/trading/check-risk` and `/api/trading/execute` endpoints, ensure timeframe-aware thresholds are synchronized
7. **Exit strategy changes:** Modify Position Manager logic + update on-chain order placement in `placeExitOrders()`
8. **TradingView alert changes:** Ensure alerts pass `timeframe` field (e.g., `"timeframe": "5"`) to enable proper signal quality scoring

## Development Roadmap

See `POSITION_SCALING_ROADMAP.md` for planned optimizations:
- **Phase 1 (CURRENT):** Collect data with quality scores (20-50 trades needed)
- **Phase 2:** ATR-based dynamic targets (adapt to volatility)
- **Phase 3:** Signal quality-based scaling (high quality = larger runners)
- **Phase 4:** Direction-based optimization (shorts vs longs have different performance)
- **Phase 5:** Optimize runner size (5% → 10-25%) and trailing stop (0.3% fixed → ATR-based)
- **Phase 6:** ML-based exit prediction (future)

**Data-driven approach:** Each phase requires validation through SQL analysis before implementation. No premature optimization.

## Integration Points

- **n8n:** Expects exact response format from `/api/trading/execute` (see n8n-complete-workflow.json)
- **Drift Protocol:** Uses SDK v2.75.0 - check docs at docs.drift.trade for API changes
- **Pyth Network:** WebSocket + HTTP fallback for price feeds (handles reconnection)
- **PostgreSQL:** Version 16-alpine, must be running before bot starts

---

**Key Mental Model:** Think of this as two parallel systems (on-chain orders + software monitoring) working together. The Position Manager is the "backup brain" that constantly watches and acts if on-chain orders fail. Both write to the same database for complete trade history.