werkzeuge/teamleader_test/.github/copilot-instructions.md

Copilot Instructions for Network Scanner Tool

🚨 MANDATORY: Read Before Making Changes

Documentation-First Workflow (ENFORCED)

BEFORE suggesting any changes:

1. Check docs/index.md - Find relevant documentation
2. Search docs/guides/troubleshooting.md - Known issues and solutions
3. Review CONTRIBUTING.md - Development workflow and standards
4. Verify docs/project-status.md - Current feature status

AFTER making changes:

1. Update relevant documentation in docs/ directory
2. Add troubleshooting entry if fixing a bug
3. Update docs/project-status.md if feature status changes
4. Never create ad-hoc markdown files - use existing docs/ structure

---

Quick Overview

This is a containerized full-stack network scanning and visualization tool:

• Backend: Python 3.11 + FastAPI (async) + SQLite with SQLAlchemy ORM
• Frontend: React 18 + TypeScript + TailwindCSS + React Flow visualization
• Deployment: Docker Compose with nginx reverse proxy
• Purpose: Discover hosts, detect open ports/services, visualize network topology with Visio-style interactive diagrams

Key Path: /home/rwiegand/Nextcloud/entwicklung/Werkzeuge/teamleader_test/

---

Critical Architecture Patterns

1. Data Flow: Backend → Frontend

Database (SQLite) → SQLAlchemy ORM → Pydantic Schemas → REST API/JSON
                                                              ↓
                                                    Frontend React Components
                                                    (via hooks + axios)

Important: Schema changes in app/schemas.py must match TypeScript types in frontend/src/types/api.ts. Mismatch causes 500 errors or type failures.

2. Session Management (Critical Bug Source)

Problem: Async background tasks can't use scoped sessions—they close after request completion.

Solution (used in app/api/endpoints/scans.py):

# ❌ DON'T: Use shared session from request
background_tasks.add_task(scan_service.execute_scan, scan_id, db)

# ✅ DO: Create new session in background task wrapper
def scan_wrapper(scan_id: int):
    db = SessionLocal()  # New session!
    try:
        scan_service.execute_scan(scan_id, db)
    finally:
        db.close()

background_tasks.add_task(scan_wrapper, scan_id)

3. Database Constraints (Critical)

Rule: Always commit() and refresh() host objects before adding dependent records (services, connections).

Example (from app/services/scan_service.py):

host = self._get_or_create_host(ip)
self.db.commit()  # ← CRITICAL: Ensure host.id is set
self.db.refresh(host)

# NOW safe to add services
service = Service(host_id=host.id, port=80)  # host.id exists
self.db.add(service)

4. WebSocket Broadcasting

Located in app/api/endpoints/websocket.py. Pattern:

# Broadcast to all connected clients
await connection_manager.broadcast({
    "type": "scan_progress",
    "data": {"progress": 0.75, "current_host": "192.168.1.5"}
})

Integrated in ScanService to push real-time updates during scans.

---

Project Structure Deep Dive

Backend (app/)

Component	Purpose	Critical Points
models.py	SQLAlchemy ORM	Use Connection.extra_data not .metadata (reserved). Check cascade rules.
schemas.py	Pydantic validation	Must have .model_rebuild() for forward refs (e.g., HostDetailResponse).
services/scan_service.py	Scan orchestration	Uses WebSocket callbacks. Commit after host creation.
services/topology_service.py	Graph generation	Simplified to return flat TopologyNode/TopologyEdge structures.
api/endpoints/	REST routes	Use SessionLocal() wrapper for background tasks.
scanner/	Network scanning	Socket-based (default, no root) + optional nmap integration.

Frontend (frontend/src/)

Component	Purpose	Critical Points
pages/Dashboard.tsx	Home + scan control	Displays real-time progress bar via WebSocket.
pages/NetworkPage.tsx	Network map page	Integrated click handler via HostDetailsPanel.
components/NetworkMap.tsx	React Flow visualization	Creates nodes with circular layout, handles node clicks.
components/HostDetailsPanel.tsx	Host detail sidebar	Right-side panel, fetches via hostApi.getHost().
hooks/useTopology.ts	Topology data	Fetches from /api/topology.
services/api.ts	Axios client	Base URL from VITE_API_URL env var.
types/api.ts	TypeScript interfaces	Must match backend schemas exactly.

---

Build & Deployment

Development

cd teamleader_test
docker compose up -d --build
# Frontend: http://localhost (nginx reverse proxy)
# Backend API: http://localhost:8000
# Docs: http://localhost:8000/docs

Common Issues & Fixes

Error	Cause	Fix
500 Error on /api/topology	TopologyNode missing position field	Remove _calculate_layout() call (positions not in simplified schema)
Frontend type mismatch	Backend returns network_range, frontend expects target	Update frontend/src/types/api.ts interface
DetachedInstanceError in async task	Session closed after request	Use SessionLocal() wrapper in background task
NOT NULL constraint failed: host_id	Services added before host committed	Add db.commit() + db.refresh(host) after creation
Scan progress not updating	WebSocket not connected	Verify useWebSocket() hook in Dashboard, check /api/ws endpoint

---

Database Schema (Key Tables)

# 5 main tables:
Scan        # Scan operations (status, network_range, timestamps)
Host        # Discovered hosts (ip_address UNIQUE, status, hostname)
Service     # Open ports (host_id FK, port, state, banner)
Connection  # Relationships (source_host_id, target_host_id, confidence)
scan_hosts  # Many-to-many (scan_id, host_id)

Column rename: Connection.metadata → Connection.extra_data (metadata is SQLAlchemy reserved).

---

API Response Contract

Topology Response (/api/topology)

{
  nodes: TopologyNode[],    // id, ip, hostname, type, status, service_count, connections
  edges: TopologyEdge[],    // source, target, type, confidence
  statistics: {
    total_nodes: number,
    total_edges: number,
    isolated_nodes: number,
    avg_connections: number
  }
}

Scan Start Response (POST /api/scans/start)

{
  scan_id: number,
  message: string,
  status: 'pending' | 'running'
}

Note: Frontend uses scan.scan_id not scan.id.

---

Docker Compose Setup

• Backend container: network-scanner-backend on port 8000
• Frontend container: network-scanner-frontend on port 80 (nginx)
• Shared volume: ./data/ for SQLite database
• Network: scanner-network (bridge)

# Rebuild after code changes
docker compose up -d --build

# View logs
docker compose logs backend --tail=50
docker compose logs frontend --tail=50

# Stop all
docker compose down

---

Common Development Tasks

Add New API Endpoint

1. Create handler in app/api/endpoints/new_feature.py
2. Add to app/api/__init__.py router
3. Define request/response Pydantic models in app/schemas.py
4. Add TypeScript type in frontend/src/types/api.ts
5. Add service method in frontend/src/services/api.ts
6. Create React hook or component to call it

Modify Database Schema

1. Update SQLAlchemy model in app/models.py
2. Delete network_scanner.db (dev only) or create migration
3. Restart backend: docker compose up -d --build backend

Add Frontend Component

1. Create .tsx file in frontend/src/components/
2. Import API types from frontend/src/types/api.ts
3. Use Axios via frontend/src/services/api.ts
4. Integrate into page/parent component
5. Rebuild: docker compose up -d --build frontend

---

Debugging Checklist

• Backend not responding? Check docker compose logs backend for errors
• Frontend showing blank page? Open browser console (F12) for errors, check VITE_API_URL
• Type errors on build? Verify frontend/src/types/api.ts matches backend response
• Database locked? Only one writer at a time with SQLite—check for stuck processes
• WebSocket not connecting? Verify /api/ws endpoint exists, check backend logs
• Scan never completes? Check docker compose logs backend for exceptions, verify cancel_requested flag

---

Conventions

1. Logging: Use logger.info() for key events, logger.error() for failures
2. Database: Use SQLAlchemy relationships, avoid raw SQL
3. API Responses: Always wrap in Pydantic models, validate input
4. Async: Use asyncio.Task for background scans, new SessionLocal() for session
5. Frontend: Use TypeScript strict mode, never bypass type checking
6. Styling: TailwindCSS only, no inline CSS (except React Flow inline styles)
7. Comments: Document "why", not "what"—code is self-documenting

---

Key Files to Reference

• Architecture overview: docs/architecture/overview.md (comprehensive design decisions)
• Database models: app/models.py
• API schemas: app/schemas.py
• Scan orchestration: app/services/scan_service.py (async patterns, WebSocket integration)
• Topology service: app/services/topology_service.py (graph generation logic)
• Frontend types: frontend/src/types/api.ts
• Network Map: frontend/src/components/NetworkMap.tsx (React Flow usage, click handling)
• Docker setup: docker-compose.yml

---

Testing Commands

# Health check
curl -s http://localhost/health | jq .

# Start scan (quick 192.168.1.0/24)
curl -X POST http://localhost:8000/api/scans/start \
  -H "Content-Type: application/json" \
  -d '{"network_range":"192.168.1.0/24","scan_type":"quick"}'

# Get topology
curl -s http://localhost/api/topology | jq '.nodes | length'

# List hosts
curl -s http://localhost/api/hosts | jq 'length'

# Get specific host
curl -s http://localhost/api/hosts/1 | jq .

---

Version Info

• Python: 3.11
• FastAPI: 0.109.0
• React: 18.2.0
• TypeScript: 5.2.2
• Last Updated: December 4, 2025

---

Contact & Escalation

For questions on:

• Architecture: See ARCHITECTURE.md
• Build issues: Check docker compose logs backend/frontend
• Type errors: Verify app/schemas.py ↔ frontend/src/types/api.ts alignment
• Runtime crashes: Enable DEBUG=True in .env, check logs directory