feat: implement strategy metadata and dashboard simulation panel
- Added display_name and description to BaseStrategy - Updated MA44 and MA125 strategies with metadata - Added /api/v1/strategies endpoint for dynamic discovery - Added Strategy Simulation panel to dashboard with date picker and tooltips - Implemented JS polling for backtest results in dashboard - Added performance test scripts and DB connection guide - Expanded indicator config to all 15 timeframes
This commit is contained in:
BTC Bot
273
AGENTS.md
273
AGENTS.md
@ -1,223 +1,128 @@
|
||||
# AGENTS.md - AI Coding Assistant Guidelines
|
||||
|
||||
## Project Overview
|
||||
BTC Accumulation Bot - Data Collection Phase. High-performance async data collection
|
||||
system for cbBTC on Hyperliquid with TimescaleDB storage. Python 3.11, asyncio,
|
||||
FastAPI, asyncpg, WebSockets.
|
||||
BTC Accumulation Bot - Data Collection & Backtesting Phase. High-performance system for
|
||||
cbBTC on Hyperliquid with TimescaleDB. Core components: Data Collector (WS),
|
||||
Indicator Engine (SMA, etc.), Brain (Decision Logic), and Backtester.
|
||||
|
||||
## Build/Run Commands
|
||||
|
||||
### Docker (Primary deployment - Synology DS218+)
|
||||
```bash
|
||||
# Build and start all services (timescaledb, data_collector, api_server)
|
||||
cd docker && docker-compose up -d --build
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f data_collector
|
||||
docker-compose logs -f api_server
|
||||
|
||||
# Full deploy (creates dirs, pulls, builds, starts)
|
||||
bash scripts/deploy.sh
|
||||
cd docker && docker-compose up -d --build # Build and start all services
|
||||
docker-compose logs -f data_collector # View logs
|
||||
bash scripts/deploy.sh # Full deploy
|
||||
```
|
||||
|
||||
### Development
|
||||
```bash
|
||||
# API server (requires DB running)
|
||||
cd src/api && uvicorn server:app --reload --host 0.0.0.0 --port 8000
|
||||
# Docs: http://localhost:8000/docs | Dashboard: http://localhost:8000/dashboard
|
||||
|
||||
# Data collector
|
||||
cd src/data_collector && python -m data_collector.main
|
||||
```
|
||||
|
||||
### Testing
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Run a specific test file
|
||||
pytest tests/data_collector/test_websocket_client.py
|
||||
|
||||
# Run a single test by name
|
||||
pytest tests/data_collector/test_websocket_client.py::test_websocket_connection -v
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=src --cov-report=html
|
||||
```
|
||||
Note: The tests/ directory structure exists but test files have not been written yet.
|
||||
When creating tests, use pytest with pytest-asyncio for async test support.
|
||||
|
||||
### Linting & Formatting
|
||||
```bash
|
||||
# No config files exist for these tools; use these flags:
|
||||
flake8 src/ --max-line-length=100 --extend-ignore=E203,W503
|
||||
black --check src/ # Check formatting
|
||||
black src/ # Auto-format
|
||||
mypy src/ --ignore-missing-imports
|
||||
pytest # All tests
|
||||
pytest tests/data_collector/test_websocket_client.py # Single file
|
||||
pytest --cov=src --cov-report=html # With coverage
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
```
|
||||
src/
|
||||
├── data_collector/ # WebSocket client, buffer, database
|
||||
│ ├── __init__.py
|
||||
│ ├── main.py # Entry point, orchestration, signal handling
|
||||
│ ├── websocket_client.py # Hyperliquid WS client, Candle dataclass
|
||||
│ ├── __init__.py # Package exports (all public classes)
|
||||
│ ├── main.py # Entry point, orchestration
|
||||
│ ├── websocket_client.py # Hyperliquid WS client
|
||||
│ ├── candle_buffer.py # Circular buffer with async flush
|
||||
│ ├── database.py # asyncpg/TimescaleDB interface
|
||||
│ └── backfill.py # Historical data backfill from REST API
|
||||
│ ├── backfill.py # Historical data backfill
|
||||
│ ├── custom_timeframe_generator.py # 37m, 148m, 1d aggregation
|
||||
│ ├── indicator_engine.py # SMA/EMA computation & storage
|
||||
│ ├── brain.py # Strategy evaluation & decision logging
|
||||
│ └── backtester.py # Historical replay driver
|
||||
└── api/
|
||||
├── server.py # FastAPI app, all endpoints
|
||||
└── dashboard/static/
|
||||
└── index.html # Real-time web dashboard
|
||||
config/data_config.yaml # Non-secret operational config
|
||||
docker/
|
||||
├── docker-compose.yml # 3-service orchestration
|
||||
├── Dockerfile.api / .collector # python:3.11-slim based
|
||||
└── init-scripts/ # 01-schema.sql, 02-optimization.sql
|
||||
scripts/ # deploy.sh, backup.sh, health_check.sh, backfill.sh
|
||||
tests/data_collector/ # Test directory (empty - tests not yet written)
|
||||
├── server.py # FastAPI app, endpoints for data/backtests
|
||||
└── dashboard/static/index.html # Real-time web dashboard
|
||||
config/data_config.yaml # Operational config & indicator settings
|
||||
docker/ # Docker orchestration & init-scripts
|
||||
scripts/ # Deploy, backup, & utility scripts
|
||||
```
|
||||
|
||||
## Architecture & Data Flow
|
||||
|
||||
```
|
||||
Live: WS -> Buffer -> DB -> CustomTF -> IndicatorEngine -> Brain -> Decisions
|
||||
│ │
|
||||
Backtest: DB (History) -> Backtester ─────────┴─────────────┘
|
||||
```
|
||||
|
||||
- **Stateless Logic**: `IndicatorEngine` and `Brain` are driver-agnostic. They read from DB
|
||||
and write to DB, unaware if the trigger is live WS or backtest replay.
|
||||
- **Consistency**: Indicators are computed exactly the same way for live and backtest.
|
||||
- **Visualization**: Dashboard queries `indicators` and `decisions` tables directly.
|
||||
Decisions contain a JSON snapshot of indicators at the moment of decision.
|
||||
|
||||
## Key Dataclasses
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class Candle: # Standard OHLCV
|
||||
time: datetime; symbol: str; interval: str; ...
|
||||
|
||||
@dataclass
|
||||
class Decision: # Brain output
|
||||
time: datetime; symbol: str; decision_type: str; confidence: float
|
||||
indicator_snapshot: Dict; # Values seen by Brain at decision time
|
||||
backtest_id: Optional[str] # UUID if backtest, None if live
|
||||
```
|
||||
|
||||
## Database Schema (TimescaleDB)
|
||||
|
||||
| Table | Purpose | Key Columns |
|
||||
|-------|---------|-------------|
|
||||
| `candles` | OHLCV data | `(time, symbol, interval)` UNIQUE |
|
||||
| `indicators` | Computed values | `(time, symbol, interval, indicator_name)` UNIQUE |
|
||||
| `decisions` | Buy/sell signals | `(time, symbol, interval, backtest_id)` |
|
||||
| `backtest_runs` | Run metadata | `(id, strategy, config, results)` |
|
||||
|
||||
- `decisions` table stores `indicator_snapshot` JSONB for exact replay/audit.
|
||||
- Compression enabled on all hypertables (7-day policy).
|
||||
|
||||
## API Endpoints (src/api/server.py)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/v1/candles` | Query raw candles |
|
||||
| GET | `/api/v1/indicators` | Query computed indicators (MA, RSI, etc.) |
|
||||
| GET | `/api/v1/decisions` | Query signals (live or backtest) |
|
||||
| GET | `/api/v1/backtests` | List historical backtest runs |
|
||||
| POST | `/api/v1/backtests` | Trigger a new backtest (async background task) |
|
||||
| GET | `/api/v1/stats` | 24h trading stats |
|
||||
|
||||
## Code Style Guidelines
|
||||
|
||||
### Imports
|
||||
Group in this order, separated by blank lines:
|
||||
1. Standard library (`import asyncio`, `from datetime import datetime`)
|
||||
2. Third-party (`import websockets`, `import asyncpg`, `from fastapi import FastAPI`)
|
||||
3. Local/relative (`from .websocket_client import Candle`)
|
||||
|
||||
Use relative imports (`.module`) within the `data_collector` package.
|
||||
Use absolute imports for third-party packages.
|
||||
|
||||
### Formatting
|
||||
- Line length: 100 characters max
|
||||
- Indentation: 4 spaces
|
||||
- Strings: double quotes (single only to avoid escaping)
|
||||
- Trailing commas in multi-line collections
|
||||
- Formatter: black
|
||||
|
||||
### Type Hints
|
||||
- Required on all function parameters and return values
|
||||
- `Optional[Type]` for nullable values
|
||||
- `List[Type]`, `Dict[str, Any]` from `typing` module
|
||||
- `@dataclass` for data-holding classes (e.g., `Candle`, `BufferStats`)
|
||||
- Callable types for callbacks: `Callable[[Candle], Awaitable[None]]`
|
||||
|
||||
### Naming Conventions
|
||||
- Classes: `PascalCase` (DataCollector, CandleBuffer)
|
||||
- Functions/variables: `snake_case` (get_candles, buffer_size)
|
||||
- Constants: `UPPER_SNAKE_CASE` (DB_HOST, MAX_BUFFER_SIZE)
|
||||
- Private methods: `_leading_underscore` (_handle_reconnect, _flush_loop)
|
||||
|
||||
### Docstrings
|
||||
- Triple double quotes on all modules, classes, and public methods
|
||||
- Brief one-line description on first line
|
||||
- Optional blank line + detail if needed
|
||||
- No Args/Returns sections (not strict Google-style)
|
||||
```python
|
||||
"""Add a candle to the buffer
|
||||
Returns True if added, False if buffer full and candle dropped"""
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
- `try/except` with specific exceptions (never bare `except:`)
|
||||
- Log errors with `logger.error()` before re-raising in critical paths
|
||||
- Catch `asyncio.CancelledError` to break loops cleanly
|
||||
- Use `finally` blocks for cleanup (always call `self.stop()`)
|
||||
- Use `@asynccontextmanager` for resource acquisition (DB connections)
|
||||
|
||||
### Async Patterns
|
||||
- `async/await` for all I/O operations
|
||||
- `asyncio.Lock()` for thread-safe buffer access
|
||||
- `asyncio.Event()` for stop/flush coordination
|
||||
- `asyncio.create_task()` for background loops
|
||||
- `asyncio.gather(*tasks, return_exceptions=True)` for parallel cleanup
|
||||
- `asyncio.wait_for(coro, timeout)` for graceful shutdown
|
||||
- `asyncio.run(main())` as the entry point
|
||||
|
||||
### Logging
|
||||
- Module-level: `logger = logging.getLogger(__name__)` in every file
|
||||
- Format: `'%(asctime)s - %(name)s - %(levelname)s - %(message)s'`
|
||||
- Log level from env: `getattr(logging, os.getenv('LOG_LEVEL', 'INFO'))`
|
||||
- Use f-strings in log messages with relevant context
|
||||
- Levels: DEBUG (candle receipt), INFO (lifecycle), WARNING (gaps), ERROR (failures)
|
||||
|
||||
### Database (asyncpg + TimescaleDB)
|
||||
- Connection pool: `asyncpg.create_pool(min_size=1, max_size=N)`
|
||||
- `@asynccontextmanager` wrapper for connection acquisition
|
||||
- Batch inserts with `executemany()`
|
||||
- Upserts with `ON CONFLICT ... DO UPDATE`
|
||||
- Positional params: `$1, $2, ...` (not `%s`)
|
||||
- Use `conn.fetch()`, `conn.fetchrow()`, `conn.fetchval()` for results
|
||||
|
||||
### Configuration
|
||||
- Secrets via environment variables (`os.getenv('DB_PASSWORD')`)
|
||||
- Non-secret config in `config/data_config.yaml`
|
||||
- Constructor defaults fall back to env vars
|
||||
- Never commit `.env` files (contains real credentials)
|
||||
- **Imports**: Stdlib, then Third-party, then Local (relative within package).
|
||||
- **Async**: Use `async/await` for all I/O. Use `asyncpg` pool.
|
||||
- **Typing**: strict type hints required. `Optional[T]`, `List[T]`.
|
||||
- **Logging**: Use `logger = logging.getLogger(__name__)`.
|
||||
- **Config**: Load from `config/data_config.yaml` or env vars.
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Add New API Endpoint
|
||||
1. Add route in `src/api/server.py` with `@app.get()`/`@app.post()`
|
||||
2. Type-hint query params with `Query()`; return `dict` or raise `HTTPException`
|
||||
3. Use `asyncpg` pool for database queries
|
||||
### Add New Indicator
|
||||
1. Add to `config/data_config.yaml` under `indicators`.
|
||||
2. Update `IndicatorEngine._compute_indicator` in `src/data_collector/indicator_engine.py` if new type (non-SMA).
|
||||
3. No DB schema change needed (rows are generic).
|
||||
|
||||
### Add New Data Source
|
||||
1. Create module in `src/data_collector/` following `websocket_client.py` pattern
|
||||
2. Implement async `connect()`, `disconnect()`, `receive()` methods
|
||||
3. Use callback architecture: `on_data`, `on_error` callables
|
||||
|
||||
### Database Schema Changes
|
||||
1. Update `docker/init-scripts/01-schema.sql`
|
||||
2. Update `DatabaseManager` methods in `src/data_collector/database.py`
|
||||
3. Rebuild: `docker-compose down -v && docker-compose up -d --build`
|
||||
|
||||
### Writing Tests
|
||||
1. Create test files in `tests/data_collector/` (e.g., `test_websocket_client.py`)
|
||||
2. Use `pytest-asyncio` for async tests: `@pytest.mark.asyncio`
|
||||
3. Mock external services (WebSocket, database) with `unittest.mock`
|
||||
4. Descriptive names: `test_websocket_reconnection_with_backoff`
|
||||
|
||||
### Historical Data Backfill
|
||||
The `backfill.py` module downloads historical candle data from Hyperliquid's REST API.
|
||||
|
||||
**API Limitations:**
|
||||
- Max 5000 candles per coin/interval combination
|
||||
- 500 candles per response (requires pagination)
|
||||
- Available intervals: 1m, 3m, 5m, 15m, 30m, 1h, 2h, 4h, 8h, 12h, 1d, 3d, 1w, 1M
|
||||
|
||||
**Usage - Python Module:**
|
||||
```python
|
||||
from data_collector.backfill import HyperliquidBackfill
|
||||
from data_collector.database import DatabaseManager
|
||||
|
||||
async with HyperliquidBackfill(db, coin="BTC", intervals=["1m", "1h"]) as backfill:
|
||||
# Backfill last 7 days for all configured intervals
|
||||
results = await backfill.backfill_all_intervals(days_back=7)
|
||||
|
||||
# Or backfill specific interval
|
||||
count = await backfill.backfill_interval("1m", days_back=3)
|
||||
```
|
||||
|
||||
**Usage - CLI:**
|
||||
### Run Backtest
|
||||
```bash
|
||||
# Backfill 7 days of 1m candles for BTC
|
||||
cd src/data_collector && python -m data_collector.backfill --coin BTC --days 7 --intervals 1m
|
||||
# CLI
|
||||
python -m data_collector.backtester --symbol BTC --intervals 37m --start 2025-01-01
|
||||
|
||||
# Backfill multiple intervals
|
||||
python -m data_collector.backfill --coin BTC --days 30 --intervals 1m 5m 1h
|
||||
|
||||
# Backfill MAXIMUM available data (5000 candles per interval)
|
||||
python -m data_collector.backfill --coin BTC --days max --intervals 1m 1h 1d
|
||||
|
||||
# Or use the convenience script
|
||||
bash scripts/backfill.sh BTC 7 "1m 5m 1h"
|
||||
bash scripts/backfill.sh BTC max "1m 1h 1d" # Maximum data
|
||||
# API
|
||||
curl -X POST http://localhost:8000/api/v1/backtests \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"symbol": "BTC", "intervals": ["37m"], "start_date": "2025-01-01"}'
|
||||
```
|
||||
|
||||
**Data Coverage by Interval:**
|
||||
- 1m candles: ~3.5 days (5000 candles)
|
||||
- 1h candles: ~7 months (5000 candles)
|
||||
- 1d candles: ~13.7 years (5000 candles)
|
||||
|
||||
Reference in New Issue
Block a user