# BTC Bot Project - Migration Context

**Created:** 2024-02-11
**Phase:** 1 of 4 (Data Collection) - COMPLETE
**Status:** Ready for deployment on Synology DS218+

---

## 📁 Project Structure

```
btc_bot/
├── docker/                          # Docker configurations
│   ├── docker-compose.yml          # Main orchestration file
│   ├── Dockerfile.collector        # Data collector service (no apt-get)
│   ├── Dockerfile.api              # API server service
│   ├── timescaledb.conf            # Database optimization for NAS
│   └── init-scripts/               # Auto-run SQL on first start
│       ├── 01-schema.sql           # Main tables & hypertables
│       └── 02-optimization.sql     # Indexes & compression
│
├── config/
│   └── data_config.yaml            # Data collection settings
│
├── src/
│   ├── data_collector/             # Data ingestion module
│   │   ├── __init__.py
│   │   ├── main.py                 # Entry point & orchestrator
│   │   ├── websocket_client.py     # Hyperliquid WebSocket client
│   │   ├── candle_buffer.py        # In-memory circular buffer
│   │   └── database.py             # TimescaleDB interface
│   │
│   └── api/                        # REST API & dashboard
│       ├── server.py               # FastAPI application
│       └── dashboard/
│           └── static/
│               └── index.html      # Real-time web dashboard
│
├── scripts/                        # Operations
│   ├── deploy.sh                   # One-command deployment
│   ├── backup.sh                   # Automated backup script
│   └── health_check.sh             # Health monitoring
│
├── requirements.txt                # Python dependencies
├── .env.example                    # Environment template
└── README.md                       # Full documentation
```

---

## ✅ Completed Features

### Phase 1: Data Collection (DONE)

**Components Built:**

1. **Hyperliquid WebSocket Client**
   - Real-time cbBTC-PERP 1m candles
   - Auto-reconnection with exponential backoff
   - Connection health monitoring
   - File: `src/data_collector/websocket_client.py`

2. **Candle Buffer System**
   - Circular buffer (1000 candles max)
   - Automatic batching (every 30s or 100 candles)
   - Gap detection
   - File: `src/data_collector/candle_buffer.py`

3. **TimescaleDB Integration**
   - Hypertables with weekly partitioning
   - Automatic compression after 7 days
   - Connection pooling
   - Batch inserts with conflict resolution
   - File: `src/data_collector/database.py`

4. **Main Orchestrator**
   - Async event loop
   - Health monitoring (every 60s)
   - Gap detection (every 5 min)
   - Graceful shutdown handling
   - File: `src/data_collector/main.py`

5. **REST API**
   - FastAPI with auto-generated docs
   - Endpoints: /candles, /candles/latest, /health, /export/csv
   - Real-time dashboard with charts
   - File: `src/api/server.py`

6. **Database Schema**
   - `candles` - Main price data (hypertable)
   - `indicators` - Computed values (hypertable)
   - `data_quality` - Issues & gaps log
   - `collector_state` - Metadata tracking
   - Compression enabled for old data
   - Files: `docker/init-scripts/*.sql`

7. **Operations Scripts**
   - Automated deployment
   - Backup with retention
   - Health monitoring
   - Files: `scripts/*.sh`

---

## ⚙️ Configuration

### Environment Variables (.env)

```bash
# Database
DB_HOST=timescaledb          # Use 'timescaledb' for Docker, 'localhost' for direct
DB_PORT=5432
DB_NAME=btc_data
DB_USER=btc_bot
DB_PASSWORD=your_secure_password_here

# Validation (optional)
BASE_RPC_URL=https://base-mainnet.g.alchemy.com/v2/YOUR_KEY

# Notifications (optional)
TELEGRAM_BOT_TOKEN=your_token
TELEGRAM_CHAT_ID=your_chat_id
```

### Data Collection Settings (config/data_config.yaml)

```yaml
# Key settings:
# - Primary: Hyperliquid WebSocket
# - Symbol: cbBTC-PERP
# - Interval: 1m (base), custom intervals computed
# - Buffer: 1000 candles, 30s flush
# - Validation: Every 5 minutes
```

---

## 🚀 Deployment Steps

### Prerequisites
- Synology DS218+ (or similar NAS)
- Docker package installed
- 6GB RAM recommended (upgraded from 2GB)
- SSH access enabled

### Deploy Command

```bash
# On NAS:
cd /volume1/btc_bot
chmod +x scripts/deploy.sh
./scripts/deploy.sh
```

### Post-Deployment Verification

```bash
# Check services
cd docker
docker-compose ps

# View logs
docker-compose logs -f data_collector
docker-compose logs -f api_server

# Test database
docker exec btc_timescale psql -U btc_bot -d btc_data -c "SELECT COUNT(*) FROM candles;"

# Access dashboard
http://your-nas-ip:8000/dashboard
```

---

## 📊 Database Access

### Direct Connection
```bash
# From NAS
docker exec -it btc_timescale psql -U btc_bot -d btc_data

# Useful queries:
# Latest data: SELECT * FROM candles ORDER BY time DESC LIMIT 10;
# Check gaps: SELECT * FROM data_quality WHERE resolved = false;
# Health: SELECT * FROM data_health;
```

### Connection String
```
postgresql://btc_bot:password@localhost:5432/btc_data
```

---

## 🔧 Known Issues & Solutions

### 1. Docker DNS Resolution (FIXED)
**Problem:** `apt-get update` fails in containers
**Solution:** Removed `apt-get` from Dockerfiles - using pre-compiled Python packages only
**Files Modified:** `docker/Dockerfile.collector`, `docker/Dockerfile.api`

### 2. CPU Architecture (opencode incompatibility)
**Problem:** Intel Atom D2701 lacks SSE4.2 instructions
**Solution:** Run opencode on modern PC, connect to NAS via VS Code Remote-SSH
**Workflow:** Edit on PC → Files on NAS via SSH → Docker sees changes

### 3. Memory Constraints on DS218+
**Mitigation:**
- TimescaleDB limited to 1.5GB RAM
- Collector limited to 256MB
- API limited to 512MB
- Compression enabled after 7 days

---

## 📈 Storage Estimates

| Data Type | Growth Rate | Compression |
|-----------|-------------|-------------|
| 1m Candles | ~50MB/year | ~70% reduction |
| Indicators | ~100MB/year | ~70% reduction |
| Backups | Configurable | gzip compressed |

**Total with 1 year retention:** ~200MB compressed

---

## 🎯 Next Phases (TODO)

### Phase 2: Indicators & Brain
**Status:** Not Started
**Components:**
- [ ] RSI, MACD, EMA calculations
- [ ] Custom interval builder (37m, etc.)
- [ ] Indicator storage in database
- [ ] Backfill system for gaps
- [ ] Brain/decision engine
- [ ] Weighted signal combination

**Files to Create:**
- `src/indicators/*.py` (base, rsi, macd, ema, bollinger, volume)
- `src/brain/decision_engine.py`
- `src/brain/weights.py`

### Phase 3: Wallet & Execution
**Status:** Not Started
**Components:**
- [ ] Web3.py EVM integration
- [ ] Wallet management (EOA)
- [ ] Uniswap V3 swap execution
- [ ] Gas management
- [ ] Aave V3 integration (supply cbBTC)

**Files to Create:**
- `src/execution/wallet.py`
- `src/execution/uniswap.py`
- `src/execution/aave.py`
- `src/execution/gas_manager.py`

### Phase 4: Trading Bot Integration
**Status:** Not Started
**Components:**
- [ ] Signal → Trade execution flow
- [ ] $25 USDC trade size
- [ ] Automatic Aave deposit
- [ ] Risk management
- [ ] Telegram notifications
- [ ] Performance tracking

---

## 🔌 Integration Points

### Hyperliquid API
- **WebSocket:** `wss://api.hyperliquid.xyz/ws`
- **REST:** `https://api.hyperliquid.xyz/info`
- **Subscription:** `{"method": "subscribe", "subscription": {"type": "candle", "coin": "cbBTC", "interval": "1m"}}`

### Base Chain (for validation)
- **RPC:** Alchemy/Infura/QuickNode
- **Contract:** cbBTC token on Base
- **Purpose:** Cross-validate Hyperliquid prices

### Aave V3 (Phase 3)
- **Network:** Base
- **Pool:** `0xA238Dd80C259a72e81d7e4664a9801593F98d1c5`
- **Action:** Supply cbBTC as collateral

---

## 📋 Dependencies

### Python Packages (requirements.txt)
Key packages:
- `websockets` - WebSocket client
- `asyncpg` - PostgreSQL async driver
- `fastapi` + `uvicorn` - API server
- `pandas` + `numpy` - Data processing
- `web3` - Ethereum integration (Phase 3)
- `pydantic` - Data validation
- `pyyaml` - Configuration

### Docker Images
- `timescale/timescaledb:2.11.2-pg15`
- `python:3.11-slim` (for custom builds)

---

## 🎨 Development Workflow

### Recommended Setup
```
PC (opencode + VS Code) ──SSH──► NAS (Docker containers)
```

### VS Code Extensions
- Remote - SSH
- Python
- Docker (optional)

### File Editing Options
1. **VS Code Remote-SSH:** Edit directly on NAS
2. **SSHFS:** Mount NAS locally
3. **WinSCP:** Sync local ↔ remote
4. **Synology Drive:** Bidirectional sync

---

## 🔒 Security Notes

1. **Environment File:** `.env` contains secrets - never commit to git
2. **Database:** Not exposed externally by default
3. **API:** No authentication (assumes local network)
4. **Wallet Keys:** Will be in `.env` for Phase 3 - use hardware wallet for large amounts

---

## 📞 Troubleshooting Guide

### Container Won't Start
```bash
# Check logs
docker-compose logs service_name

# Common fixes:
docker-compose down
docker system prune -f
docker-compose build --no-cache
docker-compose up -d
```

### Database Connection Issues
```bash
# Check if DB is ready
docker exec btc_timescale pg_isready -U btc_bot

# Reset (WARNING: deletes all data!)
docker-compose down -v
docker-compose up -d
```

### High Memory Usage
- Edit `docker/timescaledb.conf` - reduce `shared_buffers`
- Edit `docker/docker-compose.yml` - reduce memory limits
- Enable more aggressive compression

### Data Gaps Detected
- Check WebSocket logs for disconnections
- Verify Hyperliquid API status
- Consider implementing REST API backfill (Phase 2)

---

## 📝 Additional Notes

### Design Decisions
1. **1m candles as base:** Custom intervals computed on-demand
2. **TimescaleDB over InfluxDB:** Better SQL support, handles custom intervals
3. **Docker over native:** Easier deployment, resource isolation
4. **Asyncio:** Handles many concurrent connections efficiently
5. **Batch writes:** Minimizes database load on NAS

### Performance Targets
- **Latency:** < 1s from trade to database
- **Throughput:** Handle 1 candle/minute easily
- **Memory:** < 2.5GB total usage
- **Storage:** < 1GB/year with compression

### Future Enhancements
- [ ] Multi-asset support (ETH, SOL, etc.)
- [ ] Historical backfill from REST API
- [ ] Machine learning layer for signals
- [ ] WebSocket multiplexing for multiple symbols
- [ ] Prometheus metrics export
- [ ] Grafana dashboard (alternative to custom UI)

---

## ✅ Pre-Migration Checklist

Before moving to new folder/location:

- [ ] All files listed in structure present
- [ ] Dockerfiles don't contain `apt-get` commands
- [ ] `.env` file configured with your passwords
- [ ] `scripts/*.sh` have execute permissions (`chmod +x`)
- [ ] `docker/init-scripts/*.sql` present
- [ ] `requirements.txt` includes all dependencies
- [ ] Tested on current location (if possible)

---

## 🚀 Quick Start in New Location

```bash
# 1. Copy all files to new location
cp -r btc_bot /new/location/

# 2. Set permissions
chmod +x /new/location/btc_bot/scripts/*.sh

# 3. Configure environment
cd /new/location/btc_bot
cp .env.example .env
nano .env  # Edit passwords

# 4. Deploy
cd docker
docker-compose up -d

# 5. Verify
docker-compose ps
curl http://localhost:8000/api/v1/health
```

---

**End of Context File**
**Ready for migration to new location**