hyper/DOCKER_MIGRATION_GUIDE.md

# Docker Image Build & SQLite-to-PostgreSQL Migration Guide

## Prerequisites

- Docker and Docker Compose installed on the host
- Access to the `hyper` project directory
- SQLite database file (`_data/market_data.db`)
- `.env.docker` file with PostgreSQL credentials (see `.env.docker.example`)

## 1. Building / Rebuilding Docker Images

The `data-collector` image is built from `Dockerfile` and contains all Python source
files. Whenever source code changes (e.g., `db.py`, `migrate_sqlite_to_pg.py`),
**you must rebuild the image** — the container does not mount source files from
the host.

### Build Command

```bash
docker build --network host -t hyper-data-collector:latest .
```

> **Note:** `--network host` is used to speed up `pip install` by avoiding Docker's
> default bridge network. Omit it if building on a system where host networking is
> not available.

### Rebuild Checklist

1. Make code changes in the project directory
2. Rebuild the image: `docker build -t hyper-data-collector:latest .`
3. Restart containers: `docker-compose down && docker-compose up -d`
4. Wait for PostgreSQL healthcheck:
   ```bash
   until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done
   ```

## 2. Running the Migration

### Step 1: Ensure Containers Are Running

```bash
docker-compose up -d
```

Wait for PostgreSQL to be ready:

```bash
until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done
```

### Step 2: Run the Migration Script

```bash
docker-compose run --rm data-collector \
  python migrate_sqlite_to_pg.py \
  --sqlite-path _data/market_data.db \
  --log-level normal
```

**Arguments:**

| Argument | Description | Default |
|---|---|---|
| `--sqlite-path` | Path to the SQLite database file | `_data/market_data.db` |
| `--log-level` | Logging verbosity: `off`, `normal`, `debug` | `normal` |

### Step 3: Verify Migration

Check row counts in PostgreSQL:

```bash
docker-compose exec postgres psql -U hyper -d hyper -c \
  "SELECT COUNT(*) FROM \"0G_1m\";"
```

Compare with the source SQLite row count:

```bash
sqlite3 _data/market_data.db "SELECT COUNT(*) FROM \"0G_1m\";"
```

## 3. Migration Process Details

The migration script (`migrate_sqlite_to_pg.py`) performs the following:

1. **Connects** to SQLite (source) and PostgreSQL (destination)
2. **Enumerates** all candle tables by matching suffixes (`_1m`, `_3m`, `_5m`, etc.)
3. **Skips** legacy tables (`market_cap`, `candles`, `daily`)
4. **For each table:**
   - Reads all rows from SQLite via `pandas.read_sql`
   - Creates the PostgreSQL table if it doesn't exist (`db.create_candle_table`)
   - **Deduplicates** records by `timestamp_ms` (handles duplicate timestamps in source)
   - Batch-upserts records using `execute_values` with `ON CONFLICT DO UPDATE`
5. **Commits** after each table

### Tables Migrated

Tables are identified by their timeframe suffix. Supported timeframes:

```
1m, 3m, 5m, 15m, 30m, 37m, 148m, 1h, 2h, 4h, 8h, 12h, 1d, 3d, 1w, 1month
```

### Table Name Sanitization

Coin symbols containing colons (e.g., `xyz:BRENTOIL`) are sanitized to
`xyz_BRENTOIL` for PostgreSQL compatibility.

### Deduplication

SQLite databases may contain duplicate `timestamp_ms` entries within the same
table. The `upsert_candles` function in `db.py` deduplicates records by
`timestamp_ms` before batch insertion, keeping the last occurrence. This prevents
PostgreSQL's `ON CONFLICT` cardinality violation:

```
ON CONFLICT DO UPDATE command cannot affect row a second time
```

## 4. Re-running the Migration

The migration is **idempotent** — you can safely re-run it:

- `CREATE TABLE IF NOT EXISTS` skips existing tables
- `ON CONFLICT DO UPDATE` overwrites existing rows with the same `timestamp_ms`
- Deduplication ensures no cardinality errors on re-runs

To re-run after code changes:

```bash
docker build -t hyper-data-collector:latest .
docker-compose down
docker-compose up -d
until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done
docker-compose run --rm data-collector \
  python migrate_sqlite_to_pg.py \
  --sqlite-path _data/market_data.db \
  --log-level normal
```

## 5. Troubleshooting

### `ON CONFLICT DO UPDATE command cannot affect row a second time`

**Cause:** Duplicate `timestamp_ms` values in the same batch being inserted.

**Fix:** The `upsert_candles` function in `db.py` deduplicates records by
`timestamp_ms` before insertion. Ensure you're running the latest image:

```bash
docker build -t hyper-data-collector:latest .
```

### `connection to server at "postgres" failed: Connection timed out`

**Cause:** PostgreSQL container is not ready or not running.

**Fix:** Wait for the healthcheck to pass before running the migration:

```bash
until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done
```

### `SQLite database not found at '_data/market_data.db'`

**Cause:** The SQLite database file is not mounted into the container.

**Fix:** Ensure the `_data` volume is mounted in `docker-compose.yml`:

```yaml
volumes:
  - ./_data:/app/_data
```

And the database file exists on the host:

```bash
ls -la _data/market_data.db
```

### Migration is slow (12+ hours for large databases)

**Tips:**
- Use `--log-level off` to reduce I/O from logging
- The `page_size=1000` in `execute_values` is already optimal
- Ensure PostgreSQL has adequate `shared_buffers` (see `postgres/postgresql.conf`)

### `psycopg2.errors.DuplicateTable` or table already exists

**Cause:** The table was partially migrated in a previous run.

**Fix:** This is handled gracefully by `CREATE TABLE IF NOT EXISTS`. The migration
will continue from where it left off. Re-run the migration script.

### Container exits immediately after `docker-compose up -d`

**Cause:** Missing `.env.docker` file or missing secrets.

**Fix:**

```bash
cp .env.docker.example .env.docker
cp secrets/pg_password.txt.example secrets/pg_password.txt
```

Edit `.env.docker` to set the correct `PG_CONN_STR` if needed.

### Check Container Logs

```bash
# PostgreSQL logs
docker-compose logs postgres

# Data collector logs
docker-compose logs data-collector

# Follow logs in real-time
docker-compose logs -f
```

### Verify PostgreSQL Data

```bash
# List all tables
docker-compose exec postgres psql -U hyper -d hyper -c \
  "\dt"

# Check row count for a specific table
docker-compose exec postgres psql -U hyper -d hyper -c \
  "SELECT COUNT(*) FROM \"BTC_1m\";"

# Check for data gaps
docker-compose exec postgres psql -U hyper -d hyper -c \
  "SELECT datetime_utc FROM \"BTC_1m\" ORDER BY timestamp_ms LIMIT 5;"
```

## 6. Post-Migration

After migration completes successfully:

1. **Update host applications** to connect to `localhost:5432` instead of SQLite
2. **Start the data collector** for ongoing data collection:
   ```bash
   docker-compose up -d
   ```
3. **Set up backups** using the backup runner script
4. **Monitor** the gap detector for any missing data