hyper/DOCKER_MIGRATION_GUIDE.md

Docker Image Build & SQLite-to-PostgreSQL Migration Guide

Prerequisites

• Docker and Docker Compose installed on the host
• Access to the hyper project directory
• SQLite database file (_data/market_data.db)
• .env.docker file with PostgreSQL credentials (see .env.docker.example)

1. Building / Rebuilding Docker Images

The data-collector image is built from Dockerfile and contains all Python source files. Whenever source code changes (e.g., db.py, migrate_sqlite_to_pg.py), you must rebuild the image — the container does not mount source files from the host.

Build Command

docker build --network host -t hyper-data-collector:latest .

Note: --network host is used to speed up pip install by avoiding Docker's default bridge network. Omit it if building on a system where host networking is not available.

Rebuild Checklist

1. Make code changes in the project directory
2. Rebuild the image: docker build -t hyper-data-collector:latest .
3. Restart containers: docker-compose down && docker-compose up -d
4. Wait for PostgreSQL healthcheck:

   until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done
   

2. Running the Migration

Step 1: Ensure Containers Are Running

docker-compose up -d

Wait for PostgreSQL to be ready:

until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done

Step 2: Run the Migration Script

docker-compose run --rm data-collector \
  python migrate_sqlite_to_pg.py \
  --sqlite-path _data/market_data.db \
  --log-level normal

Arguments:

Argument	Description	Default
--sqlite-path	Path to the SQLite database file	_data/market_data.db
--log-level	Logging verbosity: off, normal, debug	normal

Step 3: Verify Migration

Check row counts in PostgreSQL:

docker-compose exec postgres psql -U hyper -d hyper -c \
  "SELECT COUNT(*) FROM \"0G_1m\";"

Compare with the source SQLite row count:

sqlite3 _data/market_data.db "SELECT COUNT(*) FROM \"0G_1m\";"

3. Migration Process Details

The migration script (migrate_sqlite_to_pg.py) performs the following:

1. Connects to SQLite (source) and PostgreSQL (destination)
2. Enumerates all candle tables by matching suffixes (_1m, _3m, _5m, etc.)
3. Skips legacy tables (market_cap, candles, daily)
4. For each table:
   • Reads all rows from SQLite via pandas.read_sql
   • Creates the PostgreSQL table if it doesn't exist (db.create_candle_table)
   • Deduplicates records by timestamp_ms (handles duplicate timestamps in source)
   • Batch-upserts records using execute_values with ON CONFLICT DO UPDATE
5. Commits after each table

Tables Migrated

Tables are identified by their timeframe suffix. Supported timeframes:

1m, 3m, 5m, 15m, 30m, 37m, 148m, 1h, 2h, 4h, 8h, 12h, 1d, 3d, 1w, 1month

Table Name Sanitization

Coin symbols containing colons (e.g., xyz:BRENTOIL) are sanitized to xyz_BRENTOIL for PostgreSQL compatibility.

Deduplication

SQLite databases may contain duplicate timestamp_ms entries within the same table. The upsert_candles function in db.py deduplicates records by timestamp_ms before batch insertion, keeping the last occurrence. This prevents PostgreSQL's ON CONFLICT cardinality violation:

ON CONFLICT DO UPDATE command cannot affect row a second time

4. Re-running the Migration

The migration is idempotent — you can safely re-run it:

• CREATE TABLE IF NOT EXISTS skips existing tables
• ON CONFLICT DO UPDATE overwrites existing rows with the same timestamp_ms
• Deduplication ensures no cardinality errors on re-runs

To re-run after code changes:

docker build -t hyper-data-collector:latest .
docker-compose down
docker-compose up -d
until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done
docker-compose run --rm data-collector \
  python migrate_sqlite_to_pg.py \
  --sqlite-path _data/market_data.db \
  --log-level normal

5. Troubleshooting

ON CONFLICT DO UPDATE command cannot affect row a second time

Cause: Duplicate timestamp_ms values in the same batch being inserted.

Fix: The upsert_candles function in db.py deduplicates records by timestamp_ms before insertion. Ensure you're running the latest image:

docker build -t hyper-data-collector:latest .

connection to server at "postgres" failed: Connection timed out

Cause: PostgreSQL container is not ready or not running.

Fix: Wait for the healthcheck to pass before running the migration:

until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done

SQLite database not found at '_data/market_data.db'

Cause: The SQLite database file is not mounted into the container.

Fix: Ensure the _data volume is mounted in docker-compose.yml:

volumes:
  - ./_data:/app/_data

And the database file exists on the host:

ls -la _data/market_data.db

Migration is slow (12+ hours for large databases)

Tips:

• Use --log-level off to reduce I/O from logging
• The page_size=1000 in execute_values is already optimal
• Ensure PostgreSQL has adequate shared_buffers (see postgres/postgresql.conf)

psycopg2.errors.DuplicateTable or table already exists

Cause: The table was partially migrated in a previous run.

Fix: This is handled gracefully by CREATE TABLE IF NOT EXISTS. The migration will continue from where it left off. Re-run the migration script.

Container exits immediately after docker-compose up -d

Cause: Missing .env.docker file or missing secrets.

Fix:

cp .env.docker.example .env.docker
cp secrets/pg_password.txt.example secrets/pg_password.txt

Edit .env.docker to set the correct PG_CONN_STR if needed.

Check Container Logs

# PostgreSQL logs
docker-compose logs postgres

# Data collector logs
docker-compose logs data-collector

# Follow logs in real-time
docker-compose logs -f

Verify PostgreSQL Data

# List all tables
docker-compose exec postgres psql -U hyper -d hyper -c \
  "\dt"

# Check row count for a specific table
docker-compose exec postgres psql -U hyper -d hyper -c \
  "SELECT COUNT(*) FROM \"BTC_1m\";"

# Check for data gaps
docker-compose exec postgres psql -U hyper -d hyper -c \
  "SELECT datetime_utc FROM \"BTC_1m\" ORDER BY timestamp_ms LIMIT 5;"

6. Post-Migration

After migration completes successfully:

1. Update host applications to connect to localhost:5432 instead of SQLite
2. Start the data collector for ongoing data collection:

   docker-compose up -d
   

3. Set up backups using the backup runner script
4. Monitor the gap detector for any missing data