Clean up unused files, organize structure, update docs
- Delete obsolete files: data_fetcher_old.py, market_old.py, base_strategy.py (root), strategy_sma_cross.py, and old architecture remnants (address_monitor.py, position_monitor.py, trade_log.py, wallet_data.py, whale_tracker.py) - Delete zero-byte Docker artifacts and runtime files (clp_hedger.log, clp_hedger/hedge_status.json) - Move one-off utility scripts to scripts/ directory - Move example/template files to .temp/ directory - Update .gitignore: add entries for clp_hedger.log, clp_hedger/hedge_status.json, Docker layer hash files, Using, Running, and backups/ - Update .dockerignore: add clp_hedger.log, clp_hedger/hedge_status.json, backups/ - Create example config files: _data/strategies.json.example, _data/backtesting_conf.json.example, _data/coin_precision.json.example - Update GEMINI.md: remove outdated session summaries and duplicate review section - Update review.md: add cleanup status section, update remaining recommendations - Update MIGRATION_PLAN.md: mark completed phases, update file references - Update DOCKER_MIGRATION_GUIDE.md: update import_csv.py path reference
This commit is contained in:
DiTus
247
DOCKER_MIGRATION_GUIDE.md
Normal file
247
DOCKER_MIGRATION_GUIDE.md
Normal file
@ -0,0 +1,247 @@
|
||||
# Docker Image Build & SQLite-to-PostgreSQL Migration Guide
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker and Docker Compose installed on the host
|
||||
- Access to the `hyper` project directory
|
||||
- SQLite database file (`_data/market_data.db`)
|
||||
- `.env.docker` file with PostgreSQL credentials (see `.env.docker.example`)
|
||||
|
||||
## 1. Building / Rebuilding Docker Images
|
||||
|
||||
The `data-collector` image is built from `Dockerfile` and contains all Python source
|
||||
files. Whenever source code changes (e.g., `db.py`, `migrate_sqlite_to_pg.py`),
|
||||
**you must rebuild the image** — the container does not mount source files from
|
||||
the host.
|
||||
|
||||
### Build Command
|
||||
|
||||
```bash
|
||||
docker build --network host -t hyper-data-collector:latest .
|
||||
```
|
||||
|
||||
> **Note:** `--network host` is used to speed up `pip install` by avoiding Docker's
|
||||
> default bridge network. Omit it if building on a system where host networking is
|
||||
> not available.
|
||||
|
||||
### Rebuild Checklist
|
||||
|
||||
1. Make code changes in the project directory
|
||||
2. Rebuild the image: `docker build -t hyper-data-collector:latest .`
|
||||
3. Restart containers: `docker-compose down && docker-compose up -d`
|
||||
4. Wait for PostgreSQL healthcheck:
|
||||
```bash
|
||||
until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done
|
||||
```
|
||||
|
||||
## 2. Running the Migration
|
||||
|
||||
### Step 1: Ensure Containers Are Running
|
||||
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
Wait for PostgreSQL to be ready:
|
||||
|
||||
```bash
|
||||
until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done
|
||||
```
|
||||
|
||||
### Step 2: Run the Migration Script
|
||||
|
||||
```bash
|
||||
docker-compose run --rm data-collector \
|
||||
python migrate_sqlite_to_pg.py \
|
||||
--sqlite-path _data/market_data.db \
|
||||
--log-level normal
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
|
||||
| Argument | Description | Default |
|
||||
|---|---|---|
|
||||
| `--sqlite-path` | Path to the SQLite database file | `_data/market_data.db` |
|
||||
| `--log-level` | Logging verbosity: `off`, `normal`, `debug` | `normal` |
|
||||
|
||||
### Step 3: Verify Migration
|
||||
|
||||
Check row counts in PostgreSQL:
|
||||
|
||||
```bash
|
||||
docker-compose exec postgres psql -U hyper -d hyper -c \
|
||||
"SELECT COUNT(*) FROM \"0G_1m\";"
|
||||
```
|
||||
|
||||
Compare with the source SQLite row count:
|
||||
|
||||
```bash
|
||||
sqlite3 _data/market_data.db "SELECT COUNT(*) FROM \"0G_1m\";"
|
||||
```
|
||||
|
||||
## 3. Migration Process Details
|
||||
|
||||
The migration script (`migrate_sqlite_to_pg.py`) performs the following:
|
||||
|
||||
1. **Connects** to SQLite (source) and PostgreSQL (destination)
|
||||
2. **Enumerates** all candle tables by matching suffixes (`_1m`, `_3m`, `_5m`, etc.)
|
||||
3. **Skips** legacy tables (`market_cap`, `candles`, `daily`)
|
||||
4. **For each table:**
|
||||
- Reads all rows from SQLite via `pandas.read_sql`
|
||||
- Creates the PostgreSQL table if it doesn't exist (`db.create_candle_table`)
|
||||
- **Deduplicates** records by `timestamp_ms` (handles duplicate timestamps in source)
|
||||
- Batch-upserts records using `execute_values` with `ON CONFLICT DO UPDATE`
|
||||
5. **Commits** after each table
|
||||
|
||||
### Tables Migrated
|
||||
|
||||
Tables are identified by their timeframe suffix. Supported timeframes:
|
||||
|
||||
```
|
||||
1m, 3m, 5m, 15m, 30m, 37m, 148m, 1h, 2h, 4h, 8h, 12h, 1d, 3d, 1w, 1month
|
||||
```
|
||||
|
||||
### Table Name Sanitization
|
||||
|
||||
Coin symbols containing colons (e.g., `xyz:BRENTOIL`) are sanitized to
|
||||
`xyz_BRENTOIL` for PostgreSQL compatibility.
|
||||
|
||||
### Deduplication
|
||||
|
||||
SQLite databases may contain duplicate `timestamp_ms` entries within the same
|
||||
table. The `upsert_candles` function in `db.py` deduplicates records by
|
||||
`timestamp_ms` before batch insertion, keeping the last occurrence. This prevents
|
||||
PostgreSQL's `ON CONFLICT` cardinality violation:
|
||||
|
||||
```
|
||||
ON CONFLICT DO UPDATE command cannot affect row a second time
|
||||
```
|
||||
|
||||
## 4. Re-running the Migration
|
||||
|
||||
The migration is **idempotent** — you can safely re-run it:
|
||||
|
||||
- `CREATE TABLE IF NOT EXISTS` skips existing tables
|
||||
- `ON CONFLICT DO UPDATE` overwrites existing rows with the same `timestamp_ms`
|
||||
- Deduplication ensures no cardinality errors on re-runs
|
||||
|
||||
To re-run after code changes:
|
||||
|
||||
```bash
|
||||
docker build -t hyper-data-collector:latest .
|
||||
docker-compose down
|
||||
docker-compose up -d
|
||||
until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done
|
||||
docker-compose run --rm data-collector \
|
||||
python migrate_sqlite_to_pg.py \
|
||||
--sqlite-path _data/market_data.db \
|
||||
--log-level normal
|
||||
```
|
||||
|
||||
## 5. Troubleshooting
|
||||
|
||||
### `ON CONFLICT DO UPDATE command cannot affect row a second time`
|
||||
|
||||
**Cause:** Duplicate `timestamp_ms` values in the same batch being inserted.
|
||||
|
||||
**Fix:** The `upsert_candles` function in `db.py` deduplicates records by
|
||||
`timestamp_ms` before insertion. Ensure you're running the latest image:
|
||||
|
||||
```bash
|
||||
docker build -t hyper-data-collector:latest .
|
||||
```
|
||||
|
||||
### `connection to server at "postgres" failed: Connection timed out`
|
||||
|
||||
**Cause:** PostgreSQL container is not ready or not running.
|
||||
|
||||
**Fix:** Wait for the healthcheck to pass before running the migration:
|
||||
|
||||
```bash
|
||||
until docker-compose exec postgres pg_isready -U hyper -d hyper; do sleep 2; done
|
||||
```
|
||||
|
||||
### `SQLite database not found at '_data/market_data.db'`
|
||||
|
||||
**Cause:** The SQLite database file is not mounted into the container.
|
||||
|
||||
**Fix:** Ensure the `_data` volume is mounted in `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- ./_data:/app/_data
|
||||
```
|
||||
|
||||
And the database file exists on the host:
|
||||
|
||||
```bash
|
||||
ls -la _data/market_data.db
|
||||
```
|
||||
|
||||
### Migration is slow (12+ hours for large databases)
|
||||
|
||||
**Tips:**
|
||||
- Use `--log-level off` to reduce I/O from logging
|
||||
- The `page_size=1000` in `execute_values` is already optimal
|
||||
- Ensure PostgreSQL has adequate `shared_buffers` (see `postgres/postgresql.conf`)
|
||||
|
||||
### `psycopg2.errors.DuplicateTable` or table already exists
|
||||
|
||||
**Cause:** The table was partially migrated in a previous run.
|
||||
|
||||
**Fix:** This is handled gracefully by `CREATE TABLE IF NOT EXISTS`. The migration
|
||||
will continue from where it left off. Re-run the migration script.
|
||||
|
||||
### Container exits immediately after `docker-compose up -d`
|
||||
|
||||
**Cause:** Missing `.env.docker` file or missing secrets.
|
||||
|
||||
**Fix:**
|
||||
|
||||
```bash
|
||||
cp .env.docker.example .env.docker
|
||||
cp secrets/pg_password.txt.example secrets/pg_password.txt
|
||||
```
|
||||
|
||||
Edit `.env.docker` to set the correct `PG_CONN_STR` if needed.
|
||||
|
||||
### Check Container Logs
|
||||
|
||||
```bash
|
||||
# PostgreSQL logs
|
||||
docker-compose logs postgres
|
||||
|
||||
# Data collector logs
|
||||
docker-compose logs data-collector
|
||||
|
||||
# Follow logs in real-time
|
||||
docker-compose logs -f
|
||||
```
|
||||
|
||||
### Verify PostgreSQL Data
|
||||
|
||||
```bash
|
||||
# List all tables
|
||||
docker-compose exec postgres psql -U hyper -d hyper -c \
|
||||
"\dt"
|
||||
|
||||
# Check row count for a specific table
|
||||
docker-compose exec postgres psql -U hyper -d hyper -c \
|
||||
"SELECT COUNT(*) FROM \"BTC_1m\";"
|
||||
|
||||
# Check for data gaps
|
||||
docker-compose exec postgres psql -U hyper -d hyper -c \
|
||||
"SELECT datetime_utc FROM \"BTC_1m\" ORDER BY timestamp_ms LIMIT 5;"
|
||||
```
|
||||
|
||||
## 6. Post-Migration
|
||||
|
||||
After migration completes successfully:
|
||||
|
||||
1. **Update host applications** to connect to `localhost:5432` instead of SQLite
|
||||
2. **Start the data collector** for ongoing data collection:
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
3. **Set up backups** using the backup runner script
|
||||
4. **Monitor** the gap detector for any missing data
|
||||
Reference in New Issue
Block a user