Clean up unused files, organize structure, update docs
- Delete obsolete files: data_fetcher_old.py, market_old.py, base_strategy.py (root), strategy_sma_cross.py, and old architecture remnants (address_monitor.py, position_monitor.py, trade_log.py, wallet_data.py, whale_tracker.py) - Delete zero-byte Docker artifacts and runtime files (clp_hedger.log, clp_hedger/hedge_status.json) - Move one-off utility scripts to scripts/ directory - Move example/template files to .temp/ directory - Update .gitignore: add entries for clp_hedger.log, clp_hedger/hedge_status.json, Docker layer hash files, Using, Running, and backups/ - Update .dockerignore: add clp_hedger.log, clp_hedger/hedge_status.json, backups/ - Create example config files: _data/strategies.json.example, _data/backtesting_conf.json.example, _data/coin_precision.json.example - Update GEMINI.md: remove outdated session summaries and duplicate review section - Update review.md: add cleanup status section, update remaining recommendations - Update MIGRATION_PLAN.md: mark completed phases, update file references - Update DOCKER_MIGRATION_GUIDE.md: update import_csv.py path reference
This commit is contained in:
@ -34,7 +34,7 @@
|
||||
│ └──────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
|
||||
Host Machine: indicators.py, base_strategy.py, main_app.py
|
||||
Host Machine: indicators.py, strategies/base_strategy.py, main_app.py
|
||||
→ connect to localhost:5432
|
||||
```
|
||||
|
||||
@ -79,9 +79,9 @@ wal_buffers = 4MB
|
||||
2. `resampler.py` — `sqlite3` → `db.py`
|
||||
3. `data_fetcher.py` — `sqlite3` → `db.py`
|
||||
4. `fetch_history.py` — `sqlite3` → `db.py`
|
||||
5. `import_csv.py` — `sqlite3` → `db.py`
|
||||
5. `scripts/import_csv.py` — `sqlite3` → `db.py`
|
||||
6. `indicators.py` — `sqlite3` → `psycopg2`
|
||||
7. `base_strategy.py` — `sqlite3` → `psycopg2`
|
||||
7. `strategies/base_strategy.py` — `sqlite3` → `psycopg2`
|
||||
|
||||
## TODO List
|
||||
|
||||
@ -90,34 +90,34 @@ wal_buffers = 4MB
|
||||
- [x] Add `psycopg2-binary` to `requirements.txt`
|
||||
|
||||
### Phase 2: Modify Data Collection Components
|
||||
- [ ] Modify `live_candle_fetcher.py` — replace `sqlite3.connect()` with `db.get_connection()`, `INSERT OR REPLACE` with `db.upsert_candles()`, sanitize table names
|
||||
- [ ] Modify `resampler.py` — replace `sqlite3` with `db.py`, `INSERT OR REPLACE` with `db.upsert_candles()`, `?` → `%s`
|
||||
- [ ] Modify `data_fetcher.py` — replace `sqlite3` with `db.py`, `to_sql()` → `db.upsert_candles()`
|
||||
- [ ] Modify `fetch_history.py` — replace `sqlite3` with `db.py`
|
||||
- [ ] Modify `import_csv.py` — replace `sqlite3` with `db.py`, `to_sql()` → `db.upsert_candles()`
|
||||
- [x] Modify `live_candle_fetcher.py` — replace `sqlite3.connect()` with `db.get_connection()`, `INSERT OR REPLACE` with `db.upsert_candles()`, sanitize table names
|
||||
- [x] Modify `resampler.py` — replace `sqlite3` with `db.py`, `INSERT OR REPLACE` with `db.upsert_candles()`, `?` → `%s`
|
||||
- [x] Modify `data_fetcher.py` — replace `sqlite3` with `db.py`, `to_sql()` → `db.upsert_candles()`
|
||||
- [x] Modify `fetch_history.py` — replace `sqlite3` with `db.py`
|
||||
- [x] Modify `import_csv.py` — replace `sqlite3` with `db.py`, `to_sql()` → `db.upsert_candles()` (moved to `scripts/import_csv.py`)
|
||||
|
||||
### Phase 3: New Components
|
||||
- [ ] Create `scripts/resampler_loop.py` — wraps resampler in a while loop with 60s sleep
|
||||
- [ ] Create `scripts/gap_detector.py` — detects gaps in 1m data, backfills via HTTP API
|
||||
- [ ] Create `scripts/backup_runner.py` — daily pg_dump with 7-day retention
|
||||
- [ ] Create `scripts/cron_scheduler.py` — schedules data_fetcher, fetch_history, gap_detector, backup
|
||||
- [x] Create `scripts/resampler_loop.py` — wraps resampler in a while loop with 60s sleep
|
||||
- [x] Create `scripts/gap_detector.py` — detects gaps in 1m data, backfills via HTTP API
|
||||
- [x] Create `scripts/backup_runner.py` — daily pg_dump with 7-day retention
|
||||
- [x] Create `scripts/cron_scheduler.py` — schedules data_fetcher, fetch_history, gap_detector, backup
|
||||
|
||||
### Phase 4: Docker Setup
|
||||
- [ ] Create `Dockerfile` (python:3.11-slim + supervisor + psycopg2-binary)
|
||||
- [ ] Create `docker-compose.yml` (postgres + data-collector services)
|
||||
- [ ] Create `supervisord.conf` (live_candle_fetcher, resampler_loop, indicators_fetcher, cron_scheduler)
|
||||
- [ ] Create `postgres/postgresql.conf` (tuned for 4GB RAM)
|
||||
- [ ] Create `.dockerignore`
|
||||
- [ ] Create `.env.docker.example`
|
||||
- [ ] Create `secrets/pg_password.txt.example`
|
||||
- [ ] Update `.gitignore`
|
||||
- [x] Create `Dockerfile` (python:3.11-slim + supervisor + psycopg2-binary)
|
||||
- [x] Create `docker-compose.yml` (postgres + data-collector services)
|
||||
- [x] Create `supervisord.conf` (live_candle_fetcher, resampler_loop, indicators_fetcher, cron_scheduler)
|
||||
- [x] Create `postgres/postgresql.conf` (tuned for 4GB RAM)
|
||||
- [x] Create `.dockerignore`
|
||||
- [x] Create `.env.docker.example`
|
||||
- [x] Create `secrets/pg_password.txt.example`
|
||||
- [x] Update `.gitignore`
|
||||
|
||||
### Phase 5: Host-Side Updates
|
||||
- [ ] Modify `indicators.py` on host — connect to `localhost:5432`
|
||||
- [ ] Modify `base_strategy.py` on host — connect to `localhost:5432`
|
||||
- [ ] Modify `strategies/base_strategy.py` on host — connect to `localhost:5432`
|
||||
|
||||
### Phase 6: Migration Tool
|
||||
- [ ] Create `migrate_sqlite_to_pg.py` — reads from SQLite, writes to PostgreSQL
|
||||
- [x] Create `migrate_sqlite_to_pg.py` — reads from SQLite, writes to PostgreSQL
|
||||
|
||||
### Phase 7: Testing & Deployment
|
||||
- [ ] Commit and push to remote
|
||||
|
||||
Reference in New Issue
Block a user