Clean up unused files, organize structure, update docs

- Delete obsolete files: data_fetcher_old.py, market_old.py, base_strategy.py (root),
  strategy_sma_cross.py, and old architecture remnants (address_monitor.py,
  position_monitor.py, trade_log.py, wallet_data.py, whale_tracker.py)
- Delete zero-byte Docker artifacts and runtime files (clp_hedger.log,
  clp_hedger/hedge_status.json)
- Move one-off utility scripts to scripts/ directory
- Move example/template files to .temp/ directory
- Update .gitignore: add entries for clp_hedger.log, clp_hedger/hedge_status.json,
  Docker layer hash files, Using, Running, and backups/
- Update .dockerignore: add clp_hedger.log, clp_hedger/hedge_status.json, backups/
- Create example config files: _data/strategies.json.example,
  _data/backtesting_conf.json.example, _data/coin_precision.json.example
- Update GEMINI.md: remove outdated session summaries and duplicate review section
- Update review.md: add cleanup status section, update remaining recommendations
- Update MIGRATION_PLAN.md: mark completed phases, update file references
- Update DOCKER_MIGRATION_GUIDE.md: update import_csv.py path reference
This commit is contained in:
DiTus
2026-08-05 09:50:36 +02:00
parent 967c86e8e9
commit 1a95fe1caa
39 changed files with 599 additions and 3116 deletions

View File

@ -34,7 +34,7 @@
│ └──────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
Host Machine: indicators.py, base_strategy.py, main_app.py
Host Machine: indicators.py, strategies/base_strategy.py, main_app.py
→ connect to localhost:5432
```
@ -79,9 +79,9 @@ wal_buffers = 4MB
2. `resampler.py``sqlite3``db.py`
3. `data_fetcher.py``sqlite3``db.py`
4. `fetch_history.py``sqlite3``db.py`
5. `import_csv.py``sqlite3``db.py`
5. `scripts/import_csv.py``sqlite3``db.py`
6. `indicators.py``sqlite3``psycopg2`
7. `base_strategy.py``sqlite3``psycopg2`
7. `strategies/base_strategy.py``sqlite3``psycopg2`
## TODO List
@ -90,34 +90,34 @@ wal_buffers = 4MB
- [x] Add `psycopg2-binary` to `requirements.txt`
### Phase 2: Modify Data Collection Components
- [ ] Modify `live_candle_fetcher.py` — replace `sqlite3.connect()` with `db.get_connection()`, `INSERT OR REPLACE` with `db.upsert_candles()`, sanitize table names
- [ ] Modify `resampler.py` — replace `sqlite3` with `db.py`, `INSERT OR REPLACE` with `db.upsert_candles()`, `?``%s`
- [ ] Modify `data_fetcher.py` — replace `sqlite3` with `db.py`, `to_sql()``db.upsert_candles()`
- [ ] Modify `fetch_history.py` — replace `sqlite3` with `db.py`
- [ ] Modify `import_csv.py` — replace `sqlite3` with `db.py`, `to_sql()``db.upsert_candles()`
- [x] Modify `live_candle_fetcher.py` — replace `sqlite3.connect()` with `db.get_connection()`, `INSERT OR REPLACE` with `db.upsert_candles()`, sanitize table names
- [x] Modify `resampler.py` — replace `sqlite3` with `db.py`, `INSERT OR REPLACE` with `db.upsert_candles()`, `?``%s`
- [x] Modify `data_fetcher.py` — replace `sqlite3` with `db.py`, `to_sql()``db.upsert_candles()`
- [x] Modify `fetch_history.py` — replace `sqlite3` with `db.py`
- [x] Modify `import_csv.py` — replace `sqlite3` with `db.py`, `to_sql()``db.upsert_candles()` (moved to `scripts/import_csv.py`)
### Phase 3: New Components
- [ ] Create `scripts/resampler_loop.py` — wraps resampler in a while loop with 60s sleep
- [ ] Create `scripts/gap_detector.py` — detects gaps in 1m data, backfills via HTTP API
- [ ] Create `scripts/backup_runner.py` — daily pg_dump with 7-day retention
- [ ] Create `scripts/cron_scheduler.py` — schedules data_fetcher, fetch_history, gap_detector, backup
- [x] Create `scripts/resampler_loop.py` — wraps resampler in a while loop with 60s sleep
- [x] Create `scripts/gap_detector.py` — detects gaps in 1m data, backfills via HTTP API
- [x] Create `scripts/backup_runner.py` — daily pg_dump with 7-day retention
- [x] Create `scripts/cron_scheduler.py` — schedules data_fetcher, fetch_history, gap_detector, backup
### Phase 4: Docker Setup
- [ ] Create `Dockerfile` (python:3.11-slim + supervisor + psycopg2-binary)
- [ ] Create `docker-compose.yml` (postgres + data-collector services)
- [ ] Create `supervisord.conf` (live_candle_fetcher, resampler_loop, indicators_fetcher, cron_scheduler)
- [ ] Create `postgres/postgresql.conf` (tuned for 4GB RAM)
- [ ] Create `.dockerignore`
- [ ] Create `.env.docker.example`
- [ ] Create `secrets/pg_password.txt.example`
- [ ] Update `.gitignore`
- [x] Create `Dockerfile` (python:3.11-slim + supervisor + psycopg2-binary)
- [x] Create `docker-compose.yml` (postgres + data-collector services)
- [x] Create `supervisord.conf` (live_candle_fetcher, resampler_loop, indicators_fetcher, cron_scheduler)
- [x] Create `postgres/postgresql.conf` (tuned for 4GB RAM)
- [x] Create `.dockerignore`
- [x] Create `.env.docker.example`
- [x] Create `secrets/pg_password.txt.example`
- [x] Update `.gitignore`
### Phase 5: Host-Side Updates
- [ ] Modify `indicators.py` on host — connect to `localhost:5432`
- [ ] Modify `base_strategy.py` on host — connect to `localhost:5432`
- [ ] Modify `strategies/base_strategy.py` on host — connect to `localhost:5432`
### Phase 6: Migration Tool
- [ ] Create `migrate_sqlite_to_pg.py` — reads from SQLite, writes to PostgreSQL
- [x] Create `migrate_sqlite_to_pg.py` — reads from SQLite, writes to PostgreSQL
### Phase 7: Testing & Deployment
- [ ] Commit and push to remote