eaceeb7e3b
- Documented debugging session that resolved critical path resolution error - Added comprehensive session summary covering problem identification, solution, and testing - Recorded decisions made and files modified during the fix - Included next steps for ongoing monitoring and improvement - Maintained structured format consistent with sessionsummary agent specifications
246 lines
11 KiB
Markdown
246 lines
11 KiB
Markdown
# Project Overview
|
|
|
|
This project is a sophisticated, multi-process automated trading bot for the Hyperliquid decentralized exchange. It is written in Python and uses a modular architecture to separate concerns like data fetching, strategy execution, and trade management.
|
|
|
|
The bot uses a high-performance data pipeline with SQLite for storing market data. Trading strategies are defined and configured in a JSON file, allowing for easy adjustments without code changes. The system supports multiple, independent trading agents for risk segregation and PNL tracking. A live terminal dashboard provides real-time monitoring of market data, strategy signals, and the status of all background processes.
|
|
|
|
## Building and Running
|
|
|
|
### 1. Setup
|
|
|
|
1. **Create and activate a virtual environment:**
|
|
```bash
|
|
# For Windows
|
|
python -m venv .venv
|
|
.\.venv\Scripts\activate
|
|
|
|
# For macOS/Linux
|
|
python3 -m venv .venv
|
|
source .venv/bin/activate
|
|
```
|
|
|
|
2. **Install dependencies:**
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
3. **Configure environment variables:**
|
|
Create a `.env` file in the root of the project (you can copy `.env.example`) and add your Hyperliquid wallet private key and any agent keys.
|
|
|
|
4. **Configure strategies:**
|
|
Edit `_data/strategies.json` to enable and configure your desired trading strategies.
|
|
|
|
### 2. Running the Bot
|
|
|
|
To run the main application, which includes the dashboard and all background processes, execute the following command:
|
|
|
|
```bash
|
|
python main_app.py
|
|
```
|
|
|
|
## Development Conventions
|
|
|
|
* **Modularity:** The project is divided into several scripts, each with a specific responsibility (e.g., `data_fetcher.py`, `trade_executor.py`).
|
|
* **Configuration-driven:** Strategies are defined in `_data/strategies.json`, not hardcoded. This allows for easy management of strategies.
|
|
* **Multi-processing:** The application uses the `multiprocessing` module to run different components in parallel for performance and stability.
|
|
* **Strategies:** Custom strategies should inherit from the `BaseStrategy` class (defined in `strategies/base_strategy.py`) and implement the `calculate_signals` method.
|
|
* **Documentation:** The `WIKI/` directory contains detailed documentation for the project. Start with `WIKI/SUMMARY.md`.
|
|
|
|
## Session Summary
|
|
|
|
**Date:** 2025-11-10
|
|
|
|
**Objective(s):**
|
|
Fix urllib3 SSL compatibility warning and create sessionsummary agent following OpenCode.ai guidelines
|
|
|
|
**Key Accomplishments:**
|
|
* Resolved NotOpenSSLWarning by downgrading urllib3 from 2.5.0 to 1.26.20
|
|
* Updated requirements.txt to prevent future SSL compatibility issues
|
|
* Created sessionsummary agent in .opencode/agent/ following OpenCode.ai specifications
|
|
* Removed incorrect Python implementation and created proper markdown agent configuration
|
|
|
|
**Decisions Made:**
|
|
* Chose to downgrade urllib3 instead of upgrading SSL environment for stability
|
|
* Followed OpenCode.ai agent guidelines instead of creating custom Python implementation
|
|
* Configured sessionsummary as subagent with proper permissions and tools
|
|
|
|
**Key Files Modified:**
|
|
* `requirements.txt`
|
|
* `GEMINI.md`
|
|
* `.opencode/agent/sessionsummary.md`
|
|
|
|
**Next Steps/Open Questions:**
|
|
* Test trading bot functionality after SSL fix to ensure no regressions
|
|
* Integrate sessionsummary agent into regular development workflow
|
|
* Add .opencode/ to .gitignore if not already present
|
|
|
|
## Session Summary
|
|
|
|
**Date:** 2025-11-11
|
|
|
|
**Objective(s):**
|
|
Start new Gemini session and organize project files by creating .temp folder for examples and temporary files
|
|
|
|
**Key Accomplishments:**
|
|
* Created .temp folder for organizing examples and temporary files
|
|
* Updated .gitignore to include .temp/ directory
|
|
* Moved model_comparison_examples.md to .temp folder for better organization
|
|
* Established file management practices for future development
|
|
|
|
**Decisions Made:**
|
|
* Chose to use .temp folder instead of mixing examples with main project files
|
|
* Added .temp to .gitignore to prevent accidental commits of temporary files
|
|
* Followed user instruction to organize project structure for better maintainability
|
|
|
|
**Key Files Modified:**
|
|
* `.gitignore`
|
|
* `.temp/` (created)
|
|
* `model_comparison_examples.md` (moved to .temp/)
|
|
|
|
**Next Steps/Open Questions:**
|
|
* Continue organizing any other example or temporary files into .temp folder
|
|
* Maintain consistent file organization practices in future development
|
|
* Consider creating additional organizational directories if needed
|
|
|
|
## Session Summary
|
|
|
|
**Date:** 2025-11-11
|
|
|
|
**Objective(s):**
|
|
Fix DashboardDataFetcher path resolution error causing file operation failures
|
|
|
|
**Key Accomplishments:**
|
|
* Identified root cause of file path error in dashboard_data_fetcher.py subprocess execution
|
|
* Fixed path resolution by using absolute paths instead of relative paths
|
|
* Added os.makedirs() call to ensure _logs directory exists before file operations
|
|
* Tested fix and confirmed DashboardDataFetcher now works correctly
|
|
* Committed and pushed fix to remote repository
|
|
|
|
**Decisions Made:**
|
|
* Used os.path.dirname(os.path.abspath(__file__)) to get correct project root
|
|
* Ensured backward compatibility while fixing the path resolution issue
|
|
* Maintained atomic file write pattern for data integrity
|
|
|
|
**Key Files Modified:**
|
|
* `dashboard_data_fetcher.py`
|
|
* `GEMINI.md`
|
|
|
|
**Next Steps/Open Questions:**
|
|
* Monitor DashboardDataFetcher to ensure no further path-related errors occur
|
|
* Consider reviewing other subprocess scripts for similar path resolution issues
|
|
* Test main_app.py to ensure dashboard displays data correctly
|
|
|
|
## Session Summary
|
|
|
|
**Date:** 2025-11-11
|
|
|
|
**Objective(s):**
|
|
Debug and fix DashboardDataFetcher path resolution error causing file operation failures
|
|
|
|
**Key Accomplishments:**
|
|
* Identified root cause of file path error in dashboard_data_fetcher.py subprocess execution
|
|
* Fixed path resolution by using absolute paths instead of relative paths
|
|
* Added os.makedirs() call to ensure _logs directory exists before file operations
|
|
* Tested fix and confirmed DashboardDataFetcher now works correctly
|
|
* Committed and pushed fix to remote repository
|
|
* Organized project files with .temp folder for better structure
|
|
|
|
**Decisions Made:**
|
|
* Used os.path.dirname(os.path.abspath(__file__)) to get correct project root
|
|
* Ensured backward compatibility while fixing path resolution issue
|
|
* Maintained atomic file write pattern for data integrity
|
|
* Added proper directory existence checks to prevent runtime errors
|
|
|
|
**Key Files Modified:**
|
|
* `dashboard_data_fetcher.py`
|
|
* `GEMINI.md`
|
|
* `.gitignore`
|
|
* `.temp/` (created)
|
|
|
|
**Next Steps/Open Questions:**
|
|
* Monitor DashboardDataFetcher to ensure no further path-related errors occur
|
|
* Consider reviewing other subprocess scripts for similar path resolution issues
|
|
* Test main_app.py to ensure dashboard displays data correctly
|
|
* Continue improving project organization and file management practices
|
|
|
|
---
|
|
|
|
# Project Review and Recommendations
|
|
|
|
This review provides an analysis of the current state of the automated trading bot project, proposes specific code improvements, and identifies files that appear to be unused or are one-off utilities that could be reorganized.
|
|
|
|
The project is a well-structured, multi-process Python application for crypto trading. It has a clear separation of concerns between data fetching, strategy execution, and trade management. The use of `multiprocessing` and a centralized `main_app.py` orchestrator is a solid architectural choice.
|
|
|
|
The following sections detail recommendations for improving configuration management, code structure, and robustness, along with a list of files recommended for cleanup.
|
|
|
|
---
|
|
|
|
## Proposed Code Changes
|
|
|
|
### 1. Centralize Configuration
|
|
|
|
- **Issue:** Key configuration variables like `WATCHED_COINS` and `required_timeframes` are hardcoded in `main_app.py`. This makes them difficult to change without modifying the source code.
|
|
- **Proposal:**
|
|
- Create a central configuration file, e.g., `_data/config.json`.
|
|
- Move `WATCHED_COINS` and `required_timeframes` into this new file.
|
|
- Load this configuration in `main_app.py` at startup.
|
|
- **Benefit:** Decouples configuration from code, making the application more flexible and easier to manage.
|
|
|
|
### 2. Refactor `main_app.py` for Clarity
|
|
|
|
- **Issue:** `main_app.py` is long and handles multiple responsibilities: process orchestration, dashboard rendering, and data reading.
|
|
- **Proposal:**
|
|
- **Abstract Process Management:** The functions for running subprocesses (e.g., `run_live_candle_fetcher`, `run_resampler_job`) contain repetitive logic for logging, shutdown handling, and process looping. This could be abstracted into a generic `ProcessRunner` class.
|
|
- **Create a Dashboard Class:** The complex dashboard rendering logic could be moved into a separate `Dashboard` class to improve separation of concerns and make the main application loop cleaner.
|
|
- **Benefit:** Improves code readability, reduces duplication, and makes the application easier to maintain and extend.
|
|
|
|
### 3. Improve Project Structure
|
|
|
|
- **Issue:** The root directory is cluttered with numerous Python scripts, making it difficult to distinguish between core application files, utility scripts, and old/example files.
|
|
- **Proposal:**
|
|
- Create a `scripts/` directory and move all one-off utility and maintenance scripts into it.
|
|
- Consider creating a `src/` or `app/` directory to house the core application source code (`main_app.py`, `trade_executor.py`, etc.), separating it clearly from configuration, data, and documentation.
|
|
- **Benefit:** A cleaner, more organized project structure that is easier for new developers to understand.
|
|
|
|
### 4. Enhance Robustness and Error Handling
|
|
|
|
- **Issue:** The agent loading in `trade_executor.py` relies on discovering environment variables by a naming convention (`_AGENT_PK`). This is clever but can be brittle if environment variables are named incorrectly.
|
|
- **Proposal:**
|
|
- Explicitly define the agent names and their corresponding environment variable keys in the proposed `_data/config.json` file. The `trade_executor` would then load only the agents specified in the configuration.
|
|
- **Benefit:** Makes agent configuration more explicit and less prone to errors from stray environment variables.
|
|
|
|
---
|
|
|
|
## Identified Unused/Utility Files
|
|
|
|
The following files were identified as likely being unused by the core application, being obsolete, or serving as one-off utilities. It is recommended to **move them to a `scripts/` directory** or **delete them** if they are obsolete.
|
|
|
|
### Obsolete / Old Versions:
|
|
- `data_fetcher_old.py`
|
|
- `market_old.py`
|
|
- `base_strategy.py` (The one in the root directory; the one in `strategies/` is used).
|
|
|
|
### One-Off Utility Scripts (Recommend moving to `scripts/`):
|
|
- `!migrate_to_sqlite.py`
|
|
- `import_csv.py`
|
|
- `del_market_cap_tables.py`
|
|
- `fix_timestamps.py`
|
|
- `list_coins.py`
|
|
- `create_agent.py`
|
|
|
|
### Examples / Unused Code:
|
|
- `basic_ws.py` (Appears to be an example file).
|
|
- `backtester.py`
|
|
- `strategy_sma_cross.py` (A strategy file in the root, not in the `strategies` folder).
|
|
- `strategy_template.py`
|
|
|
|
### Standalone / Potentially Unused Core Files:
|
|
The following files seem to have their logic already integrated into the main multi-process application. They might be remnants of a previous architecture and may not be needed as standalone scripts.
|
|
- `address_monitor.py`
|
|
- `position_monitor.py`
|
|
- `trade_log.py`
|
|
- `wallet_data.py`
|
|
- `whale_tracker.py`
|
|
|
|
### Data / Log Files (Recommend archiving or deleting):
|
|
- `hyperliquid_wallet_data_*.json` (These appear to be backups or logs). |