diff --git a/.gitignore b/.gitignore index 72bdfb1..fe03a56 100644 --- a/.gitignore +++ b/.gitignore @@ -38,4 +38,5 @@ agents/ .vscode/ .idea/ .DS_Store -Thumbs.db \ No newline at end of file +Thumbs.db +.opencode/ \ No newline at end of file diff --git a/WIKI/.gitattributes b/WIKI/.gitattributes deleted file mode 100644 index b6e88d6..0000000 --- a/WIKI/.gitattributes +++ /dev/null @@ -1,5 +0,0 @@ -# Treat markdown files as text with LF normalization -*.md text eol=lf - -# Ensure JSON files are treated as text -*.json text diff --git a/WIKI/AGENTS.md b/WIKI/AGENTS.md deleted file mode 100644 index 35e8a55..0000000 --- a/WIKI/AGENTS.md +++ /dev/null @@ -1,34 +0,0 @@ -Agents and Keys - -This project supports running multiple agent identities (private keys) to place orders on Hyperliquid. Agents are lightweight keys authorized on-chain by your main wallet. - -Agent storage and environment - -- For security, agent private keys should be stored as environment variables and not checked into source control. -- Supported patterns: - - `AGENT_PRIVATE_KEY` (single default agent) - - `_AGENT_PK` or `_AGENT_PRIVATE_KEY` (per-agent keys) - -Discovering agents - -- `trade_executor.py` scans environment variables for agent keys and loads them into `Exchange` objects so each agent can sign orders independently. - -Creating and authorizing agents - -- Use `create_agent.py` with your `MAIN_WALLET_PRIVATE_KEY` to authorize a new agent name. The script will attempt to call `exchange.approve_agent(agent_name)` and print the returned agent private key. - -Security notes - -- Never commit private keys to Git. Keep them in a secure secrets store or local `.env` file excluded from version control. -- Rotate keys if they are ever exposed and re-authorize agents using your main wallet. - -Example `.env` snippet - -MAIN_WALLET_PRIVATE_KEY= -MAIN_WALLET_ADDRESS= -AGENT_PRIVATE_KEY= -EXECUTOR_SCALPER_AGENT_PK= - -File `agents` - -- This repository may contain a local `agents` file used as a quick snapshot; treat it as insecure and remove it from the repo or add it to `.gitignore` if it contains secrets. diff --git a/WIKI/CONTRIBUTING.md b/WIKI/CONTRIBUTING.md deleted file mode 100644 index aca1168..0000000 --- a/WIKI/CONTRIBUTING.md +++ /dev/null @@ -1,20 +0,0 @@ -Contributing - -Thanks for considering contributing! Please follow these guidelines to make the process smooth. - -How to contribute - -1. Fork the repository and create a feature branch for your change. -2. Keep changes focused and add tests where appropriate. -3. Submit a Pull Request with a clear description and the reason for the change. - -Coding standards - -- Keep functions small and well-documented. -- Use the existing logging utilities for consistent output. -- Prefer safe, incremental changes for financial code. - -Security and secrets - -- Never commit private keys, API keys, or secrets. Use environment variables or a secrets manager. -- If you accidentally commit secrets, rotate them immediately. diff --git a/WIKI/DATA.md b/WIKI/DATA.md deleted file mode 100644 index f091f33..0000000 --- a/WIKI/DATA.md +++ /dev/null @@ -1,31 +0,0 @@ -Data layout and formats - -This section describes the `_data/` directory and the important files used by the scripts. - -Important files - -- `_data/market_data.db` — SQLite database that stores candle tables. Tables are typically named `_` (e.g., `BTC_1m`, `ETH_5m`). -- `_data/coin_precision.json` — Mapping of coin names to their size precision (created by `list_coins.py`). -- `_data/current_prices.json` — Latest market prices that `market.py` writes. -- `_data/fetcher_status.json` — Last run metadata from `data_fetcher.py`. -- `_data/market_cap_data.json` — Market cap summary saved by `market_cap_fetcher.py`. -- `_data/strategies.json` — Configuration for strategies (enabled flag, parameters). -- `_data/strategy_status_.json` — Per-strategy runtime status including last signal and price. -- `_data/executor_managed_positions.json` — Which strategy is currently managing which live position (used by `trade_executor`). - -Candle schema - -Each candle table contains columns similar to: -- `timestamp_ms` (INTEGER) — milliseconds since epoch -- `open`, `high`, `low`, `close` (FLOAT) -- `volume` (FLOAT) -- `number_of_trades` (INTEGER) - -Trade logs - -- Persistent trade history is stored in `_logs/trade_history.csv` with the following columns: `timestamp_utc`, `strategy`, `coin`, `action`, `price`, `size`, `signal`, `pnl`. - -Backups and maintenance - -- Periodically back up `_data/market_data.db`. The WAL and SHM files are also present when SQLite uses WAL mode. -- Keep JSON config/state files under version control only if they contain no secrets. diff --git a/WIKI/DEVELOPMENT.md b/WIKI/DEVELOPMENT.md deleted file mode 100644 index 7091f50..0000000 --- a/WIKI/DEVELOPMENT.md +++ /dev/null @@ -1,24 +0,0 @@ -Development and testing - -Code style and conventions - -- Python 3.11+ with typing hints where helpful. -- Use `logging_utils.setup_logging` for consistent logs across scripts. - -Running tests - -- This repository doesn't currently include a formal test suite. Suggested quick checks: - - Run `python list_coins.py` to verify connectivity to Hyperliquid Info. - - Run `python -m pyflakes .` or `python -m pylint` if you have linters installed. - -Adding a new strategy - -1. Create a new script following the pattern in `strategy_template.py`. -2. Add an entry to `_data/strategies.json` with `enabled: true` and relevant parameters. -3. Ensure the strategy writes a status JSON file (`_data/strategy_status_.json`) and uses `trade_log.log_trade` to record actions. - -Recommended improvements (low-risk) - -- Add a lightweight unit test suite (pytest) for core functions like timeframe parsing, SQL helpers, and signal calculation. -- Add CI (GitHub Actions) to run flake/pylint and unit tests on PRs. -- Move secrets handling to a `.env.example` and document environment variables in `WIKI/SETUP.md`. diff --git a/WIKI/OVERVIEW.md b/WIKI/OVERVIEW.md deleted file mode 100644 index 51f5063..0000000 --- a/WIKI/OVERVIEW.md +++ /dev/null @@ -1,29 +0,0 @@ -Hyperliquid Trading Toolkit - -This repository contains a collection of utility scripts, data fetchers, resamplers, trading strategies, and a trade executor for working with Hyperliquid trading APIs and crawled data. It is organized to support data collection, transformation, strategy development, and automated execution via agents. - -Key components - -- Data fetching and management: `data_fetcher.py`, `market.py`, `resampler.py`, `market_cap_fetcher.py`, `list_coins.py` -- Strategies: `strategy_sma_cross.py`, `strategy_template.py`, `strategy_sma_125d.py` (if present) -- Execution: `trade_executor.py`, `create_agent.py`, `agents` helper -- Utilities: `logging_utils.py`, `trade_log.py` -- Data storage: SQLite database in `_data/market_data.db` and JSON files in `_data` - -Intended audience - -- Developers building strategies and automations on Hyperliquid -- Data engineers collecting and processing market data -- Operators running the fetchers and executors on a scheduler or as system services - -Project goals - -- Reliable collection of 1m candles and resampling to common timeframes -- Clean separation between data, strategies, and execution -- Lightweight logging and traceable trade records - -Where to start - -- Read `WIKI/SETUP.md` to prepare your environment -- Use `WIKI/SCRIPTS.md` for a description of individual scripts and how to run them -- Inspect `WIKI/AGENTS.md` to understand agent keys and how to manage them diff --git a/WIKI/SCRIPTS.md b/WIKI/SCRIPTS.md deleted file mode 100644 index c5bd179..0000000 --- a/WIKI/SCRIPTS.md +++ /dev/null @@ -1,47 +0,0 @@ -Scripts and How to Use Them - -This file documents the main scripts in the repository and their purpose, typical runtime parameters, and key notes. - -list_coins.py -- Purpose: Fetches asset metadata from Hyperliquid (name and size/precision) and saves `_data/coin_precision.json`. -- Usage: `python list_coins.py` -- Notes: Reads `hyperliquid.info.Info` and writes a JSON file. Useful to run before market feeders. - -market.py (MarketDataFeeder) -- Purpose: Fetches live prices from Hyperliquid and writes `_data/current_prices.json` while printing a live table. -- Usage: `python market.py --log-level normal` -- Notes: Expects `_data/coin_precision.json` to exist. - -data_fetcher.py (CandleFetcherDB) -- Purpose: Fetches historical 1m candles and stores them in `_data/market_data.db` using a table-per-coin naming convention. -- Usage: `python data_fetcher.py --coins BTC ETH --interval 1m --days 7` -- Notes: Can be run regularly by a scheduler to keep the DB up to date. - -resampler.py (Resampler) -- Purpose: Reads 1m candles from SQLite and resamples to configured timeframes (e.g. 5m, 15m, 1h), appending new candles to tables. -- Usage: `python resampler.py --coins BTC ETH --timeframes 5m 15m 1h --log-level normal` - -market_cap_fetcher.py (MarketCapFetcher) -- Purpose: Pulls CoinGecko market cap numbers and maintains historical daily tables in the same SQLite DB. -- Usage: `python market_cap_fetcher.py --coins BTC ETH --log-level normal` -- Notes: Optional `COINGECKO_API_KEY` in `.env` avoids throttling. - -strategy_sma_cross.py (SmaCrossStrategy) -- Purpose: Run an SMA-based trading strategy. Reads candles from `_data/market_data.db` and writes status to `_data/strategy_status_.json`. -- Usage: `python strategy_sma_cross.py --name sma_cross_1 --params '{"coin":"BTC","timeframe":"1m","fast":5,"slow":20}' --log-level normal` - -trade_executor.py (TradeExecutor) -- Purpose: Orchestrates agent-based order execution using agent private keys found in environment variables. Uses `_data/strategies.json` to determine active strategies. -- Usage: `python trade_executor.py --log-level normal` -- Notes: Requires `MAIN_WALLET_ADDRESS` and agent keys. See `create_agent.py` to authorize agents on-chain. - -create_agent.py -- Purpose: Authorizes a new on-chain agent using your main wallet (requires `MAIN_WALLET_PRIVATE_KEY`). -- Usage: `python create_agent.py` -- Notes: Prints the new agent private key to stdout — save it securely. - -trade_log.py -- Purpose: Provides a thread-safe CSV trade history logger. Used by the executor and strategies to record actions. - -Other utility scripts -- import_csv.py, fix_timestamps.py, list_coins.py, etc. — see file headers for details. diff --git a/WIKI/SETUP.md b/WIKI/SETUP.md deleted file mode 100644 index 0022342..0000000 --- a/WIKI/SETUP.md +++ /dev/null @@ -1,42 +0,0 @@ -Setup and Installation - -Prerequisites - -- Python 3.11+ (project uses modern dependencies) -- Git (optional) -- A Hyperliquid account and an activated main wallet if you want to authorize agents and trade - -Virtual environment - -1. Create a virtual environment: - - python -m venv .venv - -2. Activate the virtual environment (PowerShell on Windows): - - .\.venv\Scripts\Activate.ps1 - -3. Upgrade pip and install dependencies: - - python -m pip install --upgrade pip - pip install -r requirements.txt - -Configuration - -- Copy `.env.example` to `.env` and set the following variables as required: - - MAIN_WALLET_PRIVATE_KEY (used by `create_agent.py` to authorize agents) - - MAIN_WALLET_ADDRESS (used by `trade_executor.py`) - - AGENT_PRIVATE_KEY or per-agent keys like `EXECUTOR_SCALPER_AGENT_PK` - - Optional: COINGECKO_API_KEY for `market_cap_fetcher.py` to avoid rate limits - -Data directory - -- The project writes and reads data from the `_data/` folder. Ensure the directory exists and is writable by the user running the scripts. - -Quick test - -After installing packages, run `list_coins.py` in a dry run to verify connectivity to the Hyperliquid info API: - - python list_coins.py - -If you encounter import errors, ensure the virtual environment is active and the `requirements.txt` dependencies are installed. diff --git a/WIKI/SUMMARY.md b/WIKI/SUMMARY.md deleted file mode 100644 index 55fe447..0000000 --- a/WIKI/SUMMARY.md +++ /dev/null @@ -1,15 +0,0 @@ -Project Wiki Summary - -This directory contains human-friendly documentation for the project. Files: - -- `OVERVIEW.md` — High-level overview and where to start -- `SETUP.md` — Environment setup and quick test steps -- `SCRIPTS.md` — Per-script documentation and usage examples -- `AGENTS.md` — How agents work and secure handling of keys -- `DATA.md` — Data folder layout and schema notes -- `DEVELOPMENT.md` — Developer guidance and recommended improvements -- `CONTRIBUTING.md` — How to contribute safely -- `TROUBLESHOOTING.md` — Common problems and solutions - -Notes: -- These pages were generated from repository source files and common patterns in trading/data projects. Validate any sensitive information (agent keys) and remove them from the repository when sharing. diff --git a/WIKI/TROUBLESHOOTING.md b/WIKI/TROUBLESHOOTING.md deleted file mode 100644 index 6320ff2..0000000 --- a/WIKI/TROUBLESHOOTING.md +++ /dev/null @@ -1,21 +0,0 @@ -Troubleshooting common issues - -1. Import errors - - Ensure the virtual environment is active. - - Run `pip install -r requirements.txt`. - -2. Agent authorization failures - - Ensure your main wallet is activated on Hyperliquid and has funds. - - The `create_agent.py` script will print helpful messages if the vault (main wallet) cannot act. - -3. SQLite locked errors - - Increase the SQLite timeout when opening connections (this project uses a 10s timeout in fetcher). Close other programs that may hold the DB open. - -4. Missing coin precision file - - Run `python list_coins.py` to regenerate `_data/coin_precision.json`. - -5. Rate limits from CoinGecko - - Set `COINGECKO_API_KEY` in your `.env` file and ensure the fetcher respects backoff. - -6. Agent keys in `agents` file or other local files - - Treat any `agents` file with private keys as compromised; rotate keys and remove the file from the repository.