From 2b558511363931494b5415a959e5ea51afa8ec05 Mon Sep 17 00:00:00 2001 From: DiTus Date: Sat, 18 Oct 2025 15:55:53 +0200 Subject: [PATCH] wiki --- .env.example | 24 +++++++++++++ README.md | 9 +++++ WIKI/.gitattributes | 5 +++ WIKI/AGENTS.md | 34 +++++++++++++++++++ WIKI/CONTRIBUTING.md | 20 +++++++++++ WIKI/DATA.md | 31 +++++++++++++++++ WIKI/DEVELOPMENT.md | 24 +++++++++++++ WIKI/OVERVIEW.md | 29 ++++++++++++++++ WIKI/SCRIPTS.md | 47 ++++++++++++++++++++++++++ WIKI/SETUP.md | 42 +++++++++++++++++++++++ WIKI/SUMMARY.md | 15 ++++++++ WIKI/TROUBLESHOOTING.md | 21 ++++++++++++ _data/strategy_status_sma_cross_1.json | 6 ++-- _data/strategy_status_sma_cross_2.json | 2 +- 14 files changed, 305 insertions(+), 4 deletions(-) create mode 100644 .env.example create mode 100644 WIKI/.gitattributes create mode 100644 WIKI/AGENTS.md create mode 100644 WIKI/CONTRIBUTING.md create mode 100644 WIKI/DATA.md create mode 100644 WIKI/DEVELOPMENT.md create mode 100644 WIKI/OVERVIEW.md create mode 100644 WIKI/SCRIPTS.md create mode 100644 WIKI/SETUP.md create mode 100644 WIKI/SUMMARY.md create mode 100644 WIKI/TROUBLESHOOTING.md diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..1e508fa --- /dev/null +++ b/.env.example @@ -0,0 +1,24 @@ +# Example environment variables for the Hyperliquid trading toolkit +# Copy this file to .env and fill in real values. Do NOT commit your real .env file. + +# Main wallet (used only to authorize agents on-chain) +# Example: MAIN_WALLET_PRIVATE_KEY=0x... +MAIN_WALLET_PRIVATE_KEY= +MAIN_WALLET_ADDRESS= + +# Agent keys (private keys authorized via create_agent.py) +# Preferred patterns: +# - AGENT_PRIVATE_KEY: default agent +# - _AGENT_PK or _AGENT_PRIVATE_KEY: per-agent keys (e.g., SCALPER_AGENT_PK) +# Example: AGENT_PRIVATE_KEY=0x... +AGENT_PRIVATE_KEY= +# Example per-agent key: +# SCALPER_AGENT_PK= +# SWING_AGENT_PK= + +# Optional: CoinGecko API key to reduce rate limits for market cap fetches +COINGECKO_API_KEY= + +# Optional: Set a custom environment for development/testing +# E.g., DEBUG=true +DEBUG= diff --git a/README.md b/README.md index 3cb529d..897c952 100644 --- a/README.md +++ b/README.md @@ -77,3 +77,12 @@ Before running the application, you must configure your wallets, agents, and API ```bash python main_app.py ``` + + ## Documentation + + Detailed project documentation is available in the `WIKI/` directory. Start with the summary page: + + `WIKI/SUMMARY.md` + + This contains links and explanations for `OVERVIEW.md`, `SETUP.md`, `SCRIPTS.md`, and other helpful pages that describe usage, data layout, agent management, development notes, and troubleshooting. + diff --git a/WIKI/.gitattributes b/WIKI/.gitattributes new file mode 100644 index 0000000..b6e88d6 --- /dev/null +++ b/WIKI/.gitattributes @@ -0,0 +1,5 @@ +# Treat markdown files as text with LF normalization +*.md text eol=lf + +# Ensure JSON files are treated as text +*.json text diff --git a/WIKI/AGENTS.md b/WIKI/AGENTS.md new file mode 100644 index 0000000..35e8a55 --- /dev/null +++ b/WIKI/AGENTS.md @@ -0,0 +1,34 @@ +Agents and Keys + +This project supports running multiple agent identities (private keys) to place orders on Hyperliquid. Agents are lightweight keys authorized on-chain by your main wallet. + +Agent storage and environment + +- For security, agent private keys should be stored as environment variables and not checked into source control. +- Supported patterns: + - `AGENT_PRIVATE_KEY` (single default agent) + - `_AGENT_PK` or `_AGENT_PRIVATE_KEY` (per-agent keys) + +Discovering agents + +- `trade_executor.py` scans environment variables for agent keys and loads them into `Exchange` objects so each agent can sign orders independently. + +Creating and authorizing agents + +- Use `create_agent.py` with your `MAIN_WALLET_PRIVATE_KEY` to authorize a new agent name. The script will attempt to call `exchange.approve_agent(agent_name)` and print the returned agent private key. + +Security notes + +- Never commit private keys to Git. Keep them in a secure secrets store or local `.env` file excluded from version control. +- Rotate keys if they are ever exposed and re-authorize agents using your main wallet. + +Example `.env` snippet + +MAIN_WALLET_PRIVATE_KEY= +MAIN_WALLET_ADDRESS= +AGENT_PRIVATE_KEY= +EXECUTOR_SCALPER_AGENT_PK= + +File `agents` + +- This repository may contain a local `agents` file used as a quick snapshot; treat it as insecure and remove it from the repo or add it to `.gitignore` if it contains secrets. diff --git a/WIKI/CONTRIBUTING.md b/WIKI/CONTRIBUTING.md new file mode 100644 index 0000000..aca1168 --- /dev/null +++ b/WIKI/CONTRIBUTING.md @@ -0,0 +1,20 @@ +Contributing + +Thanks for considering contributing! Please follow these guidelines to make the process smooth. + +How to contribute + +1. Fork the repository and create a feature branch for your change. +2. Keep changes focused and add tests where appropriate. +3. Submit a Pull Request with a clear description and the reason for the change. + +Coding standards + +- Keep functions small and well-documented. +- Use the existing logging utilities for consistent output. +- Prefer safe, incremental changes for financial code. + +Security and secrets + +- Never commit private keys, API keys, or secrets. Use environment variables or a secrets manager. +- If you accidentally commit secrets, rotate them immediately. diff --git a/WIKI/DATA.md b/WIKI/DATA.md new file mode 100644 index 0000000..f091f33 --- /dev/null +++ b/WIKI/DATA.md @@ -0,0 +1,31 @@ +Data layout and formats + +This section describes the `_data/` directory and the important files used by the scripts. + +Important files + +- `_data/market_data.db` — SQLite database that stores candle tables. Tables are typically named `_` (e.g., `BTC_1m`, `ETH_5m`). +- `_data/coin_precision.json` — Mapping of coin names to their size precision (created by `list_coins.py`). +- `_data/current_prices.json` — Latest market prices that `market.py` writes. +- `_data/fetcher_status.json` — Last run metadata from `data_fetcher.py`. +- `_data/market_cap_data.json` — Market cap summary saved by `market_cap_fetcher.py`. +- `_data/strategies.json` — Configuration for strategies (enabled flag, parameters). +- `_data/strategy_status_.json` — Per-strategy runtime status including last signal and price. +- `_data/executor_managed_positions.json` — Which strategy is currently managing which live position (used by `trade_executor`). + +Candle schema + +Each candle table contains columns similar to: +- `timestamp_ms` (INTEGER) — milliseconds since epoch +- `open`, `high`, `low`, `close` (FLOAT) +- `volume` (FLOAT) +- `number_of_trades` (INTEGER) + +Trade logs + +- Persistent trade history is stored in `_logs/trade_history.csv` with the following columns: `timestamp_utc`, `strategy`, `coin`, `action`, `price`, `size`, `signal`, `pnl`. + +Backups and maintenance + +- Periodically back up `_data/market_data.db`. The WAL and SHM files are also present when SQLite uses WAL mode. +- Keep JSON config/state files under version control only if they contain no secrets. diff --git a/WIKI/DEVELOPMENT.md b/WIKI/DEVELOPMENT.md new file mode 100644 index 0000000..7091f50 --- /dev/null +++ b/WIKI/DEVELOPMENT.md @@ -0,0 +1,24 @@ +Development and testing + +Code style and conventions + +- Python 3.11+ with typing hints where helpful. +- Use `logging_utils.setup_logging` for consistent logs across scripts. + +Running tests + +- This repository doesn't currently include a formal test suite. Suggested quick checks: + - Run `python list_coins.py` to verify connectivity to Hyperliquid Info. + - Run `python -m pyflakes .` or `python -m pylint` if you have linters installed. + +Adding a new strategy + +1. Create a new script following the pattern in `strategy_template.py`. +2. Add an entry to `_data/strategies.json` with `enabled: true` and relevant parameters. +3. Ensure the strategy writes a status JSON file (`_data/strategy_status_.json`) and uses `trade_log.log_trade` to record actions. + +Recommended improvements (low-risk) + +- Add a lightweight unit test suite (pytest) for core functions like timeframe parsing, SQL helpers, and signal calculation. +- Add CI (GitHub Actions) to run flake/pylint and unit tests on PRs. +- Move secrets handling to a `.env.example` and document environment variables in `WIKI/SETUP.md`. diff --git a/WIKI/OVERVIEW.md b/WIKI/OVERVIEW.md new file mode 100644 index 0000000..51f5063 --- /dev/null +++ b/WIKI/OVERVIEW.md @@ -0,0 +1,29 @@ +Hyperliquid Trading Toolkit + +This repository contains a collection of utility scripts, data fetchers, resamplers, trading strategies, and a trade executor for working with Hyperliquid trading APIs and crawled data. It is organized to support data collection, transformation, strategy development, and automated execution via agents. + +Key components + +- Data fetching and management: `data_fetcher.py`, `market.py`, `resampler.py`, `market_cap_fetcher.py`, `list_coins.py` +- Strategies: `strategy_sma_cross.py`, `strategy_template.py`, `strategy_sma_125d.py` (if present) +- Execution: `trade_executor.py`, `create_agent.py`, `agents` helper +- Utilities: `logging_utils.py`, `trade_log.py` +- Data storage: SQLite database in `_data/market_data.db` and JSON files in `_data` + +Intended audience + +- Developers building strategies and automations on Hyperliquid +- Data engineers collecting and processing market data +- Operators running the fetchers and executors on a scheduler or as system services + +Project goals + +- Reliable collection of 1m candles and resampling to common timeframes +- Clean separation between data, strategies, and execution +- Lightweight logging and traceable trade records + +Where to start + +- Read `WIKI/SETUP.md` to prepare your environment +- Use `WIKI/SCRIPTS.md` for a description of individual scripts and how to run them +- Inspect `WIKI/AGENTS.md` to understand agent keys and how to manage them diff --git a/WIKI/SCRIPTS.md b/WIKI/SCRIPTS.md new file mode 100644 index 0000000..c5bd179 --- /dev/null +++ b/WIKI/SCRIPTS.md @@ -0,0 +1,47 @@ +Scripts and How to Use Them + +This file documents the main scripts in the repository and their purpose, typical runtime parameters, and key notes. + +list_coins.py +- Purpose: Fetches asset metadata from Hyperliquid (name and size/precision) and saves `_data/coin_precision.json`. +- Usage: `python list_coins.py` +- Notes: Reads `hyperliquid.info.Info` and writes a JSON file. Useful to run before market feeders. + +market.py (MarketDataFeeder) +- Purpose: Fetches live prices from Hyperliquid and writes `_data/current_prices.json` while printing a live table. +- Usage: `python market.py --log-level normal` +- Notes: Expects `_data/coin_precision.json` to exist. + +data_fetcher.py (CandleFetcherDB) +- Purpose: Fetches historical 1m candles and stores them in `_data/market_data.db` using a table-per-coin naming convention. +- Usage: `python data_fetcher.py --coins BTC ETH --interval 1m --days 7` +- Notes: Can be run regularly by a scheduler to keep the DB up to date. + +resampler.py (Resampler) +- Purpose: Reads 1m candles from SQLite and resamples to configured timeframes (e.g. 5m, 15m, 1h), appending new candles to tables. +- Usage: `python resampler.py --coins BTC ETH --timeframes 5m 15m 1h --log-level normal` + +market_cap_fetcher.py (MarketCapFetcher) +- Purpose: Pulls CoinGecko market cap numbers and maintains historical daily tables in the same SQLite DB. +- Usage: `python market_cap_fetcher.py --coins BTC ETH --log-level normal` +- Notes: Optional `COINGECKO_API_KEY` in `.env` avoids throttling. + +strategy_sma_cross.py (SmaCrossStrategy) +- Purpose: Run an SMA-based trading strategy. Reads candles from `_data/market_data.db` and writes status to `_data/strategy_status_.json`. +- Usage: `python strategy_sma_cross.py --name sma_cross_1 --params '{"coin":"BTC","timeframe":"1m","fast":5,"slow":20}' --log-level normal` + +trade_executor.py (TradeExecutor) +- Purpose: Orchestrates agent-based order execution using agent private keys found in environment variables. Uses `_data/strategies.json` to determine active strategies. +- Usage: `python trade_executor.py --log-level normal` +- Notes: Requires `MAIN_WALLET_ADDRESS` and agent keys. See `create_agent.py` to authorize agents on-chain. + +create_agent.py +- Purpose: Authorizes a new on-chain agent using your main wallet (requires `MAIN_WALLET_PRIVATE_KEY`). +- Usage: `python create_agent.py` +- Notes: Prints the new agent private key to stdout — save it securely. + +trade_log.py +- Purpose: Provides a thread-safe CSV trade history logger. Used by the executor and strategies to record actions. + +Other utility scripts +- import_csv.py, fix_timestamps.py, list_coins.py, etc. — see file headers for details. diff --git a/WIKI/SETUP.md b/WIKI/SETUP.md new file mode 100644 index 0000000..0022342 --- /dev/null +++ b/WIKI/SETUP.md @@ -0,0 +1,42 @@ +Setup and Installation + +Prerequisites + +- Python 3.11+ (project uses modern dependencies) +- Git (optional) +- A Hyperliquid account and an activated main wallet if you want to authorize agents and trade + +Virtual environment + +1. Create a virtual environment: + + python -m venv .venv + +2. Activate the virtual environment (PowerShell on Windows): + + .\.venv\Scripts\Activate.ps1 + +3. Upgrade pip and install dependencies: + + python -m pip install --upgrade pip + pip install -r requirements.txt + +Configuration + +- Copy `.env.example` to `.env` and set the following variables as required: + - MAIN_WALLET_PRIVATE_KEY (used by `create_agent.py` to authorize agents) + - MAIN_WALLET_ADDRESS (used by `trade_executor.py`) + - AGENT_PRIVATE_KEY or per-agent keys like `EXECUTOR_SCALPER_AGENT_PK` + - Optional: COINGECKO_API_KEY for `market_cap_fetcher.py` to avoid rate limits + +Data directory + +- The project writes and reads data from the `_data/` folder. Ensure the directory exists and is writable by the user running the scripts. + +Quick test + +After installing packages, run `list_coins.py` in a dry run to verify connectivity to the Hyperliquid info API: + + python list_coins.py + +If you encounter import errors, ensure the virtual environment is active and the `requirements.txt` dependencies are installed. diff --git a/WIKI/SUMMARY.md b/WIKI/SUMMARY.md new file mode 100644 index 0000000..55fe447 --- /dev/null +++ b/WIKI/SUMMARY.md @@ -0,0 +1,15 @@ +Project Wiki Summary + +This directory contains human-friendly documentation for the project. Files: + +- `OVERVIEW.md` — High-level overview and where to start +- `SETUP.md` — Environment setup and quick test steps +- `SCRIPTS.md` — Per-script documentation and usage examples +- `AGENTS.md` — How agents work and secure handling of keys +- `DATA.md` — Data folder layout and schema notes +- `DEVELOPMENT.md` — Developer guidance and recommended improvements +- `CONTRIBUTING.md` — How to contribute safely +- `TROUBLESHOOTING.md` — Common problems and solutions + +Notes: +- These pages were generated from repository source files and common patterns in trading/data projects. Validate any sensitive information (agent keys) and remove them from the repository when sharing. diff --git a/WIKI/TROUBLESHOOTING.md b/WIKI/TROUBLESHOOTING.md new file mode 100644 index 0000000..6320ff2 --- /dev/null +++ b/WIKI/TROUBLESHOOTING.md @@ -0,0 +1,21 @@ +Troubleshooting common issues + +1. Import errors + - Ensure the virtual environment is active. + - Run `pip install -r requirements.txt`. + +2. Agent authorization failures + - Ensure your main wallet is activated on Hyperliquid and has funds. + - The `create_agent.py` script will print helpful messages if the vault (main wallet) cannot act. + +3. SQLite locked errors + - Increase the SQLite timeout when opening connections (this project uses a 10s timeout in fetcher). Close other programs that may hold the DB open. + +4. Missing coin precision file + - Run `python list_coins.py` to regenerate `_data/coin_precision.json`. + +5. Rate limits from CoinGecko + - Set `COINGECKO_API_KEY` in your `.env` file and ensure the fetcher respects backoff. + +6. Agent keys in `agents` file or other local files + - Treat any `agents` file with private keys as compromised; rotate keys and remove the file from the repository. diff --git a/_data/strategy_status_sma_cross_1.json b/_data/strategy_status_sma_cross_1.json index 91d1414..c83b076 100644 --- a/_data/strategy_status_sma_cross_1.json +++ b/_data/strategy_status_sma_cross_1.json @@ -1,7 +1,7 @@ { "strategy_name": "sma_cross_1", "current_signal": "SELL", - "last_signal_change_utc": "2025-10-18T13:02:00+00:00", - "signal_price": 3873.4, - "last_checked_utc": "2025-10-18T13:09:05.009625+00:00" + "last_signal_change_utc": "2025-10-18T13:03:00+00:00", + "signal_price": 3871.9, + "last_checked_utc": "2025-10-18T13:55:05.015097+00:00" } \ No newline at end of file diff --git a/_data/strategy_status_sma_cross_2.json b/_data/strategy_status_sma_cross_2.json index e0f6cbc..e5d23a6 100644 --- a/_data/strategy_status_sma_cross_2.json +++ b/_data/strategy_status_sma_cross_2.json @@ -3,5 +3,5 @@ "current_signal": "SELL", "last_signal_change_utc": "2025-10-14T00:00:00+00:00", "signal_price": 113026.0, - "last_checked_utc": "2025-10-18T13:13:03.538350+00:00" + "last_checked_utc": "2025-10-18T13:55:45.714315+00:00" } \ No newline at end of file