hyper/WIKI/symbol_management.md

# Symbol Management Guide

This guide explains how to add or remove Hyperliquid trading symbols (coins) from the trading bot's dashboard, data pipeline, and market cap tracking.

## Overview

The system tracks coins through multiple interconnected components. Each component reads its coin list from a specific source:

| Component | Source | Purpose |
|-----------|--------|---------|
| Dashboard display | `WATCHED_COINS` in `main_app.py` | Shows live prices in terminal |
| Live candle fetcher | `--coins` CLI arg (from `WATCHED_COINS`) | Collects 1-minute candle data |
| Resampler | `--coins` CLI arg (from `WATCHED_COINS`) | Resamples 1m data to 15+ timeframes |
| Live price feed | `coins_to_watch` arg (from `WATCHED_COINS`) | WebSocket BBO/trade subscriptions |
| Market cap fetcher | `coin_id_map.json` | CoinGecko market cap data |
| Resampling status | `resampling_status.json` | Tracks progress per coin/timeframe |
| Market cap summary | `market_cap_data.json` | Aggregated market cap snapshots |

## Data Pipeline

```
Hyperliquid WebSocket
    |
    +---> Live Candle Fetcher (1m candles) --> SQLite: {coin}_1m
    |         |
    |         +---> Resampler --> SQLite: {coin}_{3m,5m,15m,...,1M}
    |
    +---> Live Price Feed (BBO/trades) --> shared_prices dict --> Dashboard

CoinGecko API
    |
    +---> Market Cap Fetcher --> SQLite: {coin}_market_cap
                            --> market_cap_data.json (summary)
```

All historical data is stored in `_data/market_data.db` (SQLite). Existing data is **preserved** when removing coins; only new data collection stops.

---

## Adding a Symbol

### Step 1: Add to the Watched Coins List

Edit `main_app.py` (line 23):

```python
WATCHED_COINS = ["BTC", "ETH", "SOL", "BNB", "HYPE", "NEW_COIN", "xyz:BRENTOIL", "xyz:CL"]
```

### Step 2: Add Display Name (Optional)

If the symbol contains special characters or you want a custom display name, add it to `COIN_DISPLAY_NAMES` in `main_app.py` (lines 25-28):

```python
COIN_DISPLAY_NAMES = {
    "xyz:BRENTOIL": "BRENT",
    "xyz:CL": "WTI",
    "NEW_COIN": "NewCoin"
}
```

### Step 3: Add to Coin ID Map (for Market Cap)

Edit `_data/coin_id_map.json` and add an entry mapping the Hyperliquid symbol to the CoinGecko ID:

```json
"NEW_COIN": "new-coin-id-on-coingecko"
```

If the coin is already in the map (e.g., it was previously fetched), skip this step.

### Step 4: Add to Manual Overrides (Optional)

If the CoinGecko ID is ambiguous, add it to the `manual_overrides` dictionary in `coin_id_map.py` (lines 49-61):

```python
manual_overrides = {
    "BTC": "bitcoin",
    "ETH": "ethereum",
    "NEW_COIN": "new-coin-id-on-coingecko",
    ...
}
```

### Step 5: Restart the Application

Stop all running processes, then start `main_app.py`:

```bash
python main_app.py
```

The system will automatically:
- Create new candle tables in `market_data.db`
- Begin collecting 1-minute candle data
- Begin resampling to all timeframes
- Begin collecting market cap data
- Display the coin on the dashboard

---

## Removing a Symbol

### Step 1: Stop All Running Processes

Before making changes, stop all Python processes related to the project:

```powershell
# Find running processes
Get-WmiObject Win32_Process | Where-Object { $_.ExecutablePath -like "*python*" -and $_.CommandLine -like "*hyper*" }

# Stop them (replace PIDs with actual values)
Stop-Process -Id <PID1>, <PID2>, ... -Force
```

### Step 2: Remove from Watched Coins List

Edit `main_app.py` (line 23) and remove the coin from `WATCHED_COINS`:

```python
WATCHED_COINS = ["BTC", "ETH", "SOL", "BNB", "HYPE", "SUI", "xyz:BRENTOIL", "xyz:CL"]
```

### Step 3: Remove from Resampling Status

Edit `_data/resampling_status.json` and delete the entire block for the coin, e.g.:

```json
"REMOVED_COIN": {
    "12h": { ... },
    "148m": { ... },
    ...
}
```

### Step 4: Remove from Coin ID Map

Edit `_data/coin_id_map.json` and delete the entry:

```json
"REMOVED_COIN": "coingecko-id"
```

### Step 5: Remove from Market Cap Summary

Edit `_data/market_cap_data.json` and delete the entry:

```json
"REMOVED_COIN_market_cap": { ... }
```

### Step 6: Remove from Manual Overrides (if present)

Edit `coin_id_map.py` and remove the entry from `manual_overrides`:

```python
manual_overrides = {
    "BTC": "bitcoin",
    "ETH": "ethereum",
    ...
}
```

### Step 7: Restart the Application

```bash
python main_app.py
```

**Note:** Existing data in `_data/market_data.db` (candle tables, market cap tables) is **not deleted**. The coin's data remains available for historical analysis; only new data collection stops.

---

## File Reference

### Core Configuration

| File | Line | Description |
|------|------|-------------|
| `main_app.py` | 23 | `WATCHED_COINS` list - master coin list for dashboard, candle fetcher, resampler, and live feed |
| `main_app.py` | 25-28 | `COIN_DISPLAY_NAMES` dict - maps internal symbols to display names |
| `main_app.py` | 591-594 | `required_timeframes` list - timeframes for resampling |

### Data Files

| File | Description |
|------|-------------|
| `_data/market_data.db` | SQLite database with all candle and market cap data. Tables: `{coin}_1m`, `{coin}_{timeframe}`, `{coin}_market_cap` |
| `_data/resampling_status.json` | Tracks `last_candle_utc` and `total_candles` per coin/timeframe |
| `_data/coin_id_map.json` | Maps Hyperliquid symbols to CoinGecko IDs for market cap fetching |
| `_data/market_cap_data.json` | Summary of latest market cap data per coin |
| `_data/coin_precision.json` | All Hyperliquid coins with trade precision (reference only) |
| `_data/strategies.json` | Trading strategy configurations (separate from watched coins) |

### Scripts

| File | Description |
|------|-------------|
| `main_app.py` | Main orchestrator - starts all processes, renders dashboard |
| `live_candle_fetcher.py` | Collects 1-minute candles via WebSocket + historical catch-up |
| `resampler.py` | Resamples 1m candles to multiple timeframes using pandas |
| `live_market_utils.py` | WebSocket feed for live BBO (best bid/offer) and trade data |
| `market_cap_fetcher.py` | Fetches daily market cap data from CoinGecko API |
| `coin_id_map.py` | Generates `coin_id_map.json` from Hyperliquid + CoinGecko APIs |
| `dashboard_data_fetcher.py` | Fetches account balances and positions for dashboard |

---

## Important Notes

1. **Always stop processes before editing config files.** Running processes will overwrite changes to `resampling_status.json` and `market_data.db`.

2. **Existing data is preserved.** Removing a coin from the lists stops new data collection but does not delete existing data from the SQLite database.

3. **The `coin_id_map.json` is auto-generated.** Running `python coin_id_map.py` regenerates it from the Hyperliquid API. Manual overrides in `coin_id_map.py` ensure correct CoinGecko mappings.

4. **Market cap fetcher is not auto-started.** The market cap fetcher process is currently disabled in `main_app.py` (line 614). It can be run manually: `python market_cap_fetcher.py`.

5. **Strategy coins are separate.** Trading strategies in `_data/strategies.json` define their own coins independently of `WATCHED_COINS`. A coin can be traded by a strategy even if it's not in the watched list.

6. **Special symbols.** Coins with the `xyz:` prefix (e.g., `xyz:BRENTOIL`, `xyz:CL`) are synthetic/derivative symbols on Hyperliquid. They follow the same management process as regular coins.