test: add comprehensive Hurst strategy simulations and backtest engine enhancements (v1.7.11)

- Add Hurst_simulations.md with detailed performance data from Jan 2025 to Mar 2026 across 1m-1h TFs. - Enhance backtest_engine.py: - Add --starting_equity for memory-efficient chunked testing. - Add Max Drawdown tracking. - Add --interval override flag. - Add --brake_period/--brake_interval for safety filter testing.
feat: implement Mid-Price logic and separate error logging (v1.8.6)
2026-03-10 13:02:29 +01:00 · 2026-03-08 22:07:11 +01:00 · 2026-03-08 20:39:17 +01:00 · 2026-03-08 20:12:16 +01:00
5 changed files with 587 additions and 117 deletions
--- a/BACKTESTING_GUIDE.md
+++ b/BACKTESTING_GUIDE.md
@ -10,7 +10,23 @@ To run the backtester, use the following command:
 docker exec -it btc_ping_pong_bot python src/strategies/backtest_engine.py
 ```

-## 2. Backtest Engine (`backtest_engine.py`)
+## 2. Local PC Setup (Recommended for Memory Savings)
+Running the backtest on your local machine is much faster and saves server memory.
+
+### Steps:
+1.  **Clone/Sync your repo** to your local machine.
+2.  **Install dependencies**:
+    ```bash
+    pip install -r requirements.txt
+    ```
+3.  **Configure `.env`**: Ensure your local `.env` file has the correct `DB_HOST` (e.g., `20.20.20.20`).
+4.  **Run the engine**:
+    ```bash
+    # From the project root
+    python3 src/strategies/backtest_engine.py --start_date 2024-01-01 --end_date 2024-01-31
+    ```
+
+## 3. Backtest Engine (`backtest_engine.py`)
 The backtest engine reuses the core logic from `ping_pong_bot.py` via the `PingPongStrategy` class.

 ### Key Features:
@ -25,23 +41,35 @@ The backtest engine reuses the core logic from `ping_pong_bot.py` via the `PingP
 docker exec -it btc_ping_pong_bot python src/strategies/backtest_engine.py
 ```

-## 3. Strategy Optimization (Optional)
-To find the absolute best parameters for RSI and Hurst, you can use **Optuna**. 
+## 3. Regime Testing (MA Switching)
+You can test different Moving Average (MA) settings to see which regime detector works best for switching between `long` and `short` modes.

-### Installation:
-Inside the Docker container:
-```bash
-docker exec -it btc_ping_pong_bot pip install optuna
-```
+### Examples:
+*   **Test 15m SMA 200:**
+    ```bash
+    docker exec -it btc_ping_pong_bot python src/strategies/backtest_engine.py --ma_period 200 --ma_interval 15m
+    ```
+*   **Test 1h SMA 50:**
+    ```bash
+    docker exec -it btc_ping_pong_bot python src/strategies/backtest_engine.py --ma_period 50 --ma_interval 1h
+    ```
+*   **Test 4h SMA 100:**
+    ```bash
+    docker exec -it btc_ping_pong_bot python src/strategies/backtest_engine.py --ma_period 100 --ma_interval 4h --limit 20000
+    ```

-### Planned Optimizer (`optimize_strategy.py`):
-Once installed, we can implement an optimization script that searches for:
-*   `rsi_period`: 7 to 21
-*   `hurst_multiplier`: 1.2 to 2.5
-*   `partial_exit_pct`: 0.05 to 0.30
+### How it works:
+When `--ma_period` is provided, the engine:
+1.  Loads the MA timeframe data from the DB.
+2.  Merges it with the 1m price data.
+3.  Switches modes (`long` <=> `short`) whenever the 1m price crosses the MA.
+4.  **Automatically closes** the existing position on a mode switch, just like the live bot.

-## 4. Local DB Data
-The engine connects to your local PostgreSQL DB using the credentials in your `.env` file. It specifically queries the `candles` table for the symbol and interval defined in `config/ping_pong_config.yaml`.
+## 4. Parameter Overrides
+You can quickly override strategy settings without editing the config file:
+*   `--direction`: Force a specific mode (`long` or `short`).
+*   `--limit`: Change the number of 1m candles to test (default 10,000).
+*   `--config`: Use a different configuration file.

 ## 5. Interpreting Results
 *   **Final Equity:** Your simulated account balance after all trades.
--- a/Hurst_simulations.md
+++ b/Hurst_simulations.md
@ -0,0 +1,180 @@
+# Comprehensive Hurst Strategy Simulations (since 2025-01-01)
+Comparison of different Hurst Timeframes and Entry Filters.
+
+# Timeframe: 1m
+
+## Scenario: Without Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $3192.86 | $2192.86 | 13239 | 7.40% |
+| 2025-04-01 to 2025-06-30 | $3192.86 | $5665.00 | $2472.14 | 13242 | 3.39% |
+| 2025-07-01 to 2025-09-30 | $5665.00 | $7520.61 | $1855.61 | 13720 | 1.02% |
+| 2025-10-01 to 2025-12-31 | $7520.61 | $8891.62 | $1371.01 | 13584 | 0.56% |
+| 2026-01-01 to 2026-03-10 | $8891.62 | $10437.95 | $1546.33 | 10120 | 0.52% |
+
+**Final Results for 1m (Without Filter):**
+- Final Equity: **$10437.95**
+- Total ROI: **943.80%**
+- Total Trades: **63905**
+- Max Overall Drawdown: **7.40%**
+
+## Scenario: With 1H SMA 200 Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $2124.83 | $1124.83 | 10469 | 7.57% |
+| 2025-04-01 to 2025-06-30 | $2124.83 | $3729.53 | $1604.70 | 11016 | 5.09% |
+| 2025-07-01 to 2025-09-30 | $3729.53 | $4673.10 | $943.57 | 10923 | 2.25% |
+| 2025-10-01 to 2025-12-31 | $4673.10 | $5428.01 | $754.91 | 11319 | 1.49% |
+| 2026-01-01 to 2026-03-10 | $5428.01 | $6307.05 | $879.04 | 8485 | 1.24% |
+
+**Final Results for 1m (With 1H SMA 200 Filter):**
+- Final Equity: **$6307.05**
+- Total ROI: **530.71%**
+- Total Trades: **52212**
+- Max Overall Drawdown: **7.57%**
+
+# Timeframe: 3m
+
+## Scenario: Without Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $2309.37 | $1309.37 | 4233 | 8.10% |
+| 2025-04-01 to 2025-06-30 | $2309.37 | $3801.67 | $1492.30 | 4331 | 2.07% |
+| 2025-07-01 to 2025-09-30 | $3801.67 | $4988.92 | $1187.25 | 4513 | 1.60% |
+| 2025-10-01 to 2025-12-31 | $4988.92 | $5912.62 | $923.70 | 4370 | 1.05% |
+| 2026-01-01 to 2026-03-10 | $5912.62 | $6740.90 | $828.28 | 3306 | 1.01% |
+
+**Final Results for 3m (Without Filter):**
+- Final Equity: **$6740.90**
+- Total ROI: **574.09%**
+- Total Trades: **20753**
+- Max Overall Drawdown: **8.10%**
+
+## Scenario: With 1H SMA 200 Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $1470.40 | $470.40 | 3288 | 10.19% |
+| 2025-04-01 to 2025-06-30 | $1470.40 | $2366.47 | $896.07 | 3597 | 4.96% |
+| 2025-07-01 to 2025-09-30 | $2366.47 | $2844.79 | $478.32 | 3478 | 3.23% |
+| 2025-10-01 to 2025-12-31 | $2844.79 | $3250.87 | $406.08 | 3597 | 3.43% |
+| 2026-01-01 to 2026-03-10 | $3250.87 | $3643.24 | $392.37 | 2739 | 1.90% |
+
+**Final Results for 3m (With 1H SMA 200 Filter):**
+- Final Equity: **$3643.24**
+- Total ROI: **264.32%**
+- Total Trades: **16699**
+- Max Overall Drawdown: **10.19%**
+
+# Timeframe: 5m
+
+## Scenario: Without Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $1878.15 | $878.15 | 2482 | 14.06% |
+| 2025-04-01 to 2025-06-30 | $1878.15 | $3053.49 | $1175.34 | 2601 | 3.26% |
+| 2025-07-01 to 2025-09-30 | $3053.49 | $3944.73 | $891.24 | 2689 | 2.08% |
+| 2025-10-01 to 2025-12-31 | $3944.73 | $4578.13 | $633.40 | 2491 | 1.33% |
+| 2026-01-01 to 2026-03-10 | $4578.13 | $5122.25 | $544.12 | 1966 | 1.39% |
+
+**Final Results for 5m (Without Filter):**
+- Final Equity: **$5122.25**
+- Total ROI: **412.23%**
+- Total Trades: **12229**
+- Max Overall Drawdown: **14.06%**
+
+## Scenario: With 1H SMA 200 Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $1304.55 | $304.55 | 1915 | 14.36% |
+| 2025-04-01 to 2025-06-30 | $1304.55 | $1926.70 | $622.15 | 2134 | 7.03% |
+| 2025-07-01 to 2025-09-30 | $1926.70 | $2253.45 | $326.75 | 2052 | 4.25% |
+| 2025-10-01 to 2025-12-31 | $2253.45 | $2539.78 | $286.33 | 2038 | 2.83% |
+| 2026-01-01 to 2026-03-10 | $2539.78 | $2774.98 | $235.20 | 1583 | 2.75% |
+
+**Final Results for 5m (With 1H SMA 200 Filter):**
+- Final Equity: **$2774.98**
+- Total ROI: **177.50%**
+- Total Trades: **9722**
+- Max Overall Drawdown: **14.36%**
+
+# Timeframe: 15m
+
+## Scenario: Without Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $1373.61 | $373.61 | 785 | 12.40% |
+| 2025-04-01 to 2025-06-30 | $1373.61 | $1824.22 | $450.61 | 725 | 6.62% |
+| 2025-07-01 to 2025-09-30 | $1824.22 | $2212.35 | $388.13 | 807 | 2.60% |
+| 2025-10-01 to 2025-12-31 | $2212.35 | $2535.13 | $322.78 | 765 | 3.14% |
+| 2026-01-01 to 2026-03-10 | $2535.13 | $2821.05 | $285.92 | 607 | 1.92% |
+
+**Final Results for 15m (Without Filter):**
+- Final Equity: **$2821.05**
+- Total ROI: **182.11%**
+- Total Trades: **3689**
+- Max Overall Drawdown: **12.40%**
+
+## Scenario: With 1H SMA 200 Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $993.20 | $-6.80 | 578 | 20.65% |
+| 2025-04-01 to 2025-06-30 | $993.20 | $1127.20 | $134.00 | 559 | 10.21% |
+| 2025-07-01 to 2025-09-30 | $1127.20 | $1328.46 | $201.26 | 595 | 2.96% |
+| 2025-10-01 to 2025-12-31 | $1328.46 | $1394.87 | $66.41 | 606 | 5.36% |
+| 2026-01-01 to 2026-03-10 | $1394.87 | $1379.42 | $-15.45 | 455 | 7.15% |
+
+**Final Results for 15m (With 1H SMA 200 Filter):**
+- Final Equity: **$1379.42**
+- Total ROI: **37.94%**
+- Total Trades: **2793**
+- Max Overall Drawdown: **20.65%**
+
+# Timeframe: 37m
+
+## Scenario: Without Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $1120.15 | $120.15 | 293 | 10.05% |
+| 2025-04-01 to 2025-06-30 | $1120.15 | $1487.65 | $367.50 | 282 | 6.98% |
+| 2025-07-01 to 2025-09-30 | $1487.65 | $1520.11 | $32.46 | 289 | 4.01% |
+| 2025-10-01 to 2025-12-31 | $1520.11 | $1575.24 | $55.13 | 292 | 9.73% |
+| 2026-01-01 to 2026-03-10 | $1575.24 | $1748.85 | $173.61 | 246 | 3.27% |
+
+**Final Results for 37m (Without Filter):**
+- Final Equity: **$1748.85**
+- Total ROI: **74.88%**
+- Total Trades: **1402**
+- Max Overall Drawdown: **10.05%**
+
+## Scenario: With 1H SMA 200 Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $1003.56 | $3.56 | 206 | 10.69% |
+| 2025-04-01 to 2025-06-30 | $1003.56 | $1130.86 | $127.30 | 208 | 8.98% |
+| 2025-07-01 to 2025-09-30 | $1130.86 | $1154.90 | $24.04 | 202 | 2.81% |
+| 2025-10-01 to 2025-12-31 | $1154.90 | $1108.85 | $-46.05 | 224 | 9.50% |
+| 2026-01-01 to 2026-03-10 | $1108.85 | $1117.52 | $8.67 | 179 | 4.01% |
+
+**Final Results for 37m (With 1H SMA 200 Filter):**
+- Final Equity: **$1117.52**
+- Total ROI: **11.75%**
+- Total Trades: **1019**
+- Max Overall Drawdown: **10.69%**
+
+# Timeframe: 1h
+
+## Scenario: Without Filter
+| Period | Start Bal | End Equity | PnL | Trades | Max DD |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| 2025-01-01 to 2025-03-31 | $1000.00 | $1109.99 | $109.99 | 162 | 7.36% |
+| 2025-04-01 to 2025-06-30 | $1109.99 | $1305.05 | $195.06 | 145 | 7.26% |
+| 2025-07-01 to 2025-09-30 | $1305.05 | $1342.48 | $37.43 | 180 | 2.83% |
+| 2025-10-01 to 2025-12-31 | $1342.48 | $1448.42 | $105.94 | 166 | 6.25% |
+| 2026-01-01 to 2026-03-10 | $1448.42 | $1552.57 | $104.15 | 135 | 4.56% |
+
+**Final Results for 1h (Without Filter):**
+- Final Equity: **$1552.57**
+- Total ROI: **55.26%**
+- Total Trades: **788**
+- Max Overall Drawdown: **7.36%**
+
--- a/config/ping_pong_config.yaml
+++ b/config/ping_pong_config.yaml
@ -34,6 +34,7 @@ min_position_value_usd: 15.0 # Minimum remaining value to keep position open

 # Execution Settings
 loop_interval_seconds: 5 # How often to check for new data
+execution_type: "maker" # "maker" (Limit Post-Only) or "taker" (Market)
 debug_mode: false

 # Robustness Settings
--- a/src/strategies/backtest_engine.py
+++ b/src/strategies/backtest_engine.py
@ -4,11 +4,17 @@ import yaml
 import os
 import asyncio
 import asyncpg
+import argparse
 from datetime import datetime
+from dotenv import load_dotenv
 from ping_pong_bot import PingPongStrategy

+# Load environment variables from .env
+load_dotenv()
+
 class BacktestEngine:
-    def __init__(self, config_path="config/ping_pong_config.yaml"):
+    def __init__(self, config_path="config/ping_pong_config.yaml", starting_equity=1000.0):
+        self.version = "1.7.11"
        with open(config_path, 'r') as f:
            self.config = yaml.safe_load(f)
        
@ -17,22 +23,34 @@ class BacktestEngine:
        self.strategy.direction = self.direction
        
        # Virtual Exchange State
-        self.balance = 1000.0  # Starting USD
-        self.equity = 1000.0
+        self.start_equity = starting_equity
+        self.balance = starting_equity
+        self.equity = starting_equity
        self.position_size = 0.0 # BTC
        self.position_value = 0.0 # USD
        self.entry_price = 0.0
        
+        # Performance Tracking
+        self.max_equity = starting_equity
+        self.max_drawdown = 0.0
+        
        # Settings
        self.fee_rate = 0.0005 # 0.05% Taker
-        self.leverage = float(self.config.get('leverage_long' if self.direction == 'long' else 'leverage_short', 5.0))
+        self.leverage = 5.0 # Will be updated based on mode
        self.pos_size_margin = float(self.config.get('pos_size_margin', 20.0))
        self.partial_exit_pct = float(self.config.get('partial_exit_pct', 0.15))
        
+        # Stop Loss Settings
+        self.stop_loss_pct = 0.0 # 0.0 = Disabled
+        self.stop_on_hurst_break = False
+        
+        # Safety Brake Settings
+        self.use_brake = False
+        
        self.trades = []
        self.equity_curve = []

-    async def load_data(self, symbol, interval, limit=5000):
+    async def load_data(self, symbol, interval, limit=None, start_date=None, end_date=None):
        conn = await asyncpg.connect(
            host=os.getenv('DB_HOST', '20.20.20.20'),
            port=int(os.getenv('DB_PORT', 5433)),
@ -40,64 +58,103 @@ class BacktestEngine:
            password=os.getenv('DB_PASSWORD', ''),
            database=os.getenv('DB_NAME', 'btc_data')
        )
-        rows = await conn.fetch('''
-            SELECT time, open, high, low, close, volume 
-            FROM candles 
-            WHERE symbol = $1 AND interval = $2 
-            ORDER BY time ASC LIMIT $3
-        ''', symbol, interval, limit)
+        
+        query = "SELECT time, open, high, low, close, volume FROM candles WHERE symbol = $1 AND interval = $2"
+        params = [symbol, interval]
+        
+        if start_date:
+            query += f" AND time >= ${len(params)+1}"
+            params.append(datetime.fromisoformat(start_date))
+        if end_date:
+            query += f" AND time <= ${len(params)+1}"
+            params.append(datetime.fromisoformat(end_date))
+            
+        query += " ORDER BY time ASC"
+        
+        if limit and not (start_date or end_date):
+            query += f" LIMIT ${len(params)+1}"
+            params.append(limit)
+            
+        rows = await conn.fetch(query, *params)
        await conn.close()
        
        df = pd.DataFrame([dict(r) for r in rows])
+        if df.empty: return df
        df[['open', 'high', 'low', 'close', 'volume']] = df[['open', 'high', 'low', 'close', 'volume']].astype(float)
        return df

-    def run(self, df):
+    def run(self, df, ma_df=None, ma_period=None, brake_df=None, brake_period=None):
+        if df.empty:
+            print("No data to run backtest.")
+            return
+            
        print(f"Starting backtest on {len(df)} candles...")
+        print(f"Period: {df.iloc[0]['time']} to {df.iloc[-1]['time']}")
+        
        df = self.strategy.calculate_indicators(df)
        
-        # Start after enough candles for indicators
-        start_idx = max(self.config['rsi']['period'], self.config['hurst']['period']) + 5
+        # Prepare MA for regime switching
+        ma_values = None
+        if ma_df is not None and ma_period:
+            ma_df['ma'] = ma_df['close'].rolling(window=ma_period).mean()
+            ma_subset = ma_df[['time', 'ma']].rename(columns={'time': 'ma_time'})
+            df = pd.merge_asof(df.sort_values('time'), ma_subset.sort_values('ma_time'), 
+                               left_on='time', right_on='ma_time', direction='backward')
+            ma_values = df['ma'].values
+            print(f"Regime Switching enabled (MA {ma_period})")
+
+        # Prepare Brake MA
+        brake_values = None
+        if brake_df is not None and brake_period:
+            brake_df['brake_ma'] = brake_df['close'].rolling(window=brake_period).mean()
+            brake_subset = brake_df[['time', 'brake_ma']].rename(columns={'time': 'brake_time'})
+            df = pd.merge_asof(df.sort_values('time'), brake_subset.sort_values('brake_time'), 
+                               left_on='time', right_on='brake_time', direction='backward')
+            brake_values = df['brake_ma'].values
+            self.use_brake = True
+            print(f"Safety Brake enabled (MA {brake_period})")
+
+        start_idx = max(self.config['rsi']['period'], self.config['hurst']['period'], 100)
+        if start_idx >= len(df):
+            print(f"Error: Not enough candles. Need {start_idx}, got {len(df)}")
+            return

        for i in range(start_idx, len(df)):
            current_df = df.iloc[:i+1]
            price = df.iloc[i]['close']
            time = df.iloc[i]['time']
            
+            # 1. Regime Check
+            if ma_values is not None and not np.isnan(ma_values[i]):
+                new_direction = "long" if price > ma_values[i] else "short"
+                if new_direction != self.direction:
+                    if abs(self.position_size) > 0:
+                        self.close_full_position(price, time, reason="Regime Switch")
+                    self.direction = new_direction
+                    self.strategy.direction = new_direction
+                    self.leverage = float(self.config.get('leverage_long' if self.direction == 'long' else 'leverage_short', 5.0))
+
+            # 2. Strategy Signal
            signal = self.strategy.check_signals(current_df)
            
+            # 3. Stop Loss Check
+            if abs(self.position_size) > 0:
+                is_stopped = self.check_stop_loss(price, time, df.iloc[i])
+                if is_stopped:
+                    signal = None 
+
            if signal == "open":
-                # Entry Logic
-                qty_usd = self.pos_size_margin * self.leverage
-                qty_btc = qty_usd / price
-                fee = qty_usd * self.fee_rate
-                
-                self.balance -= fee
-                
-                if self.direction == "long":
-                    self.position_size += qty_btc
-                else: # Short
-                    self.position_size -= qty_btc
-                
-                self.entry_price = price # Simplified avg entry
-                self.trades.append({"time": time, "type": "Enter", "price": price, "fee": fee})
+                # Apply Safety Brake
+                if self.use_brake and brake_values is not None and not np.isnan(brake_values[i]):
+                    if self.direction == "short" and price > brake_values[i]:
+                        signal = None # Brake: Don't short in uptrend
+                    elif self.direction == "long" and price < brake_values[i]:
+                        signal = None # Brake: Don't long in downtrend
                
+                if signal == "open":
+                    self.open_position(price, time)
            elif signal == "close" and abs(self.position_size) > 0:
-                # Exit Logic
-                qty_btc_exit = abs(self.position_size) * self.partial_exit_pct
-                qty_usd_exit = qty_btc_exit * price
-                fee = qty_usd_exit * self.fee_rate
-                
-                # Realized PnL
-                if self.direction == "long":
-                    pnl = qty_btc_exit * (price - self.entry_price)
-                    self.position_size -= qty_btc_exit
-                else: # Short
-                    pnl = qty_btc_exit * (self.entry_price - price)
-                    self.position_size += qty_btc_exit
-                
-                self.balance += (pnl - fee)
-                self.trades.append({"time": time, "type": "Exit", "price": price, "pnl": pnl, "fee": fee})
+                self.close_partial_position(price, time)

            # Mark to Market Equity
            unrealized = 0
@ -107,36 +164,129 @@ class BacktestEngine:
                unrealized = abs(self.position_size) * (self.entry_price - price) if self.position_size < 0 else 0
            
            self.equity = self.balance + unrealized
+            
+            # Max Drawdown Tracking
+            if self.equity > self.max_equity:
+                self.max_equity = self.equity
+            
+            dd = (self.max_equity - self.equity) / self.max_equity
+            if dd > self.max_drawdown:
+                self.max_drawdown = dd
+                
            self.equity_curve.append({"time": time, "equity": self.equity})

        self.print_results()

-    def print_results(self):
-        total_pnl = self.equity - 1000.0
-        roi = (total_pnl / 1000.0) * 100
-        fees = sum(t['fee'] for t in self.trades)
+    def open_position(self, price, time):
+        qty_usd = self.pos_size_margin * self.leverage
+        qty_btc = qty_usd / price
+        fee = qty_usd * self.fee_rate
+        self.balance -= fee
+        if self.direction == "long": self.position_size += qty_btc
+        else: self.position_size -= qty_btc
+        self.entry_price = price
+        self.trades.append({"time": time, "type": f"Enter {self.direction.upper()}", "price": price, "fee": fee})

+    def check_stop_loss(self, price, time, row):
+        """Returns True if Stop Loss was triggered"""
+        if self.stop_loss_pct > 0:
+            pnl_pct = (price - self.entry_price) / self.entry_price if self.direction == "long" else (self.entry_price - price) / self.entry_price
+            if pnl_pct <= -self.stop_loss_pct:
+                self.close_full_position(price, time, reason=f"Stop Loss ({self.stop_loss_pct*100}%)")
+                return True
+        if self.stop_on_hurst_break:
+            if self.direction == "long" and price < row['hurst_lower']:
+                self.close_full_position(price, time, reason="Stop Loss (Hurst Break)")
+                return True
+            if self.direction == "short" and price > row['hurst_upper']:
+                self.close_full_position(price, time, reason="Stop Loss (Hurst Break)")
+                return True
+        return False
+
+    def close_partial_position(self, price, time):
+        qty_btc_exit = abs(self.position_size) * self.partial_exit_pct
+        self._close_qty(qty_btc_exit, price, time, "Partial Exit")
+
+    def close_full_position(self, price, time, reason="Exit"):
+        self._close_qty(abs(self.position_size), price, time, reason)
+
+    def _close_qty(self, qty_btc_exit, price, time, reason):
+        qty_usd_exit = qty_btc_exit * price
+        fee = qty_usd_exit * self.fee_rate
+        if self.direction == "long":
+            pnl = qty_btc_exit * (price - self.entry_price)
+            self.position_size -= qty_btc_exit
+        else:
+            pnl = qty_btc_exit * (self.entry_price - price)
+            self.position_size += qty_btc_exit
+        self.balance += (pnl - fee)
+        self.trades.append({"time": time, "type": reason, "price": price, "pnl": pnl, "fee": fee})
+
+    def print_results(self):
+        total_pnl = self.equity - self.start_equity
+        roi = (total_pnl / self.start_equity) * 100
+        fees = sum(t['fee'] for t in self.trades)
+        sl_hits = len([t for t in self.trades if "Stop Loss" in t['type']])
        print("\n" + "="*30)
        print(" BACKTEST RESULTS ")
        print("="*30)
        print(f"Total Trades:  {len(self.trades)}")
+        print(f"Stop Loss Hits: {sl_hits}")
        print(f"Final Equity:  ${self.equity:.2f}")
        print(f"Total PnL:     ${total_pnl:.2f}")
        print(f"ROI:           {roi:.2f}%")
+        print(f"Max Drawdown:  {self.max_drawdown*100:.2f}%")
        print(f"Total Fees:    ${fees:.2f}")
        print("="*30)

 async def main():
-    engine = BacktestEngine()
-    # Assume BTC/1m for now
-    symbol = engine.config['symbol'].replace("USDT", "").replace("USD", "")
-    interval = engine.config['interval'] + "m"
+    parser = argparse.ArgumentParser(description='Ping-Pong Strategy Backtester')
+    parser.add_argument('--config', type=str, default='config/ping_pong_config.yaml')
+    parser.add_argument('--limit', type=int, default=10000)
+    parser.add_argument('--interval', type=str, help='Strategy Interval (e.g. 5m, 15m)')
+    parser.add_argument('--start_date', type=str)
+    parser.add_argument('--end_date', type=str)
+    parser.add_argument('--ma_period', type=int)
+    parser.add_argument('--ma_interval', type=str, default='1h')
+    parser.add_argument('--brake_period', type=int, help='Safety Brake MA Period')
+    parser.add_argument('--brake_interval', type=str, default='1h', help='Safety Brake MA Interval')
+    parser.add_argument('--direction', type=str, choices=['long', 'short'])
+    parser.add_argument('--stop_loss', type=float, default=0.0, help='Stop Loss % (e.g. 0.02 for 2%)')
+    parser.add_argument('--hurst_stop', action='store_true', help='Enable Stop Loss on Hurst break')
+    parser.add_argument('--maker_fee', type=float, help='Override fee rate for Maker simulation (e.g. 0.0002)')
+    parser.add_argument('--starting_equity', type=float, default=1000.0, help='Initial balance')
    
-    df = await engine.load_data(symbol, interval, limit=10000)
-    if not df.empty:
-        engine.run(df)
-    else:
-        print("No data found in DB.")
+    args = parser.parse_args()
+    engine = BacktestEngine(config_path=args.config, starting_equity=args.starting_equity)
+    
+    if args.maker_fee:
+        engine.fee_rate = args.maker_fee
+        print(f"Fee Rate overridden to: {args.maker_fee} (Maker Simulation)")
+    
+    engine.stop_loss_pct = args.stop_loss
+    engine.stop_on_hurst_break = args.hurst_stop
+    
+    # Base Data
+    symbol = engine.config['symbol'].replace("USDT", "").replace("USD", "")
+    data_interval = args.interval if args.interval else engine.config['interval']
+    if data_interval.isdigit(): data_interval += "m"
+    
+    df = await engine.load_data(symbol, data_interval, limit=args.limit, start_date=args.start_date, end_date=args.end_date)
+    if df.empty: return
+
+    ma_df = None
+    if args.ma_period:
+        ma_df = await engine.load_data(symbol, args.ma_interval, limit=5000, start_date=None, end_date=args.end_date)
+
+    brake_df = None
+    if args.brake_period:
+        brake_df = await engine.load_data(symbol, args.brake_interval, limit=5000, start_date=None, end_date=args.end_date)
+
+    if args.direction:
+        engine.direction = args.direction
+        engine.strategy.direction = args.direction
+    
+    engine.run(df, ma_df=ma_df, ma_period=args.ma_period, brake_df=brake_df, brake_period=args.brake_period)

 if __name__ == "__main__":
    asyncio.run(main())
--- a/src/strategies/ping_pong_bot.py
+++ b/src/strategies/ping_pong_bot.py
@ -5,6 +5,7 @@ import hmac
 import hashlib
 import json
 import logging
+from logging.handlers import RotatingFileHandler
 import asyncio
 import pandas as pd
 import numpy as np
@ -30,14 +31,28 @@ load_dotenv()
 log_level = os.getenv("LOG_LEVEL", "INFO")

 # Setup Logging
-logging.basicConfig(
-    level=getattr(logging, log_level),
-    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
-    handlers=[
-        logging.StreamHandler()
-    ]
-)
+log_dir = "logs"
+os.makedirs(log_dir, exist_ok=True)
+error_log_path = os.path.join(log_dir, "ping_pong_errors.log")
+
+# Create logger
 logger = logging.getLogger("PingPongBot")
+logger.setLevel(logging.DEBUG) # Catch everything, handlers will filter
+
+# Formatter
+formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+
+# Console Handler (Normal logs)
+ch = logging.StreamHandler()
+ch.setLevel(getattr(logging, log_level))
+ch.setFormatter(formatter)
+logger.addHandler(ch)
+
+# Error File Handler (Warnings and Errors only)
+fh = RotatingFileHandler(error_log_path, maxBytes=5*1024*1024, backupCount=2)
+fh.setLevel(logging.WARNING)
+fh.setFormatter(formatter)
+logger.addHandler(fh)

 class DatabaseManager:
    """Minimal Database Manager for the bot"""
@ -165,7 +180,7 @@ class PingPongStrategy:

 class PingPongBot:
    def __init__(self, config_path="config/ping_pong_config.yaml"):
-        self.version = "1.7.5"
+        self.version = "1.8.6"
        with open(config_path, 'r') as f:
            self.config = yaml.safe_load(f)
        
@ -249,36 +264,37 @@ class PingPongBot:
        self.leverage_short = float(self.config.get('leverage_short', 3.0))
        self.leverage = 1.0 # Current leverage
        self.max_eff_lev = float(self.config.get('max_effective_leverage', 1.0))
+        self.exec_type = self.config.get('execution_type', 'taker').lower()

    def _init_tx_log(self):
        """Ensures CSV header exists and is up to date"""
-        header = "time,version,direction,symbol,trade,qty,price,leverage,pnl,fee,status\n"
+        header = "time,version,direction,symbol,trade,qty,price,leverage,pnl,fee,attempts,status\n"
        if not os.path.exists(self.tx_log_path):
            os.makedirs(os.path.dirname(self.tx_log_path), exist_ok=True)
            with open(self.tx_log_path, 'w') as f:
                f.write(header)
        else:
-            # Check if we need to update the header from 'side' to 'trade'
+            # Check if we need to update the header
            try:
                with open(self.tx_log_path, 'r') as f:
                    first_line = f.readline()
-                if "side" in first_line:
+                if "attempts" not in first_line:
                    with open(self.tx_log_path, 'r') as f:
                        lines = f.readlines()
                    if lines:
                        lines[0] = header
                        with open(self.tx_log_path, 'w') as f:
                            f.writelines(lines)
-                        logger.info("Updated CSV log header: 'side' -> 'trade'")
+                        logger.info("Updated CSV log header: Added 'attempts' column")
            except Exception as e:
                logger.error(f"Failed to update CSV header: {e}")

-    async def log_transaction(self, trade, qty, price, pnl=0, fee=0, status="Success"):
+    async def log_transaction(self, trade, qty, price, pnl=0, fee=0, attempts=1, status="Success"):
        """Appends a trade record to CSV"""
        try:
            with open(self.tx_log_path, 'a') as f:
                t_str = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
-                f.write(f"{t_str},{self.version},{self.direction},{self.symbol},{trade},{qty},{price},{self.leverage},{pnl},{fee},{status}\n")
+                f.write(f"{t_str},{self.version},{self.direction},{self.symbol},{trade},{qty},{price},{self.leverage},{pnl},{fee},{attempts},{status}\n")
        except Exception as e:
            logger.error(f"Failed to write to CSV log: {e}")

@ -360,14 +376,22 @@ class PingPongBot:
                buyLeverage=str(self.leverage),
                sellLeverage=str(self.leverage)
            )
+            
+            # If pybit returns normally, check the retCode
            if res['retCode'] == 0:
                logger.info(f"Leverage successfully set to {self.leverage}x")
            elif res['retCode'] == 110043: # Leverage not modified
                logger.info(f"Leverage is already {self.leverage}x")
            else:
                logger.warning(f"Bybit Leverage Warning: {res['retMsg']} (Code: {res['retCode']})")
+                
        except Exception as e:
-            logger.error(f"Failed to set leverage on Bybit: {e}")
+            # Check if exception contains "leverage not modified" or code 110043
+            err_str = str(e)
+            if "110043" in err_str or "leverage not modified" in err_str.lower():
+                logger.info(f"Leverage is already correctly set ({self.leverage}x)")
+            else:
+                logger.error(f"Failed to set leverage on Bybit: {e}")

    async def close_all_positions(self):
        """Closes any active position in the current category/symbol"""
@ -488,40 +512,130 @@ class PingPongBot:
        if not self.category or not self.symbol: return
        side = "Sell" if (self.direction == "long" and is_close) or (self.direction == "short" and not is_close) else "Buy"
        trade = "Exit" if is_close else "Enter"
-        
-        # Using positionIdx=0 for One-Way Mode to avoid Error 10001
        pos_idx = 0
-        try:
-            qty_str = str(int(qty)) if self.category == "inverse" else str(round(qty, 3))
+        qty_str = str(int(qty)) if self.category == "inverse" else str(round(qty, 3))
+
+        if self.exec_type != "maker":
+            try:
+                res = await asyncio.to_thread(self.session.place_order,
+                    category=self.category, symbol=self.symbol, side=side, orderType="Market",
+                    qty=qty_str, reduceOnly=is_close, positionIdx=pos_idx
+                )
+                if res['retCode'] == 0:
+                    await self._process_filled_order(res['result']['orderId'], trade, qty_str, attempts=1)
+                else:
+                    self.status_msg = f"Order Error: {res['retMsg']}"
+            except Exception as e:
+                logger.error(f"Taker Trade Error: {e}")
+            return
+
+        # Maker Chase Logic (Max 5 tries)
+        max_retries = 5
+        for attempt in range(1, max_retries + 1):
+            try:
+                # Fresh Bid/Ask for Mid-Price Limit order
+                ticker = await asyncio.to_thread(self.session.get_tickers, category=self.category, symbol=self.symbol)
+                if ticker['retCode'] == 0 and ticker['result']['list']:
+                    t = ticker['result']['list'][0]
+                    bid = float(t.get('bid1Price', 0))
+                    ask = float(t.get('ask1Price', 0))
+                    last = float(t.get('lastPrice', 0))
+                    
+                    if bid > 0 and ask > 0:
+                        self.market_price = (bid + ask) / 2
+                    else:
+                        self.market_price = last
+                
+                price_str = str(round(self.market_price, 1))
+                self.status_msg = f"Chase {trade}: {attempt}/{max_retries} @ {price_str} (Mid)"
+                
+                res = await asyncio.to_thread(self.session.place_order,
+                    category=self.category, symbol=self.symbol, side=side, orderType="Limit",
+                    qty=qty_str, price=price_str, timeInForce="PostOnly",
+                    reduceOnly=is_close, positionIdx=pos_idx
+                )
+                
+                if res['retCode'] != 0:
+                    # Specific check for race condition: order filled while trying to place/cancel
+                    if res['retCode'] in [110001, 170213, 170210]:
+                        # Check if actually filled
+                        history = await asyncio.to_thread(self.session.get_order_history, 
+                                                        category=self.category, symbol=self.symbol, limit=1)
+                        if history['retCode'] == 0 and history['result']['list']:
+                            latest = history['result']['list'][0]
+                            if latest['orderStatus'] == "Filled" and float(latest['cumExecQty']) > 0:
+                                await self._process_filled_order(latest['orderId'], trade, qty_str, attempts=attempt)
+                                return
+                    
+                    logger.warning(f"Maker rejected (Try {attempt}): {res['retMsg']}")
+                    await asyncio.sleep(2)
+                    continue

-            res = await asyncio.to_thread(self.session.place_order,
-                category=self.category, symbol=self.symbol, side=side, orderType="Market",
-                qty=qty_str, reduceOnly=is_close, positionIdx=pos_idx
-            )
-            if res['retCode'] == 0:
                order_id = res['result']['orderId']
-                self.last_signal = f"{trade} {qty_str}"
-                self.status_msg = f"Order Success: {trade}"
                
-                # Fetch execution details for fees and PnL
-                await asyncio.sleep(1.5) # Wait for fill and indexing
+                # Monitor for fill (Wait 10 seconds)
+                for _ in range(10):
+                    await asyncio.sleep(1)
+                    # Check order history for definitive status
+                    history = await asyncio.to_thread(self.session.get_order_history, 
+                                                    category=self.category, symbol=self.symbol, orderId=order_id)
+                    if history['retCode'] == 0 and history['result']['list']:
+                        status = history['result']['list'][0]['orderStatus']
+                        if status == "Filled":
+                            await self._process_filled_order(order_id, trade, qty_str, attempts=attempt)
+                            return
+                        elif status in ["Cancelled", "Rejected", "Deactivated"]:
+                            break # Go to retry
+                
+                # Timeout: Cancel and retry
+                try:
+                    cancel_res = await asyncio.to_thread(self.session.cancel_order, category=self.category, symbol=self.symbol, orderId=order_id)
+                    # Even if successful, double check if it filled in the last millisecond
+                    if cancel_res['retCode'] in [0, 110001, 170213]:
+                        history = await asyncio.to_thread(self.session.get_order_history, 
+                                                        category=self.category, symbol=self.symbol, orderId=order_id)
+                        if history['retCode'] == 0 and history['result']['list'] and history['result']['list'][0]['orderStatus'] == "Filled":
+                            await self._process_filled_order(order_id, trade, qty_str, attempts=attempt)
+                            return
+                except Exception as ce:
+                    # Handle exception for 110001
+                    if "110001" in str(ce) or "170213" in str(ce):
+                        history = await asyncio.to_thread(self.session.get_order_history, 
+                                                        category=self.category, symbol=self.symbol, orderId=order_id)
+                        if history['retCode'] == 0 and history['result']['list'] and history['result']['list'][0]['orderStatus'] == "Filled":
+                            await self._process_filled_order(order_id, trade, qty_str, attempts=attempt)
+                            return
+                    logger.warning(f"Cancel error during chase: {ce}")
+                
+                logger.info(f"Maker {trade} timed out, retrying ({attempt}/{max_retries})")
+                
+            except Exception as e:
+                logger.error(f"Maker Chase Error (Try {attempt}): {e}")
+                await asyncio.sleep(2)
+
+        self.status_msg = f"{trade} failed after {max_retries} chase attempts"
+        await self.log_transaction(trade, qty_str, self.market_price, attempts=max_retries, status="Failed (Chase Timeout)")
+
+    async def _process_filled_order(self, order_id, trade, qty_str, attempts=1):
+        """Finalizes a successful trade by logging fees and PnL"""
+        self.last_signal = f"{trade} {qty_str}"
+        self.status_msg = f"Order Success: {trade} ({self.exec_type})"
+        
+        # Wait for Bybit indexing (multiple attempts if needed)
+        for _ in range(3):
+            await asyncio.sleep(1.5)
+            try:
                exec_info = await asyncio.to_thread(self.session.get_executions, 
                                                    category=self.category, 
                                                    symbol=self.symbol, 
                                                    orderId=order_id)
                
-                exec_fee = 0.0
-                exec_pnl = 0.0
-                exec_price = self.market_price
-                
                if exec_info['retCode'] == 0 and exec_info['result']['list']:
                    fills = exec_info['result']['list']
-                    # Fees and closedPnl are in settleCoin (BTC for inverse, USDC for linear)
                    exec_fee = sum(float(f.get('execFee', 0)) for f in fills)
                    exec_pnl = sum(float(f.get('closedPnl', 0)) for f in fills)
                    exec_price = float(fills[0].get('execPrice', self.market_price))
                    
-                    # Convert to USD if in BTC for consistent tracking
                    if self.category == "inverse":
                        usd_fee = exec_fee * exec_price
                        usd_pnl = exec_pnl * exec_price
@ -531,16 +645,13 @@ class PingPongBot:
                        
                    self.total_fees += usd_fee
                    self.total_realized_pnl += usd_pnl
+                    await self.log_transaction(trade, qty_str, exec_price, pnl=usd_pnl, fee=usd_fee, attempts=attempts, status="Filled")
+                    return
+            except Exception as e:
+                logger.error(f"Execution fetch error: {e}")
        
-                    await self.log_transaction(trade, qty_str, exec_price, pnl=usd_pnl, fee=usd_fee, status="Filled")
-                else:
-                    await self.log_transaction(trade, qty_str, self.market_price, status="Filled (No Exec Info)")
-            else:
-                self.status_msg = f"Order Error: {res['retMsg']}"
-                logger.error(f"Bybit Order Error: {res['retMsg']} (Code: {res['retCode']})")
-                await self.log_transaction(trade, qty_str, self.market_price, status=f"Error: {res['retMsg']}")
-        except Exception as e:
-            logger.error(f"Trade Error: {e}")
+        # Fallback if execution list is still empty after retries
+        await self.log_transaction(trade, qty_str, self.market_price, attempts=attempts, status=f"Filled ({self.exec_type})")

    def render_dashboard(self):
        self.console.print("\n" + "="*60)
Author	SHA1	Message	Date
Gemini CLI	b462651f60	test: add comprehensive Hurst strategy simulations and backtest engine enhancements (v1.7.11) - Add Hurst_simulations.md with detailed performance data from Jan 2025 to Mar 2026 across 1m-1h TFs. - Enhance backtest_engine.py: - Add --starting_equity for memory-efficient chunked testing. - Add Max Drawdown tracking. - Add --interval override flag. - Add --brake_period/--brake_interval for safety filter testing.	2026-03-10 13:02:29 +01:00
Gemini CLI	cd66a976de	feat: implement Mid-Price logic and separate error logging (v1.8.6) - Implement Mid-Price calculation (Bid+Ask)/2 for Maker orders to improve fill rates. - Add separate 'logs/ping_pong_errors.log' for WARNING and ERROR messages. - Add RotatingFileHandler for error logs (5MB cap, 2 backups). - Refine 'leverage not modified' (110043) handling from ERROR to INFO. - Improve order verification with explicit status checks and race condition handling. - Verify script syntax and update version to 1.8.6.	2026-03-08 22:07:11 +01:00
Gemini CLI	2840d9b0b3	feat: implement Maker Chase logic and enhanced order logging (v1.8.2) - Implement 5-try Chase logic for Maker (Limit Post-Only) orders. - Add 'attempts' column to CSV transaction log for performance tracking. - Update backtest engine (v1.7.9) with Stop Loss and Maker fee simulation. - Log failed chase sequences explicitly as "Failed (Chase Timeout)". - Consolidate order processing into internal helper methods.	2026-03-08 20:39:17 +01:00
Gemini CLI	f3b186b01d	feat: add local PC support and setup instructions for backtesting (v1.7.8)	2026-03-08 20:12:16 +01:00