btc-trading/docs/synology_db_connection_guide.md

Guide: Connecting to Synology PostgreSQL from Local PC

This guide explains how to connect to your Synology's TimescaleDB from your local PC to work with historical 1m candle data.

Prerequisites

1. Install PostgreSQL Client on Your PC

Windows:

# Download from: https://www.postgresql.org/download/windows/
# Or use chocolatey:
choco install postgresql

Mac:

brew install postgresql
# Or download Postgres.app from postgresapp.com

Linux:

# Ubuntu/Debian
sudo apt-get install postgresql-client

# Or use Docker:
docker run -it --rm postgres:15 psql --version

2. Install Python Dependencies

pip install asyncpg pandas numpy
# Or use requirements.txt from the project
pip install -r requirements.txt

Step 1: Configure Synology for Remote Access

Open PostgreSQL Port

1. SSH into your Synology:

   ssh admin@YOUR_SYNOLOGY_IP
   

2. Edit PostgreSQL configuration:

   # Find postgresql.conf (usually in /var/lib/postgresql/data/)
   sudo vim /var/lib/postgresql/data/postgresql.conf
   
   # Change:
   # listen_addresses = 'localhost'
   # To:
   listen_addresses = '*'
   

3. Edit pg_hba.conf to allow remote connections:

   sudo vim /var/lib/postgresql/data/pg_hba.conf
   
   # Add at the end:
   host    all             all             0.0.0.0/0               md5
   # Or for specific IP:
   host    all             all             YOUR_PC_IP/32           md5
   

4. Restart PostgreSQL:

   sudo systemctl restart postgresql
   # Or if using Docker:
   cd ~/btc_bot/docker && docker-compose restart timescaledb
   

Configure Synology Firewall

1. Open Control Panel → Security → Firewall
2. Click Edit Rules
3. Create new rule:
   • Ports: Custom → TCP → 5433 (or your PostgreSQL port)
   • Source IP: Your PC's IP address (or allow all)
   • Action: Allow
4. Apply the rule

Step 2: Test Connection

Using psql CLI

# Replace with your Synology IP
export DB_HOST=192.168.1.100  # Your Synology IP
export DB_PORT=5433
export DB_NAME=btc_data
export DB_USER=btc_bot
export DB_PASSWORD=your_password

# Test connection
psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -c "SELECT version();"

Using Python

Create test_connection.py:

import asyncio
import asyncpg

async def test():
    conn = await asyncpg.connect(
        host='192.168.1.100',      # Your Synology IP
        port=5433,
        database='btc_data',
        user='btc_bot',
        password='your_password'
    )
    
    version = await conn.fetchval('SELECT version()')
    print(f"Connected! PostgreSQL version: {version}")
    
    # Test candle count
    count = await conn.fetchval(
        "SELECT COUNT(*) FROM candles WHERE interval = '1m'"
    )
    print(f"Total 1m candles: {count:,}")
    
    await conn.close()

asyncio.run(test())

Run it:

python test_connection.py

Step 3: Export Historical 1m Data

Option A: Using psql (Quick Export)

psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -c "
COPY (
    SELECT time, symbol, interval, open, high, low, close, volume 
    FROM candles 
    WHERE symbol = 'BTC' 
    AND interval = '1m' 
    AND time >= '2025-01-01'
    ORDER BY time
) TO STDOUT WITH CSV HEADER;
" > btc_1m_candles.csv

Option B: Using Python (With Progress Bar)

Create export_candles.py:

import asyncio
import asyncpg
import csv
from datetime import datetime, timezone
from tqdm import tqdm

async def export_candles(
    host: str,
    port: int,
    database: str,
    user: str,
    password: str,
    symbol: str = 'BTC',
    interval: str = '1m',
    start_date: str = '2025-01-01',
    output_file: str = 'candles.csv'
):
    """Export candles to CSV with progress bar"""
    
    conn = await asyncpg.connect(
        host=host, port=port, database=database,
        user=user, password=password
    )
    
    try:
        # Get total count
        total = await conn.fetchval("""
            SELECT COUNT(*) FROM candles 
            WHERE symbol = $1 AND interval = $2 AND time >= $3
        """, symbol, interval, datetime.fromisoformat(start_date).replace(tzinfo=timezone.utc))
        
        print(f"Exporting {total:,} candles...")
        
        # Export in batches
        with open(output_file, 'w', newline='') as f:
            writer = csv.writer(f)
            writer.writerow(['time', 'symbol', 'interval', 'open', 'high', 'low', 'close', 'volume'])
            
            async with conn.transaction():
                cursor = await conn.cursor(
                    "SELECT time, symbol, interval, open, high, low, close, volume "
                    "FROM candles WHERE symbol = $1 AND interval = $2 AND time >= $3 "
                    "ORDER BY time",
                    symbol, interval, datetime.fromisoformat(start_date).replace(tzinfo=timezone.utc)
                )
                
                with tqdm(total=total, unit='candles') as pbar:
                    while True:
                        rows = await cursor.fetch(1000)
                        if not rows:
                            break
                        for row in rows:
                            writer.writerow(row)
                        pbar.update(len(rows))
        
        print(f"✓ Exported to {output_file}")
        print(f"  Total rows: {total:,}")
        
    finally:
        await conn.close()

if __name__ == "__main__":
    asyncio.run(export_candles(
        host='192.168.1.100',      # Change to your Synology IP
        port=5433,
        database='btc_data',
        user='btc_bot',
        password='your_password',
        symbol='BTC',
        interval='1m',
        start_date='2025-01-01',
        output_file='btc_1m_candles.csv'
    ))

Run:

pip install tqdm
python export_candles.py

Step 4: Calculate Indicators on PC

Using pandas-ta (Recommended)

Create calculate_indicators.py:

import pandas as pd
import pandas_ta as ta
from datetime import datetime

def calculate_indicators(input_file: str, output_file: str):
    """Calculate technical indicators from candle data"""
    
    # Read candles
    print(f"Reading {input_file}...")
    df = pd.read_csv(input_file)
    df['time'] = pd.to_datetime(df['time'])
    df = df.sort_values('time')
    
    print(f"Loaded {len(df):,} candles")
    
    # Calculate indicators
    print("Calculating indicators...")
    
    # Moving Averages
    df['ma44'] = ta.sma(df['close'], length=44)
    df['ma125'] = ta.sma(df['close'], length=125)
    df['ma200'] = ta.sma(df['close'], length=200)
    
    # RSI
    df['rsi'] = ta.rsi(df['close'], length=14)
    
    # Bollinger Bands
    bb = ta.bbands(df['close'], length=20, std=2)
    df['bb_upper'] = bb['BBU_20_2.0']
    df['bb_middle'] = bb['BBM_20_2.0']
    df['bb_lower'] = bb['BBL_20_2.0']
    
    # MACD
    macd = ta.macd(df['close'], fast=12, slow=26, signal=9)
    df['macd'] = macd['MACD_12_26_9']
    df['macd_signal'] = macd['MACDs_12_26_9']
    df['macd_histogram'] = macd['MACDh_12_26_9']
    
    # Save
    print(f"Saving to {output_file}...")
    df.to_csv(output_file, index=False)
    
    print(f"✓ Calculated {len(df.columns) - 8} indicators")
    print(f"  Output: {output_file}")
    
    return df

if __name__ == "__main__":
    df = calculate_indicators(
        input_file='btc_1m_candles.csv',
        output_file='btc_1m_with_indicators.csv'
    )
    
    # Show sample
    print("\nSample data:")
    print(df[['time', 'close', 'ma44', 'ma125', 'rsi']].tail(10))

Install dependencies:

pip install pandas pandas-ta

Run:

python calculate_indicators.py

Performance Tips

• Chunk processing for very large files (> 1GB):

# Process 100k rows at a time
chunksize = 100000
for chunk in pd.read_csv(input_file, chunksize=chunksize):
    # Calculate indicators for chunk
    pass

• Use multiple cores:

from multiprocessing import Pool
# Parallelize indicator calculation

Step 5: Import Indicators Back to Synology

Option A: Direct SQL Insert (Fastest)

Create import_indicators.py:

import asyncio
import asyncpg
import pandas as pd
from datetime import datetime
from tqdm import tqdm

async def import_indicators(
    host: str,
    port: int,
    database: str,
    user: str,
    password: str,
    input_file: str,
    batch_size: int = 1000
):
    """Import calculated indicators to Synology database"""
    
    # Read calculated indicators
    print(f"Reading {input_file}...")
    df = pd.read_csv(input_file)
    print(f"Loaded {len(df):,} rows with {len(df.columns)} columns")
    
    # Connect to database
    conn = await asyncpg.connect(
        host=host, port=port, database=database,
        user=user, password=password
    )
    
    try:
        # Get indicator columns (exclude candle data)
        indicator_cols = [c for c in df.columns if c not in 
                         ['time', 'symbol', 'interval', 'open', 'high', 'low', 'close', 'volume']]
        
        print(f"Importing {len(indicator_cols)} indicators: {indicator_cols}")
        
        # Prepare data
        symbol = df['symbol'].iloc[0]
        interval = df['interval'].iloc[0]
        
        total_inserted = 0
        
        with tqdm(total=len(df) * len(indicator_cols), unit='indicators') as pbar:
            for col in indicator_cols:
                # Prepare batch
                values = []
                for _, row in df.iterrows():
                    if pd.notna(row[col]):  # Skip NaN values
                        values.append((
                            row['time'],
                            symbol,
                            interval,
                            col,  # indicator_name
                            float(row[col]),
                            {}  # parameters (JSONB)
                        ))
                
                # Insert in batches
                for i in range(0, len(values), batch_size):
                    batch = values[i:i + batch_size]
                    await conn.executemany(
                        """
                        INSERT INTO indicators (time, symbol, interval, indicator_name, value, parameters)
                        VALUES ($1, $2, $3, $4, $5, $6)
                        ON CONFLICT (time, symbol, interval, indicator_name) DO UPDATE SET
                            value = EXCLUDED.value,
                            updated_at = NOW()
                        """,
                        batch
                    )
                    total_inserted += len(batch)
                    pbar.update(len(batch))
        
        print(f"\n✓ Imported {total_inserted:,} indicator values")
        
    finally:
        await conn.close()

if __name__ == "__main__":
    asyncio.run(import_indicators(
        host='192.168.1.100',
        port=5433,
        database='btc_data',
        user='btc_bot',
        password='your_password',
        input_file='btc_1m_with_indicators.csv',
        batch_size=1000
    ))

Run:

python import_indicators.py

Option B: Using psql COPY (For Large Files)

# Convert to format for COPY command
python -c "
import pandas as pd
df = pd.read_csv('btc_1m_with_indicators.csv')
# Transform to long format for indicators table
indicators = []
for col in ['ma44', 'ma125', 'rsi', 'bb_upper', 'bb_lower']:
    temp = df[['time', 'symbol', 'interval', col]].copy()
    temp['indicator_name'] = col
    temp = temp.rename(columns={col: 'value'})
    indicators.append(temp)
result = pd.concat(indicators)
result = result[result['value'].notna()]
result.to_csv('indicators_for_import.csv', index=False)
"

# Upload and import on Synology
psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -c "
COPY indicators (time, symbol, interval, indicator_name, value)
FROM '/path/to/indicators_for_import.csv' 
DELIMITER ',' CSV HEADER;
"

Complete Workflow Script

Create sync_indicators.py for one-command workflow:

#!/usr/bin/env python3
"""
Complete workflow: Export candles → Calculate indicators → Import to Synology
"""

import asyncio
import asyncpg
import pandas as pd
import pandas_ta as ta
from datetime import datetime, timezone
from tqdm import tqdm

class IndicatorSync:
    def __init__(self, host: str, port: int, database: str, user: str, password: str):
        self.db_config = {
            'host': host, 'port': port, 'database': database,
            'user': user, 'password': password
        }
    
    async def export_and_calculate(
        self,
        symbol: str = 'BTC',
        interval: str = '1m',
        start_date: str = '2025-01-01',
        indicators_config: dict = None
    ):
        """Main workflow"""
        
        print("="*70)
        print(f"INDICATOR SYNC: {symbol}/{interval}")
        print(f"Period: {start_date} to now")
        print("="*70)
        
        # Connect to database
        conn = await asyncpg.connect(**self.db_config)
        
        try:
            # Step 1: Export candles
            print("\n📥 Step 1: Exporting candles...")
            candles = await self._export_candles(conn, symbol, interval, start_date)
            print(f"   Exported {len(candles):,} candles")
            
            # Step 2: Calculate indicators
            print("\n⚙️  Step 2: Calculating indicators...")
            indicators_df = self._calculate_indicators(candles, indicators_config)
            print(f"   Calculated {len(indicators_df)} indicator values")
            
            # Step 3: Import indicators
            print("\n📤 Step 3: Importing to database...")
            await self._import_indicators(conn, indicators_df)
            print("   Import complete!")
            
            print("\n" + "="*70)
            print("✅ SYNC COMPLETE")
            print("="*70)
            
        finally:
            await conn.close()
    
    async def _export_candles(self, conn, symbol, interval, start_date):
        """Export candles from database"""
        rows = await conn.fetch(
            "SELECT time, symbol, interval, open, high, low, close, volume "
            "FROM candles WHERE symbol = $1 AND interval = $2 AND time >= $3 "
            "ORDER BY time",
            symbol, interval, datetime.fromisoformat(start_date).replace(tzinfo=timezone.utc)
        )
        return pd.DataFrame(rows, columns=['time', 'symbol', 'interval', 'open', 'high', 'low', 'close', 'volume'])
    
    def _calculate_indicators(self, candles_df, config=None):
        """Calculate technical indicators"""
        df = candles_df.copy()
        
        # Default indicators
        config = config or {
            'ma44': lambda d: ta.sma(d['close'], length=44),
            'ma125': lambda d: ta.sma(d['close'], length=125),
            'rsi': lambda d: ta.rsi(d['close'], length=14),
        }
        
        # Calculate each indicator
        for name, func in config.items():
            df[name] = func(df)
        
        # Transform to long format
        indicators = []
        indicator_cols = list(config.keys())
        
        for col in indicator_cols:
            temp = df[['time', 'symbol', 'interval', col]].copy()
            temp['indicator_name'] = col
            temp = temp.rename(columns={col: 'value'})
            temp = temp[temp['value'].notna()]
            indicators.append(temp)
        
        return pd.concat(indicators, ignore_index=True)
    
    async def _import_indicators(self, conn, indicators_df):
        """Import indicators to database"""
        # Convert to list of tuples
        values = [
            (row['time'], row['symbol'], row['interval'], row['indicator_name'], float(row['value']), {})
            for _, row in indicators_df.iterrows()
        ]
        
        # Insert in batches
        batch_size = 1000
        for i in tqdm(range(0, len(values), batch_size), desc="Importing"):
            batch = values[i:i + batch_size]
            await conn.executemany(
                """
                INSERT INTO indicators (time, symbol, interval, indicator_name, value, parameters)
                VALUES ($1, $2, $3, $4, $5, $6)
                ON CONFLICT (time, symbol, interval, indicator_name) DO UPDATE SET
                    value = EXCLUDED.value,
                    updated_at = NOW()
                """,
                batch
            )

# Usage
if __name__ == "__main__":
    sync = IndicatorSync(
        host='192.168.1.100',      # Your Synology IP
        port=5433,
        database='btc_data',
        user='btc_bot',
        password='your_password'
    )
    
    asyncio.run(sync.export_and_calculate(
        symbol='BTC',
        interval='1m',
        start_date='2025-01-01'
    ))

Run complete workflow:

python sync_indicators.py

Troubleshooting

Connection Refused

Error: connection refused

• Check if PostgreSQL is running: docker ps | grep timescale
• Verify port is open: telnet SYNOLOGY_IP 5433
• Check firewall rules on Synology

Permission Denied

Error: password authentication failed

• Verify password is correct
• Check pg_hba.conf has proper entries
• Restart PostgreSQL after config changes

Slow Performance

• Use batch inserts (1000 rows at a time)
• Process large datasets in chunks
• Consider using COPY command for very large imports
• Close cursor after use

Memory Issues

• Don't load entire table into memory
• Use server-side cursors
• Process in smaller date ranges
• Use chunksize when reading CSV

Security Notes

⚠️ Important:

• Use strong password for database user
• Limit pg_hba.conf to specific IPs when possible
• Use VPN if accessing over internet
• Consider SSL connection for remote access
• Don't commit passwords to git

Next Steps

Once indicators are calculated and imported:

1. Backtests will be fast - Server just reads pre-calculated values
2. Dashboard will load quickly - No on-the-fly calculation needed
3. Can add more indicators - Just re-run sync with new calculations

Alternative: SSH Tunnel (More Secure)

Instead of opening PostgreSQL port, use SSH tunnel:

# On your PC
ssh -L 5433:localhost:5433 admin@SYNOLOGY_IP

# Now connect to localhost:5433 (tunnels to Synology)
export DB_HOST=localhost
export DB_PORT=5433
python sync_indicators.py

This encrypts all traffic and doesn't require opening PostgreSQL port.

---

Questions or issues? Check the logs and verify each step works before proceeding to the next.