~kris/dots

srice

ref: e98f3b030dc24445bd55c68d95d2d81933fd68b3 srice/doc/trading/research-infrastructure.md -rw-r--r-- 9.8 KiB
e98f3b03 — Kris Yotam chore: sync local state after restore (push updates, no pull) a month ago

#title: "Trading Infrastructure Patterns for STARGATE" date: 2026-05-21 source: agent research

#Trading Infrastructure for STARGATE

Production-ready patterns for running systematic trading on Arch Linux with systemd, TimescaleDB, Grafana, and Prometheus.

#1. systemd Unit Files for Trading Bots

#Core Pattern

# /etc/systemd/system/trading-strategy@.service
[Unit]
Description=Trading strategy %i
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/opt/trading/venv/bin/python /opt/trading/strategies/%i/main.py
WorkingDirectory=/opt/trading/strategies/%i
Restart=always
RestartSec=5
WatchdogSec=60

# Resource limits
MemoryMax=4G
CPUQuota=200%
TasksMax=100

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=trading-%i

# Security
User=trading
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/opt/trading/data /var/lib/postgresql

[Install]
WantedBy=multi-user.target

#Key Features

  • sd_notify watchdog: Bot sends keepalive pings; systemd auto-restarts if pings stop
  • Resource limits: MemoryMax (hard cap, triggers OOM killer), CPUQuota, TasksMax
  • Template units: trading-strategy@momentum-futures.service, trading-strategy@pairs-equities.service
  • Logging: --logfile journald sends to systemd journal; query with journalctl -f -u trading-strategy@*

#Reference

#2. Grafana Dashboards for Trading

#Available Templates

#Key Panels to Build

  • Equity curve (line chart, most critical)
  • Maximum drawdown (underwater plot with annotations)
  • Monthly/daily returns (heatmap)
  • P&L by symbol (bar chart)
  • Win rate, trade count (stat panels)
  • Sharpe, Sortino (stat panels, rolling window)
  • Strategy health: API latency, data freshness, error rate (gauge panels)

#TimescaleDB Integration

Grafana's native PostgreSQL plugin queries continuous aggregates:

SELECT time_bucket('1 day', time) AS day,
       FIRST(open), MAX(high), MIN(low), LAST(close), SUM(volume)
FROM trades GROUP BY day;

#3. Prometheus Exporters

No pre-built trading-specific exporters exist. Standard approach is custom Python exporter:

from prometheus_client import CollectorRegistry, Gauge, start_http_server

registry = CollectorRegistry()
equity = Gauge('trading_equity', 'Account equity', registry=registry)
trades_open = Gauge('trading_open_trades', 'Num open trades', registry=registry)
drawdown = Gauge('trading_drawdown_pct', 'Current drawdown %', registry=registry)

# In your bot loop:
equity.set(account_balance)
trades_open.set(len(open_positions))
drawdown.set(current_drawdown)

start_http_server(9090, registry=registry)

#Reference

#4. TimescaleDB for Financial Tick Data

#Setup

Hypertables (auto-partitioned by time):

CREATE TABLE ticks (
    time TIMESTAMPTZ NOT NULL,
    symbol TEXT NOT NULL,
    price DOUBLE PRECISION,
    volume BIGINT
);
SELECT create_hypertable('ticks', 'time');

Continuous Aggregates for fast OHLCV:

CREATE MATERIALIZED VIEW ohlcv_1h WITH (timescaledb.continuous) AS
SELECT
    time_bucket('1 hour', time) AS bucket,
    symbol,
    FIRST(price, time) AS open,
    MAX(price) AS high,
    MIN(price) AS low,
    LAST(price, time) AS close,
    SUM(volume) AS volume
FROM ticks
GROUP BY bucket, symbol;

Performance: batch insert 1000+ ticks per transaction. ~100K ticks/sec achievable.

Storage estimate: At ~100 bytes/tick, 10 years of 10,000 symbols = ~36.5GB. Well within STARGATE's 916GB.

#Reference

#5. Backup Strategies

#Three-Tier Approach

1. pg_probackup (postgrespro/pg_probackup):

  • Incremental modes: DELTA (reads all pages), PAGE (scans WAL), PTRACK (fastest)
  • Parallel backup, compression, point-in-time recovery
  • Supports PG 13-18
pg_probackup init -B /mnt/storage/backups/pg_probackup
pg_probackup backup -B /mnt/storage/backups/pg_probackup -b DELTA -d trading_db

2. WAL Archiving:

archive_mode = on
archive_command = 'pg_probackup archive-push ... %p'
archive_timeout = 300  # flush every 5 min

3. S3 Sync:

rclone sync /mnt/storage/backups/pg_probackup hetzner-s3:trading-backups/ --transfers 4

#systemd Timer for Backups

# /etc/systemd/system/trading-backup.timer
[Unit]
Description=Daily trading database backup

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

#6. Monitoring and Alerting

#Prometheus Alert Rules

groups:
  - name: trading
    rules:
      - alert: StrategyNoTrades
        expr: rate(trading_trades_total[1h]) == 0
        for: 2h
        annotations:
          summary: "Strategy {{ $labels.strategy }} not trading for 2h"

      - alert: MaxDrawdown
        expr: trading_drawdown_pct > 15
        for: 5m
        annotations:
          summary: "Drawdown {{ $value }}% > threshold"

      - alert: DataFeedStale
        expr: time() - trading_last_tick_timestamp > 300
        for: 1m
        annotations:
          summary: "Data feed stale for {{ $labels.symbol }}"

      - alert: BrokerDisconnected
        expr: trading_broker_connected == 0
        for: 2m
        annotations:
          summary: "Broker {{ $labels.broker }} disconnected"

#7. Log Aggregation (Audit Trail)

#Stack: Loki + Promtail

sudo pacman -S loki promtail
# Promtail scrapes journald for trading-* units
systemctl start loki promtail

#Structured Logging in Python

import structlog

structlog.configure(processors=[structlog.processors.JSONRenderer()])
logger = structlog.get_logger()

logger.info("trade_executed",
            symbol="BTCUSD",
            side="buy",
            price=42000,
            volume=1.5,
            order_id="ord_123",
            strategy="mean_reversion")

#Dual Write

Each trade written to both:

  • PostgreSQL audit table (queryable, compliant)
  • JSON log file (searchable via Loki)

#8. Market Data Ingestion Timers

#systemd Timer (Preferred Over Cron)

# /etc/systemd/system/trading-ingest@.timer
[Unit]
Description=Market data ingestion for %i

[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
Persistent=true

[Install]
WantedBy=timers.target
# /etc/systemd/system/trading-ingest@.service
[Unit]
Description=Ingest market data for %i
After=network-online.target

[Service]
Type=oneshot
ExecStart=/opt/trading/venv/bin/python /opt/trading/scripts/ingest.py --symbol %i --interval 1m
StandardOutput=journal
StandardError=journal

Enable per-symbol: systemctl enable trading-ingest@AAPL.timer

#9. Reconciliation

No major open-source OMS exists. Build bespoke:

def reconcile_positions():
    internal = query_trades_today()   # From your DB
    broker = get_broker_positions()    # Via API

    mismatches = []
    for symbol in set(internal.keys() | broker.keys()):
        if internal.get(symbol, 0) != broker.get(symbol, 0):
            mismatches.append({
                'symbol': symbol,
                'internal': internal[symbol],
                'broker': broker[symbol],
            })

    if mismatches:
        alert("reconciliation_failed", mismatches=mismatches)

Run via systemd timer at market close: OnCalendar=*-*-* 16:05:00

#10. Open-Source Trading Infrastructure Repos

Repo Stars What It Does Fit for STARGATE
MBATS Low Docker-compose: Backtrader + MLflow + Airflow + Postgres + Superset + Jupyter + Minio Extract patterns; too Docker-heavy for bare-metal.
trading-data Low Multi-source tick ingestion (Binance, Bybit, Yahoo, IBKR) -> CSV/TimescaleDB Good reference for multi-source ingest scripts.
freqtrade-dashboard Low Prometheus exporter for Freqtrade REST API Template for custom exporter.
freqtrade 30k+ Complete crypto bot framework with systemd service files, Prometheus export Production-ready patterns. Crypto-focused.

#Architecture Synthesis

trading-strategy@SYMBOL.service
  |-- Strategy instance (systemd + sd_notify watchdog)
  |-- Logs -> journald -> Promtail -> Loki
  |-- Metrics -> custom Prometheus exporter (:9090)
  |-- Trades -> PostgreSQL/TimescaleDB
  |-- Orders -> Audit trail (JSON logs + DB)

trading-ingest@SYMBOL.timer
  |-- Fetches ticks -> TimescaleDB (1-min intervals)

Monitoring (once per server):
  |-- Prometheus (scrapes exporters)
  |-- Grafana (queries Prometheus + PostgreSQL)
  |-- Loki (aggregates journald logs)
  |-- AlertManager (routes to ntfy + email)
  |-- pg_probackup timer (daily backup to /mnt/storage)

/mnt/storage/
  |-- backups/pg_probackup/    (incremental DELTA backups)
  |-- trading-logs/            (JSON audit trails)
  |-- tick-archive/            (Parquet exports for cold storage)