---
title: "Trading Infrastructure Patterns for STARGATE"
date: 2026-05-21
source: agent research
---

# Trading Infrastructure for STARGATE

Production-ready patterns for running systematic trading on Arch Linux with systemd, TimescaleDB, Grafana, and Prometheus.

## 1. systemd Unit Files for Trading Bots

### Core Pattern

```ini
# /etc/systemd/system/trading-strategy@.service
[Unit]
Description=Trading strategy %i
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/opt/trading/venv/bin/python /opt/trading/strategies/%i/main.py
WorkingDirectory=/opt/trading/strategies/%i
Restart=always
RestartSec=5
WatchdogSec=60

# Resource limits
MemoryMax=4G
CPUQuota=200%
TasksMax=100

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=trading-%i

# Security
User=trading
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/opt/trading/data /var/lib/postgresql

[Install]
WantedBy=multi-user.target
```

### Key Features
- **sd_notify watchdog**: Bot sends keepalive pings; systemd auto-restarts if pings stop
- **Resource limits**: `MemoryMax` (hard cap, triggers OOM killer), `CPUQuota`, `TasksMax`
- **Template units**: `trading-strategy@momentum-futures.service`, `trading-strategy@pairs-equities.service`
- **Logging**: `--logfile journald` sends to systemd journal; query with `journalctl -f -u trading-strategy@*`

### Reference
- [Freqtrade Advanced Setup (systemd)](https://www.freqtrade.io/en/stable/advanced-setup/) -- production-grade patterns with sd_notify
- [systemd Resource Limits](https://blog.theravenhub.com/post/ht-limit-service-ressources-systemd)

## 2. Grafana Dashboards for Trading

### Available Templates
- [Freqtrade Dashboard](https://grafana.com/grafana/dashboards/14915-freqtrade/) -- equity curve, underwater plot, win/loss ratio, multi-strategy comparison
- [Trade Dashboard](https://grafana.com/grafana/dashboards/18100-new-trade-dashboard/) -- MySQL/Prometheus datasource
- [Stock Portfolio](https://grafana.com/grafana/dashboards/15956-stock-portfolio/) -- community template

### Key Panels to Build
- Equity curve (line chart, most critical)
- Maximum drawdown (underwater plot with annotations)
- Monthly/daily returns (heatmap)
- P&L by symbol (bar chart)
- Win rate, trade count (stat panels)
- Sharpe, Sortino (stat panels, rolling window)
- Strategy health: API latency, data freshness, error rate (gauge panels)

### TimescaleDB Integration
Grafana's native PostgreSQL plugin queries continuous aggregates:
```sql
SELECT time_bucket('1 day', time) AS day,
       FIRST(open), MAX(high), MIN(low), LAST(close), SUM(volume)
FROM trades GROUP BY day;
```

## 3. Prometheus Exporters

No pre-built trading-specific exporters exist. Standard approach is custom Python exporter:

```python
from prometheus_client import CollectorRegistry, Gauge, start_http_server

registry = CollectorRegistry()
equity = Gauge('trading_equity', 'Account equity', registry=registry)
trades_open = Gauge('trading_open_trades', 'Num open trades', registry=registry)
drawdown = Gauge('trading_drawdown_pct', 'Current drawdown %', registry=registry)

# In your bot loop:
equity.set(account_balance)
trades_open.set(len(open_positions))
drawdown.set(current_drawdown)

start_http_server(9090, registry=registry)
```

### Reference
- [Freqtrade Dashboard exporter](https://github.com/thraizz/freqtrade-dashboard) -- scrapes Freqtrade REST API, emits Prometheus metrics
- [Prometheus Exporters Guide](https://betterstack.com/community/guides/monitoring/prometheus-exporter/)

## 4. TimescaleDB for Financial Tick Data

### Setup

**Hypertables** (auto-partitioned by time):
```sql
CREATE TABLE ticks (
    time TIMESTAMPTZ NOT NULL,
    symbol TEXT NOT NULL,
    price DOUBLE PRECISION,
    volume BIGINT
);
SELECT create_hypertable('ticks', 'time');
```

**Continuous Aggregates** for fast OHLCV:
```sql
CREATE MATERIALIZED VIEW ohlcv_1h WITH (timescaledb.continuous) AS
SELECT
    time_bucket('1 hour', time) AS bucket,
    symbol,
    FIRST(price, time) AS open,
    MAX(price) AS high,
    MIN(price) AS low,
    LAST(price, time) AS close,
    SUM(volume) AS volume
FROM ticks
GROUP BY bucket, symbol;
```

**Performance**: batch insert 1000+ ticks per transaction. ~100K ticks/sec achievable.

**Storage estimate**: At ~100 bytes/tick, 10 years of 10,000 symbols = ~36.5GB. Well within STARGATE's 916GB.

### Reference
- [TimescaleDB Financial Tick Data Tutorial](https://docs.timescale.com/tutorials/latest/financial-tick-data/)

## 5. Backup Strategies

### Three-Tier Approach

**1. pg_probackup** ([postgrespro/pg_probackup](https://github.com/postgrespro/pg_probackup)):
- Incremental modes: DELTA (reads all pages), PAGE (scans WAL), PTRACK (fastest)
- Parallel backup, compression, point-in-time recovery
- Supports PG 13-18

```bash
pg_probackup init -B /mnt/storage/backups/pg_probackup
pg_probackup backup -B /mnt/storage/backups/pg_probackup -b DELTA -d trading_db
```

**2. WAL Archiving**:
```
archive_mode = on
archive_command = 'pg_probackup archive-push ... %p'
archive_timeout = 300  # flush every 5 min
```

**3. S3 Sync**:
```bash
rclone sync /mnt/storage/backups/pg_probackup hetzner-s3:trading-backups/ --transfers 4
```

### systemd Timer for Backups

```ini
# /etc/systemd/system/trading-backup.timer
[Unit]
Description=Daily trading database backup

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target
```

## 6. Monitoring and Alerting

### Prometheus Alert Rules

```yaml
groups:
  - name: trading
    rules:
      - alert: StrategyNoTrades
        expr: rate(trading_trades_total[1h]) == 0
        for: 2h
        annotations:
          summary: "Strategy {{ $labels.strategy }} not trading for 2h"

      - alert: MaxDrawdown
        expr: trading_drawdown_pct > 15
        for: 5m
        annotations:
          summary: "Drawdown {{ $value }}% > threshold"

      - alert: DataFeedStale
        expr: time() - trading_last_tick_timestamp > 300
        for: 1m
        annotations:
          summary: "Data feed stale for {{ $labels.symbol }}"

      - alert: BrokerDisconnected
        expr: trading_broker_connected == 0
        for: 2m
        annotations:
          summary: "Broker {{ $labels.broker }} disconnected"
```

## 7. Log Aggregation (Audit Trail)

### Stack: Loki + Promtail

```bash
sudo pacman -S loki promtail
# Promtail scrapes journald for trading-* units
systemctl start loki promtail
```

### Structured Logging in Python

```python
import structlog

structlog.configure(processors=[structlog.processors.JSONRenderer()])
logger = structlog.get_logger()

logger.info("trade_executed",
            symbol="BTCUSD",
            side="buy",
            price=42000,
            volume=1.5,
            order_id="ord_123",
            strategy="mean_reversion")
```

### Dual Write
Each trade written to both:
- PostgreSQL audit table (queryable, compliant)
- JSON log file (searchable via Loki)

## 8. Market Data Ingestion Timers

### systemd Timer (Preferred Over Cron)

```ini
# /etc/systemd/system/trading-ingest@.timer
[Unit]
Description=Market data ingestion for %i

[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
Persistent=true

[Install]
WantedBy=timers.target
```

```ini
# /etc/systemd/system/trading-ingest@.service
[Unit]
Description=Ingest market data for %i
After=network-online.target

[Service]
Type=oneshot
ExecStart=/opt/trading/venv/bin/python /opt/trading/scripts/ingest.py --symbol %i --interval 1m
StandardOutput=journal
StandardError=journal
```

Enable per-symbol: `systemctl enable trading-ingest@AAPL.timer`

## 9. Reconciliation

No major open-source OMS exists. Build bespoke:

```python
def reconcile_positions():
    internal = query_trades_today()   # From your DB
    broker = get_broker_positions()    # Via API

    mismatches = []
    for symbol in set(internal.keys() | broker.keys()):
        if internal.get(symbol, 0) != broker.get(symbol, 0):
            mismatches.append({
                'symbol': symbol,
                'internal': internal[symbol],
                'broker': broker[symbol],
            })

    if mismatches:
        alert("reconciliation_failed", mismatches=mismatches)
```

Run via systemd timer at market close: `OnCalendar=*-*-* 16:05:00`

## 10. Open-Source Trading Infrastructure Repos

| Repo | Stars | What It Does | Fit for STARGATE |
|---|---|---|---|
| [MBATS](https://github.com/saeed349/Microservices-Based-Algorithmic-Trading-System) | Low | Docker-compose: Backtrader + MLflow + Airflow + Postgres + Superset + Jupyter + Minio | Extract patterns; too Docker-heavy for bare-metal. |
| [trading-data](https://github.com/lokhiufung/trading-data) | Low | Multi-source tick ingestion (Binance, Bybit, Yahoo, IBKR) -> CSV/TimescaleDB | Good reference for multi-source ingest scripts. |
| [freqtrade-dashboard](https://github.com/thraizz/freqtrade-dashboard) | Low | Prometheus exporter for Freqtrade REST API | Template for custom exporter. |
| [freqtrade](https://github.com/freqtrade/freqtrade) | 30k+ | Complete crypto bot framework with systemd service files, Prometheus export | Production-ready patterns. Crypto-focused. |

## Architecture Synthesis

```
trading-strategy@SYMBOL.service
  |-- Strategy instance (systemd + sd_notify watchdog)
  |-- Logs -> journald -> Promtail -> Loki
  |-- Metrics -> custom Prometheus exporter (:9090)
  |-- Trades -> PostgreSQL/TimescaleDB
  |-- Orders -> Audit trail (JSON logs + DB)

trading-ingest@SYMBOL.timer
  |-- Fetches ticks -> TimescaleDB (1-min intervals)

Monitoring (once per server):
  |-- Prometheus (scrapes exporters)
  |-- Grafana (queries Prometheus + PostgreSQL)
  |-- Loki (aggregates journald logs)
  |-- AlertManager (routes to ntfy + email)
  |-- pg_probackup timer (daily backup to /mnt/storage)

/mnt/storage/
  |-- backups/pg_probackup/    (incremental DELTA backups)
  |-- trading-logs/            (JSON audit trails)
  |-- tick-archive/            (Parquet exports for cold storage)
```