Markov Processes

A hybrid Rust-Python ensemble trading system for price range forecasting using advanced statistical models including TGARCH, Kalman Filters (UKF/EKPF), Jump Diffusion, and Monte Carlo simulations with cross-feeding architecture.

Overview

This project implements a sophisticated ensemble trading system that combines multiple statistical and machine learning models to forecast price ranges for financial instruments. The architecture features:

• Hybrid Rust-Python Architecture: Computationally intensive operations (Kalman filters, particle filters, Monte Carlo simulations) are accelerated using Rust via PyO3 bindings
• Cross-Feeding Design: Models share information through a staged combustion-like architecture where VIX flows into TGARCH, option chain data informs Delta UKF, and ensemble parameters are shared across models
• Dynamic Weighting: Ensemble weights adjust based on market conditions and option chain factors
• Production-Ready: Comprehensive CI/CD pipeline, testing infrastructure, and error handling

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

Architecture

Hybrid Model Architecture

The system follows a hybrid architecture pattern:

• Python Layer: Business logic, data fetching, configuration, and orchestration
• Rust Core: High-performance implementations of:
  • Unscented Kalman Filter (UKF)
  • Extended Kalman Particle Filter (EKPF)
  • Parallel Monte Carlo simulation
  • Jump diffusion simulation

Cross-Feeding Flow

VIX Data → TGARCH Model → Volatility Adjustment
Option Chain → Delta UKF → Greeks Filtering
Market Data → Ensemble → Dynamic Weighting

Model Components

Model	Implementation	Purpose
TGARCH	Python + Rust	Volatility forecasting with VIX coefficient
Delta UKF	Python (Rust UKF)	Options Greeks filtering with Kalman smoothing
Jump Diffusion VIX	Python + Rust MC	Price simulation with jump components
EKPF	Python + Rust	Particle filtering for state estimation
ARIMA	Python	Autoregressive integrated moving average
VARMAX	Python	Vector autoregressive moving average with exogenous variables

Installation

Prerequisites

• Python 3.10+
• Rust 1.70+ (for building the Rust core)
• Cargo (comes with Rust)

Install Rust

If you don't have Rust installed:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

Install Python Dependencies and Build Rust Extension

# Install Python dependencies and build Rust extension in one step
pip install -e .

# Or install with development dependencies
pip install -e ".[dev]"

The pip install -e . command will automatically:

• Install all Python dependencies
• Build the Rust extension using maturin
• Install the package in editable mode

Build Rust Extension Separately (Optional)

If you need to rebuild the Rust extension separately:

# Build the Rust core with PyO3 bindings
cd rust_core
maturin develop --release
cd ..

Or build in development mode for faster compilation:

cd rust_core
maturin develop
cd ..

Usage

CLI Interface

# Forecast price range for a ticker
markov forecast --ticker SPY --days 5

# Run ensemble with custom configuration
markov ensemble --config config/production.yaml

# Validate model performance
markov validate --ticker SPY --backtest-days 30

Python API

from ensemble.model_ensemble import ModelEnsemble
from config.config_loader import ConfigLoader
from data.data_sources import YFinanceSource, MarketDataFetcher

# Load configuration
config = ConfigLoader.load('config/production.yaml')

# Initialize ensemble
ensemble = ModelEnsemble(config)

# Fetch data
fetcher = MarketDataFetcher(YFinanceSource())
data = fetcher.fetch('SPY', days_back=90)

# Fit ensemble
ensemble.fit(data)

# Forecast
result = ensemble.forecast(steps=5)
print(f"Low: {result.low_estimate:.2f}")
print(f"High: {result.high_estimate:.2f}")

Using Individual Models

from models.tgarch_model import TGARCHModel
from models.delta_ukf import DeltaUKF
from models.jump_diffusion_vix import JumpDiffusionVIX

# TGARCH with VIX coefficient
tgarch = TGARCHModel()
tgarch.fit(data, vix_coefficient=0.15)
forecast = tgarch.forecast(steps=5)

# Delta UKF with option chain data
delta_ukf = DeltaUKF()
delta_ukf.fit(data, option_chain=option_data)
forecast = delta_ukf.forecast(steps=5)

# Jump Diffusion with VIX
jd_vix = JumpDiffusionVIX()
jd_vix.fit(data, vix=vix_data)
forecast = jd_vix.forecast(steps=5, num_simulations=10000)

Configuration

Configuration is YAML-based and supports:

ensemble:
  models:
    - name: tgarch
      weight: 0.06
    - name: delta_ukf
      weight: 0.30
    - name: jump_diffusion_vix
      weight: 0.10
    - name: ekpf
      weight: 0.10
    - name: arima
      weight: 0.02
    - name: varmax
      weight: 0.06

data:
  source: yfinance
  default_ticker: SPY
  default_days_back: 90

rust:
  enable_acceleration: true
  num_threads: 4

vix:
  low_threshold: 12.5
  high_threshold: 18.5
  coefficient: 0.1

Development

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=.

# Run specific test file
pytest tests/test_ensemble.py

Code Quality

# Format code with black
black .

# Lint with ruff
ruff check .

# Type checking with mypy
mypy .

Building for Distribution

# Build wheel with Rust extension
maturin build --release

# Build for multiple platforms
maturin build --release --universal

CI/CD

The project uses GitHub Actions for CI/CD:

• Python Tests: Runs pytest on multiple Python versions
• Rust Tests: Runs cargo test
• Cross-Platform Build: Builds wheels for Linux, macOS, Windows
• Linting: Black, Ruff, Mypy for Python; Clippy for Rust
• Security: Bandit for Python, cargo-audit for Rust
• Integration Tests: End-to-end ensemble validation
• Deployment: Automatic PyPI release on tags

See .github/workflows/ci.yml for details.

Legacy Scripts

The legacy_scripts/ directory contains original standalone Python scripts before refactoring. These are deprecated but maintained for reference:

Legacy Script	New Implementation
ARIMA	models/arima_model.py
VARMAX	models/varmax_model.py
TGARCH VIX	models/tgarch_model.py
Jump Diffusion VIX	models/jump_diffusion_vix.py
Monte-Carlo, UKF, VIX	core/rust_wrappers.py + Rust core
EKPF	core/rust_wrappers.py + Rust core
Model Average	ensemble/model_ensemble.py
Sequential Monte Carlo, UKF	core/rust_wrappers.py + Rust core
TGARCH, UKF	models/tgarch_model.py with Kalman integration

All legacy scripts have been hardened with:

• Apache 2.0 license headers
• Error handling and logging
• Command-line argument support
• Type hints and docstrings

Project Structure

Markov-Processes/
├── rust_core/              # Rust acceleration layer
│   ├── src/
│   │   ├── lib.rs         # PyO3 bindings
│   │   ├── kalman.rs      # Unscented Kalman Filter
│   │   ├── particle_filter.rs  # Extended Kalman Particle Filter
│   │   ├── monte_carlo.rs # Parallel Monte Carlo
│   │   └── jump_diffusion.rs   # Jump diffusion simulation
│   └── Cargo.toml
├── models/                 # Python model implementations
│   ├── base_model.py      # Abstract base classes
│   ├── tgarch_model.py    # TGARCH with VIX coefficient
│   ├── delta_ukf.py       # Delta UKF with option chain
│   ├── jump_diffusion_vix.py   # Jump Diffusion with VIX
│   ├── arima_model.py     # ARIMA model
│   └── varmax_model.py    # VARMAX model
├── ensemble/               # Ensemble system
│   ├── model_ensemble.py  # Ensemble orchestration
│   └── weighting.py       # Dynamic weight calculation
├── core/                   # Core utilities
│   ├── rust_wrappers.py   # Python wrappers for Rust
│   └── data_sources.py    # Data fetching
├── config/                 # Configuration
│   └── config_loader.py   # Configuration loading
├── tests/                  # Test suite
│   ├── test_ensemble.py
│   ├── test_models.py
│   └── test_rust_core.py
├── legacy_scripts/        # Deprecated legacy scripts
├── .github/workflows/     # CI/CD pipelines
│   └── ci.yml
├── pyproject.toml         # Project configuration
├── Cargo.toml            # Rust dependencies
└── LICENSE               # Apache 2.0 license

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes with tests
4. Ensure all tests pass and code is formatted
5. Submit a pull request

Development Guidelines

• Follow PEP 8 for Python code
• Use type hints for all public functions
• Write docstrings for all modules and public APIs
• Add tests for new features
• Run black . and ruff check . before committing

Performance Benchmarks

Rust acceleration provides significant performance improvements:

Operation	Python Only	Rust Accelerated	Speedup
UKF (1000 steps)	450ms	35ms	12.8x
EKPF (10000 particles)	2.3s	180ms	12.8x
Monte Carlo (10000 sims)	1.8s	120ms	15x
Jump Diffusion (10000 sims)	2.1s	150ms	14x

Acknowledgments

• PyO3 for Rust-Python bindings
• Maturin for build automation
• statsmodels for ARIMA/VARMAX
• arch library for TGARCH
• filterpy for Kalman filter implementations

License

Apache License 2.0 - see LICENSE for details.