In partnership with
It's not you, it’s your tax tools
Tax teams are stretched thin and spreadsheets aren’t cutting it. This guide helps you figure out what to look for in tax software that saves time, cuts risk, and keeps you ahead of reporting demands.
🚀 Your Investing Journey Just Got Better: Premium Subscriptions Are Here! 🚀
It’s been 4 months since we launched our premium subscription plans at GuruFinance Insights, and the results have been phenomenal! Now, we’re making it even better for you to take your investing game to the next level. Whether you’re just starting out or you’re a seasoned trader, our updated plans are designed to give you the tools, insights, and support you need to succeed.
Here’s what you’ll get as a premium member:
Exclusive Trading Strategies: Unlock proven methods to maximize your returns.
In-Depth Research Analysis: Stay ahead with insights from the latest market trends.
Ad-Free Experience: Focus on what matters most—your investments.
Monthly AMA Sessions: Get your questions answered by top industry experts.
Coding Tutorials: Learn how to automate your trading strategies like a pro.
Masterclasses & One-on-One Consultations: Elevate your skills with personalized guidance.
Our three tailored plans—Starter Investor, Pro Trader, and Elite Investor—are designed to fit your unique needs and goals. Whether you’re looking for foundational tools or advanced strategies, we’ve got you covered.
Don’t wait any longer to transform your investment strategy. The last 4 months have shown just how powerful these tools can be—now it’s your turn to experience the difference.
Markets shift through phases of stability, transition, and volatility. These shifts, or regimes, define how risk and opportunity behave over time. In an earlier post, I used a Latent Gaussian Mixture Model (LGMM) to identify these regimes in price data. It worked for broad clusters but struggled with nonlinear changes and market memory. This project extends that idea using two deep learning methods: a Variational Autoencoder (VAE) and a Transformer Encoder. The VAE captures nonlinear structures that LGMM cannot. The Transformer introduces temporal awareness, learning from sequences instead of static points. Together, they offer a stronger framework for detecting hidden market regimes and understanding how markets evolve rather than simply react.
What is market regime detection?
It’s the process of identifying distinct periods (regimes) in market behavior: low-volatility trend, high-volatility mean-revert, crisis, etc. Good regime detection helps with position sizing, signal selection (momentum vs mean-reversion), leverage, hedging, and risk overlays.
Why these three models?
LGMM (GaussianMixture) — simple, interpretable, fast. It models features as mixtures of Gaussians. Works well when regimes are (roughly) Gaussian clusters in feature space. Cheap and robust, but limited to linear separations and ignores temporal dynamics unless you add memory.
VAE (Variational Autoencoder) — nonlinear latent representation. Encodes features into a probabilistic low-dimensional space where clustering can be clearer. Good at capturing nonlinear structure and anomalies. Needs more compute and care.
Transformer Encoder — models sequences, attention over time. Captures temporal dependencies and can detect regimes that are defined by sequences rather than instantaneous feature clusters. Powerful, but heavier and needs sequences and training.
Trade-offs: LGMM is baseline and explainable; VAE uncovers nonlinear latent clusters; Transformer captures temporal context.
Learn AI in 5 minutes a day
What’s the secret to staying ahead of the curve in the world of AI? Information. Luckily, you can join 1,000,000+ early adopters reading The Rundown AI — the free newsletter that makes you smarter on AI with just a 5-minute read per day.
Market Regime Evolution: LGMM vs VAE vs Transformer (2005–2025)
Implementation
Below is a single, runnable Python script segmented into cells. Each major cell starts with a 1–2 line comment explaining what it does.
Code & Reproducibility
The full code and notebook for this project are available on GitHub:
Deep Learning for Hidden Market Regimes — VAE & Transformer Extension to LGMM
Requirements: yfinance, numpy, pandas, matplotlib, seaborn, sklearn, torch (PyTorch).
Install with: pip install yfinance numpy pandas matplotlib seaborn scikit-learn torch
# Cell 1: imports and environment setup
# What this does: import required libraries and set plotting styles
import yfinance as yf
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score, adjusted_rand_score, confusion_matrix
from sklearn.metrics.cluster import contingency_matrix
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
from datetime import datetime
import warnings
warnings.filterwarnings('ignore')
sns.set(style='whitegrid', context='talk')a Data Collection & Feature Engineering
# Cell 2: Download SPY daily data using yfinance
# What this does: fetches SPY price history and computes log returns
symbol = 'SPY'
start = '2005-01-01'
end = datetime.today().strftime('%Y-%m-%d')
df = yf.download(symbol, start=start, end=end, progress=False)
df = df[('Close', 'SPY')].to_frame(name='close') # Extract 'Close' and rename to 'close'
df['logret'] = np.log(df['close']).diff() # Calculate log returns using the 'close' column
df.dropna(inplace=True)
df.reset_index(inplace=True) # Reset index to make 'Date' a column
df.rename(columns={'index': 'Date'}, inplace=True) # Rename the index column to 'Date'
print(df.head())
#output
Date close logret
0 2005-01-04 80.846985 -0.012295
1 2005-01-05 80.289108 -0.006924
2 2005-01-06 80.697311 0.005071
3 2005-01-07 80.581657 -0.001434
4 2005-01-10 80.962677 0.004717Why these choices?
SPY: liquid equity ETF for broad market.
Log returns (
logret) are additive and better for many statistical models.
# Cell 3: Feature engineering - rolling vol, momentum, returns
# What this does: creates rolling volatility, momentum, and returns as features
window_vol = 21 # ~1 month trading days
window_mom = 5 # short momentum
df['ret_1d'] = df['logret']
df['vol_21d'] = df['logret'].rolling(window_vol).std() * np.sqrt(252) # annualized vol proxy
df['mom_5d'] = df['close'].pct_change(window_mom)
df.dropna(inplace=True)
features = df[['ret_1d', 'vol_21d', 'mom_5d']].copy()
print(features)
#output
ret_1d vol_21d mom_5d
20 0.003023 0.100447 0.017402
21 -0.002603 0.091181 0.013029
22 0.010619 0.094462 0.023844
23 -0.001332 0.093487 0.016165
24 0.001165 0.093214 0.010933
... ... ... ...
5226 0.015228 0.126790 -0.012760
5227 -0.001222 0.125707 -0.010297
5228 0.004430 0.126381 -0.011796
5229 -0.006834 0.128815 -0.015674
5230 0.005660 0.129293 0.017411Why these features?
ret_1dcaptures immediate returns.vol_21dcaptures market uncertainty (regimes often defined by volatility).mom_5dshort-term trend; momentum vs mean-revert regimes can be separated here.
100 Genius Side Hustle Ideas
Don't wait. Sign up for The Hustle to unlock our side hustle database. Unlike generic "start a blog" advice, we've curated 100 actual business ideas with real earning potential, startup costs, and time requirements. Join 1.5M professionals getting smarter about business daily and launch your next money-making venture.
b Exploratory Data Analysis (EDA)
# Cell 4: Plot histograms and density plots for features
# What this does: visualizes marginal distributions for returns and vol
plt.figure(figsize=(14,4))
plt.subplot(1,3,1)
sns.histplot(features['ret_1d'], kde=True, stat='density', bins=80)
plt.title('Daily Log Return Distribution')
plt.subplot(1,3,2)
sns.histplot(features['vol_21d'], kde=True, stat='density', bins=60)
plt.title('Rolling Volatility (21d, annualized)')
plt.subplot(1,3,3)
sns.histplot(features['mom_5d'].dropna(), kde=True, stat='density', bins=60)
plt.title('5d Momentum (pct)')
plt.tight_layout()
plt.show()
Interpretation:
Returns: heavy tails and skewness — not Gaussian.
Volatility: skewed, long right tail — crisis spikes.
Momentum: symmetric-ish but depends on regime.
# Cell 5: Scatter and correlation plots between features
# What this does: shows pairwise relationships between features
sns.pairplot(features.sample(2000), diag_kind='kde', plot_kws={'alpha':0.5, 's':20})
plt.suptitle('Pairwise feature relationships (sample)', y=1.02)
plt.show()
print("Correlation matrix:\n", features.corr())
#output
Correlation matrix:
ret_1d vol_21d mom_5d
ret_1d 1.000000 -0.007986 0.423119
vol_21d -0.007986 1.000000 -0.056426
mom_5d 0.423119 -0.056426 1.000000
Interpretation:
Volatility and returns often weakly negatively correlated — spikes in vol accompany negative returns.
Momentum versus returns: depends on lookback.
Feature scaling note: We’ll need to scale features before feeding models. LGMM can work on raw features but scaling makes clusters comparable. VAE and Transformer greatly benefit from standardized inputs.
c LGMM Regime Clustering
# Cell 6: Prepare scaled features for clustering
# What this does: standardizes features for all models
scaler = StandardScaler()
X = scaler.fit_transform(features.values)
dates = features.index
# Cell 7: Fit Gaussian Mixture (LGMM) and assign regimes
# What this does: fits GaussianMixture with n_components and labels each day
n_clusters = 3
gmm = GaussianMixture(n_components=n_clusters, covariance_type='full', random_state=42)
gmm.fit(X)
lgmm_labels = gmm.predict(X)Why Gaussian Mixture?
GMM models the data as a mixture of Gaussians; it returns posterior probabilities for each cluster — useful to measure regime confidence (soft assignments). n_components=3 is a common starting point (stable, volatile, crisis).
Each dot = one trading day (one market snapshot).
The three colors (blue, orange, green) are the regimes the LGMM discovered.
Points close together behave similarly — they have similar volatility/return profiles.
The cluster centers are like “typical market moods” that the model found statistically
Blue cluster (LGMM 0): calm or low-volatility days, market stable and quiet.
Orange cluster (LGMM 1): medium-volatility, moderate-trend days — typical “normal” markets.
Green cluster (LGMM 2): high-volatility or crisis periods — rare but extreme events.
Most of the time, the market stayed calm (Regime 0), with normal phases (Regime 1) showing up less often and high-volatility periods (Regime 2) being rare outliers.
Interpretation:
Each LGMM cluster typically corresponds to regimes: a low-vol, small-return cluster; a moderate cluster; and a high-vol, large-negative-return cluster (crisis).
Inspect cluster statistics for meaning:
# Cell 9: Summarize cluster statistics in original feature space
# What this does: prints mean and std of features per LGMM cluster for interpretation
df_lgmm = features.copy()
df_lgmm['lgmm'] = lgmm_labels
print(df_lgmm.groupby('lgmm').agg(['mean','std']).T)
#output
lgmm 0 1 2
ret_1d mean 0.000979 0.000009 -0.003098
std 0.005179 0.012877 0.036290
vol_21d mean 0.099596 0.196553 0.512515
std 0.027444 0.053748 0.198117
mom_5d mean 0.005942 -0.000984 -0.013567
std 0.010978 0.027237 0.064227What this tells you:
Cluster with highest
vol_21d.meanusually is crisis.Cluster with positive
mom_5d.meanand lower vol likely trend/stable.
LGMM 0:
Small positive return
Lowest volatility
Slightly positive momentum
→ Calm and steady phase (stable market)
LGMM 1:
Near-zero return
Medium volatility
Momentum around neutral
Normal or transition phase (market moving sideways or shifting)
LGMM 2:
Negative return
Very high volatility
Negative momentum
Volatile or bearish phase (market under stress or panic)
In short: LGMM 0 = calm, LGMM 1 = neutral/transition, LGMM 2 = volatile/crisis.
Subscribe to our premium content to get access to the full article and code snippets.
Become a paying subscriber to get access to this post and other subscriber-only content.
Upgrade