Investment Concepts
The Hidden State Problem
Imagine you’re watching a stock price move up and down. You can see the price — that’s the observable output. But you can’t directly see the underlying state driving those movements. Is the market in a trending state? A mean-reverting state? A high-volatility state? These hidden states are what Hidden Markov Models (HMMs) try to uncover.
An HMM is a statistical model that assumes observable data (like returns, volume, or volatility) is generated by a system that switches between a finite number of hidden states. The model estimates both what those states look like and which state the market is probably in right now.
How It Works in Plain English
Think of the market as having different “moods” — calm and trending up, choppy and directionless, panicked and falling. Each mood produces different patterns in returns and volatility. An HMM learns these patterns from historical data and then, given today’s market behaviour, calculates the probability of being in each mood.
The “Markov” part means the model assumes the current state depends only on the previous state — not the entire history. This simplification makes the maths tractable while still capturing the essential dynamics of regime switching.
How to Read It
- State probabilities above 80%: The model is confident about the current regime. Strategies aligned with that regime should perform well.
- Probabilities between 50–80%: Moderate confidence. The market may be transitioning between regimes. Reduce position sizes and expect choppier conditions.
- No state above 50%: The model is uncertain. This often happens during regime transitions — the most dangerous period for any strategy. Consider sitting out or running minimal exposure.
- Rapid state switching: If the model keeps flipping between states, the current market behaviour doesn’t fit neatly into any learned pattern. This itself is useful information.
Connection to Market Regimes
HMMs are a quantitative approach to market regime detection. Where a discretionary trader might look at charts and say “this looks like accumulation,” an HMM processes return distributions and volatility patterns to reach a similar conclusion with probabilities attached.
Most practical HMMs for markets use two or three states: bull (low volatility, positive drift), bear (high volatility, negative drift), and sometimes a neutral or transitional state. Adding more states increases complexity without always improving results.
Practical Example
An HMM trained on S&P 500 data might identify two states: State 1 has average daily returns of +0.04% with 0.8% daily volatility (the bull regime). State 2 has average daily returns of −0.10% with 2.1% daily volatility (the bear regime). On any given day, the model outputs something like “85% probability of State 1” — telling you the market is most likely in the calm, upward-drifting regime.
Limitations and Pitfalls
HMMs are backward-looking — they learn from history. If the market enters a regime that’s genuinely unprecedented, the model will struggle. They can also be sensitive to the training period: train on 2010–2019 (mostly bull market) and the model may not handle crashes well.
The number of states must be chosen in advance, which is somewhat arbitrary. Two states is simple but may miss nuance. Five states might capture more detail but risks overfitting — finding patterns in noise rather than signal.
Using HMMs in Practice
The best use of HMMs isn’t predicting the future — it’s characterising the present. Knowing you’re in a high-volatility regime tells you to reduce position sizes, widen stops, and favour mean-reversion strategies. Knowing you’re in a trending regime says the opposite. Pair HMM output with Sharpe Ratio analysis to confirm whether your strategy’s edge is regime-dependent.
Key takeaway: Hidden Markov Models don’t predict the future — they characterise the present. Knowing the probability of the current market regime helps you match your strategy to the environment rather than fighting it.
