Investing glossary · Quant, AI and market regimes
Hidden Markov models in investing: detecting market regimes
How it works
- Hidden states: for example two or three regimes. You choose the number; the model learns what each looks like.
- Observations: what you feed it, such as daily or weekly returns, volatility, or momentum measures like the PPO.
- Transition probabilities: how likely each state is to persist or switch next period. Regimes tend to be sticky, which is what makes them useful.
The approach was brought into economics by James Hamilton in 1989, to model switches between economic expansion and recession.
The look-ahead trap
Most HMM charts you see online are smoothed: the model was fitted on the whole history, so its view of 2008 uses data from 2009 onwards. That makes regimes look obvious and turning points look perfectly timed. For real decisions only the filtered, causal estimate counts: the probability using only data available on that date, with the model refitted on past data alone. Causal estimates switch later and give more false alarms.
Using it sensibly
- Treat the output as a probability for risk management, not a buy or sell signal.
- Test out of sample with walk-forward refitting, and include trading costs.
- Combine it with simple trend and momentum checks; agreement is more useful than any single model.
Our Market Regime page gives a weekly view of the US market’s trend, momentum and volatility climate.
Questions people ask
Can a hidden Markov model predict the stock market?
Not reliably. It estimates which regime the market is probably in now, based on recent behaviour. Regimes are persistent, which helps with risk control, but the model cannot see shocks coming.
How many states should a market HMM have?
Two or three is common: for example low-volatility rising, high-volatility falling and a middle state. More states fit history better but are harder to use and more likely to overfit.
What is the difference between filtered and smoothed probabilities?
Filtered probabilities use only data up to each date, as you would have had in real time. Smoothed probabilities use the whole sample, including the future, so they look better than anything you could have traded.
Is a hidden Markov model a form of AI?
It is a statistical machine-learning model, much older and simpler than modern neural networks, which makes it easier to understand and test.
Related terms
- Market regimeA broad market environment, such as a low-volatility uptrend or a high-volatility downtrend, in which different strategies tend to work.
- Look-ahead biasWhen a backtest accidentally uses information that was not available at the time, such as revised data or a model fitted on the full history.
- BacktestTesting a strategy on historical data to see how it would have performed.
Sources
Important: this page is education and information only. It is not financial or tax advice or a personal recommendation, and it does not take your circumstances into account. Tax rules can change and depend on your circumstances; UK figures are for the 2026/27 tax year. The value of investments and pensions can fall as well as rise, and you may get back less than you put in. Examples are illustrations, not predictions. If you are unsure what is right for you, speak to a regulated financial adviser.