As we’ve seen, it’s not pretty reading if you’re a Prophet fan: Meta’s tool manages to snatch a few folds (metric dependent) to avoid a complete whitewash. Impartial commentators might suggest a return to the clubhouse to re-evaluate tactics.
While the result isn’t great for Prophet, there are a few reasons why performance like this can be expected.
The features
The LASSO model uses features that have been specifically engineered for this particular time series. The set of input features available to it is essentially a superset of what’s available to Prophet with a little extra on the side.
Additionally, some of the features are subtly different in the LASSO model. For instance, features describing potential change points are not as constrained in the LASSO as they are in the Prophet model.
Think of it as trying to out-guess someone else, knowing less than — or slightly different — to them. Not so easy.
The modelling
The out-of-fold data is not as “unseen” as I’ve made it out to be.
In a previous article we covered the parameterisation of the LASSO model: how we use out-of-fold data to select the strength of regularisation which optimised the model’s ability to forecast. In that sense the LASSO model has been tuned to forecast well over each cut of the data while the Prophet model has been thrown straight out of the box and into the deep end.
In “normal” hyperparameter optimisation exercises, we can usually expect to see performance increases by about 1% — 2%; the performance increase in a time series context is likely much greater as “out-of-fold” really is “out-of-time”.
Time to call it a day with Prophet then?
Not so fast… this series of articles has certainly highlighted a few things — let’s talk through a few of them.
Out of the box, Prophet works incredibly well. Although it can indeed be beaten, it takes a bit of work to do so — much more than the 10 lines of code you need to get Prophet up and forecasting.
The interpretability of the LASSO model is far superior to what’s available from Prophet. Yes, Prophet gives us estimates of uncertainty for forecasts but we can’t tell what’s actually driving the predictions. I’m not even sure we can put Prophet through SHAP.
I’ve also found that Prophet isn’t so straightforward to tune. Maybe it’s because I’m not an advanced user of the package, or maybe it’s because of the roundabout way in which you have to tune parameters. This is certainly not the case with the LASSO model.
While the LASSO approach arguably represents an improvement in performance and interpretability, perhaps what we really need is to use both approaches: one as a litmus test for the other. For example, if a “naive” Prophet model produces sensible forecasts, it might be reasonable to replicate the LASSO approach (the “False Prophet”) to maximise performance.