Problems in data collection can cause missing data. This issue can arise due to various reasons, such as sensor maintenance or transmission failure.
Missing data is usually solved by data imputation strategies, such as replacing the missing value with a central statistic. For time series, the imputation process is more challenging because the observations are ordered. Besides that, it may be useful to choose a strategy that considers the mechanism that causes missing data.
In this article, you’ll learn the main patterns of time series missing data, and how to deal with them.
The patterns of missing data, such as their frequency, depend on the mechanism that causes the missingness.
Generally, the cause of missing data falls into one of the following categories:
- Missing Completely at Random: When there’s no systematic process that causes an observation to miss. So, the missingness is unrelated to both 1) the value of the observation and 2) the past or future values and whether these are also missing. Many examples fall into this category, such as random sensor malfunctions or data corruption during transmission.
- Missing at Random: When the missing value is related to other values in the series, though not related to the value itself (i.e. if it’s high or low). An example is when equipment is shut down for maintenance, so the sensor stops transmitting data for some period that spans several observations.
- Not Missing at Random: The missing observation depends on its value and it can also depend on other variables or observations. For example, a temperature sensor fails during periods of extreme heat conditions.
Understanding the mechanism that causes missing data can help you choose an appropriate imputation strategy. This can improve the robustness of models and analyses.