Correlation has a somewhat bad reputation among data scientists. Every now and then, I read bombastic titles like “Correlation is dead”, “Goodbye correlation”, “Here is a replacement for correlation”, and so on.
But the truth is, correlation is still very much alive and thriving. This is because, in practice, it works extraordinarily well as a proxy for the strength of the relationship between two variables, and is hard to match in terms of simplicity.
That said, correlation does have a major drawback. As a univariate metric, it can’t account for the influence of other variables that might skew the measurement. This leads to the well-known saying in statistics: “Correlation is not causation.”
Luckily for us, there exists a generalized version— called partial correlation — that keeps all the advantages of simple correlation while addressing its main limitation.
Yet, surprisingly, partial correlation remains largely unknown. The proof of its lack of popularity is that it’s only implemented in one Python library, Pingouin — not exactly the go-to library for most data scientists.