As a data scientist with a software engineering background doing various kinds of development on the side, I have often found myself wearing multiple hats and, at times, being the bridge between engineers and data scientists.
I would be lying to deny any friction between the two.
In this blog post, I will dive into my conflicting feelings as a data scientist and against data scientists. By the end of it, you should be taking away:
- 5 Realisations about Data Science,
- 15 Lessons on how to gear a well-functioning Data Science team and
- A bit of my dry humour.
Realisation 1: Data Science is all about the Hypothesis
The big tech companies and the get-rich-quick with data science video on YouTube have painted a rosy picture of data science. But take this:
Data science is more than modelling or fiddling with hyperparameters & model architectures.
Remember the scientific method that we learned in high school? That’s what science stands for in Data Science.
Data scientists look at a data set and the business problem, and we draft experiment plans for achieving the objective. Many of us might get into the fallacy of applying all sorts of models to find an appropriate one. Not only is it not elegant and inefficient, but when we find a model with decent performance metrics, we will scramble to find why the model worked.
If that does not sound too bad, think of this: how do you explain the spike in Bitcoin’s price to over 50,000 in Feb 2024? The number could have been perfectly fitted with a support and a resistance level. Are the two arbitrarily drawn lines now the driving factor behind Bitcoin’s movement? Or is there unexplained market psychology that we have ignored because we didn’t start from a…