Opinion
With pretrained AI models and LLM data analysis on the rise, unique data is the only thing that sets you apart
When I pivoted to a career in data science after completing my PhD in physics, I was fascinated by complex models and insightful data analysis.
Four years later, I know that these aspects of the job carry less and less weight.
The number one ingredient for successful AI applications is data. And the number one time sink in any of my projects is data processing.
Welcome to the real world
Introductory courses focus on model development and understanding the inner workings of training a neural network.
We learn to write our own training loop, choose the right validation metric, and understand the bias-variance trade-off.
Students work with readily available datasets like MNIST. Our courses gloss over the data aspect, and indeed for tutorials data processing is as simple as typing
from torchvision.datasets import MNIST
dataset = MNIST('./data', download=True)