The Microservice vs. Monolith debate rages in software, but is reduced to a gentle simmer in the data world
I don’t need to spend a lot of words convincing you that choosing tools in the data space is hard. There are hundreds, if not thousands of ways to skin this cat.
Something people overlook is how architecture impacts these decisions.
About 20 years ago, applications lived on computers owned by the companies that needed them — this is called “on-premise”. Owning these computers is an architectural decision. Consequently, cloud software vendors did not exist, since there was no demand for cloud software due to its inherent incompatibility with the architecture of the time.
Fast forward to 2024, and it’s the opposite — most people are fully in the cloud. However, some of us are still running our own servers. Others have hybrid models. This means understanding the impact your architecture has on the solutions you choose is more important than ever, and in this article we’ll dive into how a microservices vs. monolithic approach to data architecture impacts the tools you buy.
Something happening in Data is a re-ignition of the microservice vs. monolith debate. To understand what this is:
A monolithic application is built as a single unified unit while a microservices architecture is a collection of smaller, independently deployable services.
Within Data, monolithic applications were fairly common 10 years ago. An example of this might be a large Airflow repository, containing you data ingestion code, data transformation code, and business automations.
Business automations could include things like refreshing dashboards, sending reports, or sending alerts when tasks failed.
By contrast — the explosion of the cloud, partly fueled by VC funds (see this report), has given rise to data architectures that closely resemble microservices. Microservices include applications for managing batch data movement, data transformation and warehouse…