Moving data around can be slow. Here’s how you can squeeze every bit of performance optimization out of Python.
Python is often criticized for being among the slowest programming languages. While that claim does hold some weight, it’s vital to point out that Python is often the first programming language newcomers learn. Hence, most of the code is highly unoptimized.
But Python does have a couple of tricks up its sleeve. Taking advantage of concurrent function execution is stupidly simple to implement, yet it can reduce the runtime of your data pipelines tenfold. Instead of hours, it’ll take minutes. All for free.
Today you’ll see how concurrency works in Python, and you’ll also learn how to deal with exception handling, custom callbacks, and rate limiting. Let’s dig in!
The first order of business is configuring a data source. I wanted to avoid something proprietary and something that would take ages to set up. JSON Placeholder — a free REST API service — is the perfect candidate.