When using PySpark, especially if you have a background in SQL, one of the first things you’ll want to do is get the data you want to process into a DataFrame. Once the data is in a DataFrame, it’s easy to create a temporary view (or permanent table) from the DataFrame. At that stage, all of PySpark SQL’s rich set of operations becomes available for you to use to further explore and process the data.
Since many standard SQL skills are easily transferable to PySpark SQL, it’s crucial to prepare your data for direct use with PySpark SQL as early as possible in your processing pipeline. Doing this should be a top priority for efficient data handling and analysis.
You don’t have to do this of course, as anything you can do with PySpark SQL on views or tables can be done directly on DataFrames too using the API. But as someone who is far more comfortable using SQL than the DataFrame API, my goto process when using Spark has always been,
input data -> DataFrame-> temporary view-> SQL processing
To help you with this process, this article will discuss the first part of this pipeline, i.e. getting your data into DataFrames, by showcasing four of…