To deploy a pipeline, typically there is a main script, or a single point of entry that runs the whole pipeline. For example, in a data science pipeline, the point of entry of the code repository should orchestrate and run the data, feature engineering, modeling, and evaluation pipeline in sequence.
Sometimes, you may need to run different types of pipelines or make ad-hoc tweaks to the pipeline.
Tweaks may include omitting certain parts of the code or running the pipeline with different parameters. In data science, there could be a training and scoring pipeline or certain runs that require a full or partial refresh of the data.
The trivial solution will be to create multiple main scripts. However, this will result in code duplication and it is hard to maintain the multiple scripts in the long run — given that there can be many combinations of tweaks. A better solution is to have the main script accept arguments, in the form of parameter values or flags, and subsequently run the appropriate type of pipeline via the Command Line Interface (CLI).
This article will not elaborate on how your main script decides to use the arguments but will introduce different ways you can pass in arguments to your main script — you can think of it as your main script is now a function that accepts parameters! I will also elaborate on the pros and cons of each method, and provide code samples of how to implement the basic to advanced usages.
- Using sysargv: The simplest way
- Using argparse: The most common way
- Using docopts: An alternative way
- Using Typer: The newest and easiest way
The simplest way to pass in arguments
Arguments can be passed in and read directly with sysargv, making it the simplest way to pass in multiple arguments.