Why Parallelize?¶
PlantCV's Parallel module allows for running a PlantCV workflow on many images across many cores. Sometimes this is called "multiprocessing" or "batch processing". If you have a high throughput image phenotyping facility or other hardware allowing you to take many photos then working one photo at a time through a Jupyter notebook may be prohibitively slow and would not leverage the benefits of PlantCV over non-scripting image analysis tools.
In addition to making it easier for users to analyze many images parallelization also aides in making workflows reproducible since there is no need to make sure that you remember to make the right edits each time you run through a jupyter notebook or change file paths in a python script.
Taking photos with the goal of parallel analysis¶
Ideally your images should fulfill a few criteria to make parallelization easier:
- Using a basically comparable imaging station helps make parallelization easier. If you can leave a camera in one place and put your plants in front of it at the same distance, with the same lighting, and a consistently placed color card (if you are using one) then there will be less room for things to require manual attention which would slow down your analysis.
- Naming your files with a consistent and meaningful schema will let tools like the
metadata_parserwork well to subset your image dataset and provide meaningful metadata in the outputs without you then having to look up barcodes/labels in downstream analysis. - Storing images in a consistent location with meaningful file paths will make using the right images easier. By default the a parallel configuration file will go into all the subdirectories after your
input_dirand find all image files in those subdirectories. That means you could name intermediate directories however you want to, but tools likeinspect_datasetor the regex/directory filters available inworkflowconfiggive you meaningful information and options.
While plantcv.parallel is a very flexible and useful tool able to work over highly variable datasets taking some time to think about how you are going to take, name, and store images can save you and collaborators a lot of work in image analysis.
Getting ready to parallelize¶
Before you can parallelize a workflow you need to make sure that it works on a single image. Generally we recommend prototyping a workflow in Jupyter notebooks so that you can run interactively. Most of the time a previous workflow of yours or one of our many tutorials will be a good place to start. Once your Jupyter notebook works on one image try it on a few more to make sure you've caught any obvious errors that could come up.
Running a parallel workflow will make some temporary files specified by config.tmp_dir including a directory called _PCV_PARALLEL_CHECKPOINT_, with nested directories named by timestamp. By default the cleanup=true field of the parallel configuration will remove the temporary directories (including _PCV_PARALLEL_CHECKPOINT_, so if you are interested in interim analyses you may decide not to cleanup) at the end of a successful parallel workflow.
From Jupyter to Parallel¶
There are a few things to consider moving from prototyping in Jupyter to running a potentially computationally heavy parallel job.
Another helpful tool for building a workflow with the goal of running analyses in parallel is to use WorkflowInputs to set things like file paths. There are many examples of this in the tutorials, but the general pattern working interactively in Jupyter would be:
# in your .ipynb notebook
from plantcv.parallel import WorkflowInputs
# Define workflow inputs
args = WorkflowInputs(
images=["path/to/your/image.file"],
names="image1",
result="example_results.json",
outdir=".",
debug="plot"
)
# using the args object to set preferences will make parallelization easier later
pcv.params.debug = args.debug
img, path, filename = pcv.readimage(filename=args.image1)
# ... the rest of your code
When you are ready to run in parallel whether you make a python script manually and run with workflow_inputs or if you run the Jupyter notebook in parallel with JupyterConfig you would need to change the above to:
# in your .py script
from plantcv.parallel import workflow_inputs
# Parse configuration arguments
args = workflow_inputs()
# these lines can stay the same thanks to workflow_inputs()
pcv.params.debug = args.debug
img, path, filename = pcv.readimage(filename=args.image1)
# ... the rest of your code
This change will allow the workflow to read the variety of image paths and associated metadata that are used when running in parallel.
It is also a good idea to avoid making your code do extra work in parallel. Things like plotting debug images or saving intermediate data can be helpful in development but problematic in parallel. We recommend setting plantcv.params.debug = None when working in parallel and commenting out any code that explicitly draws plots/images. If you use WorkflowInputs in your Jupyter notebook then when you switch to workflow_inputs for use in parallel args.debug will default to None.
Finally, for converting from a .ipynb notebook to a .py script you can copy and paste from cells or use nbconvert.
Where to Parallelize¶
PlantCV Parallel uses dask to schedule jobs. By default all jobs are run on the machine where they are submitted (config.cluster = "LocalCluster"). That will run config.cluster_config.n_workers cores on the local machine. You can also distribute jobs across many machines if you have a computational cluster available to use, see the WorkflowConfig docs for details.