Graph partitions

JIPipe allows to partition the workflow into units that are executed one-after-the other. The feature should not be mixed up with graph compartments that instead separate the pipeline visually

The functionality behind partitions can be explained easier by an example:

Assuming there is a pipeline that has to process hundreds or thousands of images, JIPipe's default behaviour of executing all iteration steps of a node in one go before continuing with the next operation. This would require that all the images would be stored in memory, thus likely crashing the workflow.

Partitions allow the creation of loops by functionally splitting a pipeline into parts that are executed in a standard way and parts that are repeated per iteration step.

In the example above we have four partitions:

1. One containing all operations related to handling files,

2. Another partition that is repeated per file,

3. A partition that receives all data from the looped partition and is not looped anymore,

4. and finally a partition that receives all generated statistics and processes them.

As you can see, only the "Default" partition is set to loop.

When JIPipe runs a workflow with multiple partitions, it splits the pipeline by the partition borders, meaning that you can re-use partitions. Then the software will determine the order of the partitions based on inter-dependence and execute the partitions in the correct order.

In the provided example, JIPipe would first execute all nodes in "File system", then proceed to loop over "Default", and pass all the collected results to the visualization and statistics nodes.

Assigning partitions

To assign nodes to a different partition, right-click a node or selection and choose an option in the Move to partition ... menu.

Creating loops

To create a looped partition, assign a set of connected nodes to the partition that should be looped.

The partition then needs to be configured to operate as a loop by right-clicking a node assigned to the looped partition and selecting the Edit partition button.

This will open a dialog where the Iteration mode can be set from Passthrough to Loop (single data per slot) or Loop (multiple data per slot).

Recommended structure

Due to the data-oriented workflow of JIPipe, loops can be tricky to set up in more complex scenarios that would yield unexpected results. Additionally, as of now, there is not yet an input manager available for debugging looped graph partitions. We highly recommend to structure the pipeline using Check iteration steps nodes that would act as single entry point into a looped partition.

For Loop (single data per slot) partitions, add a Check iteration steps (one data per slot) before all data that is passed into the loop and ensure that this interface is the only entry point. Then you can use the input manager of that node to check the iteration steps. Also, please do not forget to assign the interface to the previous partition.