☰
Navigation :

Data table

The data table structure is the smallest unit of the data API. Its purpose is to organize data of any supported kind into an annotated table and store all required metadata to recover the data from the file system.

A data table is located in a directory within the file system and contains following parts:

Element Description
data-table.json Metadata file for this data table.
<data row index> Stores the main data for this row. Names have following RegEx format: \\d+ (e.g., 0, 1, 52). Number corresponds to the row in the data table.
data-annotations/<data row index>/<data annotation column name> Used to store the data annotation <data annotation column name> for the table row <data row index>. <data annotation column name> directories contain data standard. Please note that data annotation names are not necessarily file system compatible. Use string manipulation methods to generate approximations if required!

Here you see an example of such a folder that stores three data rows. Here we assume that the stored data types are imagej-imgplus (ImageJ images).

graph LR; Root["/"] --> DataTable["data-table.json"]; Root --> R0["0/"]; Root --> R1["1/"]; Root --> R2["2/"]; R0 --> R0Img["data.tif"]; R1 --> R1Img["data.tif"]; R2 --> R2Img["data.tif"]; Root --> DataAnnotations["data-annotations"]; DataAnnotations --> DR0["0/"]; DataAnnotations --> DR1["1/"]; DataAnnotations --> DR2["2/"]; DR0 --> Label0["Label"]; DR1 --> Label1["Label"]; DR2 --> Label2["Label"]; Label0 --> Label0Img["data.tif"]; Label1 --> Label1Img["data.tif"]; Label2 --> Label2Img["data.tif"];

data-table.json

Here you will find the structure of the data-table.json file. You also can access the document in JSON Schema format: data-table.schema.json. Please note that the description for a data table row (data-table-row.schema.json) can be found below.

Here is an example data table that stores FFT images:

{
  "node-id" : "ij1-fft-forward2d",
  "slot" : "Output",
  "internal-path" : "/fastdata/projects/JIPipe/Output",
  "rows" : [ {
    "index" : 0,
    "annotations" : [ ],
    "data-annotations": [ ],
    "true-data-type" : "imagej-imgplus-fft-2d"
  } ],
  "data-type" : "imagej-imgplus-fft-2d"
}

Data table

Defines a data table row

node-id: string

The node type ID that generated the data. Optional.

slot: string

The slot name that generated the data. Optional.

rows: object[]

The data row items

See data-table-row
data-type: string

The data type ID behind the row

Data table row

Defines a data table row

id: number

Row index within the data table

annotations: object[]

Annotations attached to the data row

See annotation
data-annotations: object[]

Data annotations attached to the data row

See data-annotation
true-data-type: string

The true (not the slot) data type ID behind the row

Annotation

An annotation

name: string

Name of the annotation

value: string

Value of the annotation

Data annotation

A data annotation

name: string

Name of the annotation

true-data-type: string

Data type ID of the stored data annotation

row-storage-folder: string

Relative path to the current data table containing the data annotation data row