📝

Form Bio Workflow Development Guide

Source code repository requirements

Create correct structure

  1. Ensure main.nf and nextflow.config live in the root of the repo.
  2. Create a “workflows” directory inside the repo, if one does not already exist.
  3. Inside that directory, create another directory with same name as the ID of the workflow (if one does not already exist).
  4. Inside that directory, create a workflow.json file (the GUI input specification) and optional documentation files: overview.md citations.md inputs.md ,outputs.md .
    1. You can symlink any documentation that already exists in the repo here under the above names, or create these docs from scratch.
    2. The workflow.json schema is covered in detail below.

Convert json schema

Convert the repo’s existing json schema to a Form Bio workflow schema in workflow.json, using our Workflow JSON Schema Guide (below) to help you.

There is a script inside our workflow-schema repo that will help with this process if you have node installed on your machine. To use it:

  1. Clone the repo to your local machine
  2. Cd into repo → workflow-schema/
  3. Run sh scripts/json-schemaToV3.sh <path to json-schema file>
  4. This will output a workflow.json in our format inside the json-schema file’s parent directory
    1. It is important to then go through the output and fill in missing fields, like id, and fix any field types that the conversion script may have missed, and verify everything looks okay
    2. It can then be moved to the correct workflows/[id] directory

Some important notes:

  1. Ensure the value you give the id property matches the parent directory’s name
  2. If the schema includes a field for specifying an output directory, make sure to hide it
    1. Keep the id of the output param in the back of your mind for future steps

Update nextflow.config

Beneath the section where default params are defined, include statement that remaps our output param to theirs:

params.out_dir = "${params.output}" where “out_dir” is their param name

Import Workflow

Our GitHub app allows for fast, observable automation driven by pushes to workflow repositories on GitHub. Once a workflow is imported, any push to that repo will trigger an upload of the workflow under the version of the branch pushed to.

Importing a new or existing workflow

  1. Inside the Form Bio web app, navigate to the Manage section under Workflows.
  2. Select Create New.
  3. If you have not linked your GitHub account to the FormBio GitHub App yet, please click Sign in with GitHub and Authorize access.
  4. Once authorized, you will be redirected to the Web App and see an Import Workflow from GitHub screen.
  5. Select formbio under Select Account.
  6. Select the repository containing the workflow you would like to import under Select Repository
    1. If you do not see the repository you are looking for you will need to install the App to it
    2. To do this, open up the Select Account dropdown again and select Add or Modify Installations
    3. From there, select formbio organization and select the repositories you would like to install the app onto.
    4. Click Update Access
    5. You will be redirected back to the Web App where you can now select that repository
  7. The Select Workflow list should automatically detect the workflows located within the repo’s workflows/ directory that contain a workflow.json for you.
  8. Click Configure on the workflow you’d like to import.
  9. The configure form should be pre-populated with the default values and paths. It is strongly discouraged to change any of these at this time.
  10. Click Import Workflow and you’ll be redirected to the main branch’s build in progress. (See below, Monitor step 6).

Monitor

  1. Inside the Form Bio web app, navigate to the Manage section under Workflows (See first step in Import).
  2. You’ll see a list of cards representing all workflows that have been uploaded to the current project (whether via CLI or GitHub App) listed here.
  3. Workflows imported via GitHub App will be indicated by a GH logo and a View Deployments button.
  4. Click View Deployments to see a record of all uploads since being imported. To see logs for a particular upload, click View Logs.
  5. You’ll be taken to a logs view, displaying data about the workflow, version, and build status.
  6. If an upload was successful, you’ll see a Go to Launch button that will take you to the docs page for that version, where you can Launch the workflow.

Workflow JSON schema

Form Bio schemas currently support the configuration of workflows as JSON. They define metadata for the workflow, the necessary and optional parameters needed for a successful run, how those parameters should be presented in forms in the web application, as well as conditionals that allow the schema to dynamically respond to certain input values being provided by users.

Here is a simple example schema:

{
	"id": "myWorkflowSchema",
	"schema": "v3",
	"displayName": "My Workflow Schema",
	"description": "Here is my workflow schema",
	"title": "formbio/nf-myWorkflow",
	"workflowVersion": "v1.0.0",
	"categories": ["My Workflow Category"],
	"inputs": {
		"myFirstTextInput": {
			"title": "My First Text Input",
			"type": "string",
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": "my default value",
			"pattern": "someRegexPatternItMustMatch"
		},
		"mySecondTextInput": {
			"title": "My Second Text Input",
			"type": "string",
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": "my default value",
			"pattern": "someRegexPatternItMustMatch"
		}
	},
	"ui": {
		"inputs": [
			{
				"id": "myGroup",
			  "title": "My Group",
				"fields": ["myFirstTextInput", "mySecondTextInput"],
			  "help_text": "More information on this grouping...",
			  "description": "Here is a description",
			  "hidden": false
			}
		]
	},
}

id

  • A unique id for identifying this workflow.

schema

  • A pointer to the schema version, should be v3.

displayName

  • A user-friendly name displayed in the UI for this workflow.

title

  • A pointer to the repo name this workflow lives inside.

workflowVersion

  • The version of this schema to be uploaded and selected in the UI. Should be incremented on any changes.

categories

  • An array of categories this workflow belongs to. These are shown in the GUI workflow “Launch” page.
  • Categories given here do not have to match an existing category.

description

  • A description of this workflow to be explained to users in the UI.

inputs

  • See “Inputs” (below)

ui

  • See “UI” (below)
  • Note that all workflow field inputs must be nested inside a Group.
  • Note that workflows do not have outputs so only inputs is supported inside ui.

Inputs

Field Inputs

Field inputs represent a run parameter whose value can be provided by a user. The Field Input configuration defines the type of input to display in the form in the web application, as well as labels, help texts, default values, and various forms of validation, depending on the field.

Every Field Input contains the following properties: title, hidden, required, description, and help_text

  • description should be a concise description of parameter as it will be displayed immediately below the input in the form. These can be dynamically updated using conditionals.
  • help_text can be more verbose explanations of the parameters as these will be displayed in a collapsable help drawer to the right of the form. These cannot be dynamically updated using conditionals, so it maybe helpful to provide insight here as to why an input might change.

Here is a list of supported Field Inputs and their run parameters:

Text Input
"myTextInput": {
			// required properties
			"title": "My Text Input",
			"type": "string",
			// optional properties
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": "my default value",
			"pattern": "someRegexPatternItMustMatch"
}

Integer Input
"myIntegerInput": {
			// required properties
			"title": "My Integer Input",
			"type": "integer",
			// optional properties
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": 20,
			"minimum": 0,
			"maximum": 100
}

Text Area Input
"myTextAreaInput": {
			// required properties
			"title": "My Text Area Input",
			"type": "string",
			"format": "textarea",
			// optional properties
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": "my default value",
			"pattern": "someRegexPatternItMustMatch"
}

Dropdown Input
"myDropdownInput": {
			// required properties
			"title": "My Dropdown Input",
			"type": "string",
			"format": "enum",
			"enum": [
						{ "id": "myDefaultValue", "name": "My Default Value" },
						{ "id": "otherValue", "name": "Other Value" }
			],
			// optional properties
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": "myDefaultValue"
}

Multi-select Input

This field type allows selecting multiple options, and submits a comma-separated string to the Nextflow workflow.

"myMultiselectInput": {
			// required properties
			"title": "My Multiselect Input",
			"type": "array",
			"uniqueItems": true,
			"items": [
						{ "id": "myDefaultValue", "name": "My Default Value" },
						{ "id": "otherValue", "name": "Other Value" }
			],
			// optional properties
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": ["myDefaultValue"]
}

Radio Input

This input type is similar to Dropdown, but better if there are only a few options.

"myRadioInput": {
			// required properties
			"title": "My Radio Input",
			"type": "string",
			"format": "radio",
			"enum": [
						{ id: "myDefaultValue", name: "My Default Value" },
						{ id: "otherValue", name: "Other Value" }
			],
			// optional properties
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": "myDefaultValue"
}

Checkbox Input
"myCheckboxInput": {
			// required properties
			"title": "My Checkbox Input",
			"type": "boolean",
			// optional properties
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": true
}

Range Input

This submits comma-separated string of numbers to the Nextflow workflow.

"myRangeInput": {
			// required properties
			"title": "My Range Input",
			"type": "array",
			"format": "range",
			"min": 0,
			"max": 100,
			// optional properties
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": { "min": 10, "max": 90 }
}

File Input
"myFileInput": {
			// required properties
			"title": "My File Input",
			"type": "string",
			"format": "file",
			// optional properties
			"filetype": ".txt|.tsv",
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": "path/to/file"
}

Directory Input

If filetype is specified, only directories containing a file with the specified extension(s) are allowed.

"myDirectoryInput": {
			// required properties
			"title": "My Directory Input",
			"type": "string",
			"format": "dir-path",
			// optional properties
			"filetype": ".txt",
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": "path/to/directory"
}

Spreadsheet Input
"mySpreadsheetInput": {
			// required properties
			"title": "My Spreadsheet Input",
			"type": "string",
			"format": "spreadsheet",
			"columnConfig": {
						".txt": {
									"staticColumns": false,
									"columns": [
												{
															// required properties
															"name": "RunID",
															// optional properties
															"required": true,
															"generateFromData": true,
															"generateConfig": {
																		// required properties
																		"dataName": "someInputDirectoryId",
																		// optional properties
																		"delimiterPattern": "_R1|_R2|_1|_2|\\\\.R"
															}
												},
												{
			// required properties
			"name": "SampleID",
			// optional properties
			"pattern": "^\\\\d|\\\\W",
			"prependIfNumber": "S",
			"error": "Sample name must be provided and cannot contain spaces, special characters or start with numbers",
			"required": true
			},
													{
			// required properties
			"name": "SampleGroup",
			// optional properties
			"pattern": "\\\\W",
			"required": true,
			"error": "Group must be provided, cannot contain spaces or special characters"
			}
										]
						}
			}
			// optional properties
			"filetype": ".txt",
			"hidden": false,
			"required": true,
			"description": "Here is a description",
			"help_text": "Here are more details....",
			"default": "path/to/directory"
}

UI