Run Lifecycle
How workflow runs are queued, executed, retried, cancelled, and debugged — including concurrency controls, error handling patterns, and the full trace schema.
Run States
| Status | Terminal | Description | Transitions to |
|---|---|---|---|
| queued | No | Run is waiting for an executor to pick it up. No nodes have run yet. | running, cancelled |
| running | No | At least one node is currently executing. | completed, failed, paused, cancelled |
| paused | No | Execution is suspended, waiting for a human approval or external signal. | running (on resume), cancelled, failed (on timeout) |
| completed | Yes | All nodes finished without error. Output captured. | — |
| failed | Yes | A node threw an error and the run's onError policy resulted in a fatal stop. | queued (via retry API) |
| cancelled | Yes | Run was cancelled by a user or the cancel API before it completed. | — |
Execution Model
FlowOS runs workflows on an async worker pool. Each run is processed by a single worker. Nodes execute sequentially by default; parallel nodes split into concurrent worker tasks that merge at a designated merge node.
Execution order
- •Trigger fires → run record created with status
queued. - •Worker picks up the run → status →
running. - •Nodes execute in topological order (breadth-first from the trigger node).
- •Each node writes its output to the run trace before the next node starts.
- •After the last node completes, status →
completedand final output is captured. - •If any node throws and
onError=throw, the run immediately →failed.
Node execution record
Each executed node produces a trace entry in run.trace:
{
"nodeId": "send-slack",
"nodeName": "Notify Slack",
"nodeType": "action",
"status": "completed", // completed | failed | skipped | paused
"attempt": 1,
"input": { "channel": "#incidents", "text": "[P1] API latency spike" },
"output": { "messageId": "msg_ABC123", "timestamp": "1717300000.000123" },
"error": null,
"startedAt": "2026-06-01T10:00:00.100Z",
"completedAt": "2026-06-01T10:00:00.412Z",
"durationMs": 312,
"logs": [
{ "level": "info", "message": "Message delivered to #incidents", "ts": "..." }
]
}Concurrency Controls
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| maxConcurrentRuns | integer | optional | unlimited | Maximum number of simultaneous runs of this workflow. Additional triggers are queued. |
| queueBehavior | enum | optional | queue | queue — wait for slot. drop — discard trigger if at limit. replace — cancel oldest running run. |
| singletonKey | string | optional | — | Template expression evaluated per trigger. Only one run with a given key value may be active at a time. e.g. "{{trigger.record.id}}" prevents duplicate runs per record. |
| rateLimit | object | optional | — | { max: N, window: "1m" | "1h" }. Maximum trigger fires within a time window. |
// Prevent duplicate runs for the same incident
{
"trigger": { "type": "itsm_event", "event": "incident.created" },
"concurrency": {
"maxConcurrentRuns": 50,
"singletonKey": "{{trigger.event.resource.id}}",
"queueBehavior": "drop"
}
}Error Handling Patterns
Node-level: onError
Each node has an onError prop that controls what happens when it fails:
- •
throw(default) — Node failure immediately fails the entire run. - •
continue— Log the error to the run trace, set node status tofailed, and continue to the next node. Downstream nodes receivenullfor this node's output. - •
retry— Retry the node according to itsretryPolicybefore failing.
Workflow-level: error_handler node
Connect an error_handler node with an onError edge to create a catch path for a group of nodes:
{
"nodes": [
{ "id": "call-jira", "type": "action", "config": { "action": "jira.createIssue", "..." }, "onError": "continue" },
{ "id": "catch-errors", "type": "error_handler", "config": { "errorVariable": "jiraErr" } },
{ "id": "log-failure", "type": "action", "config": { "action": "slack.postMessage",
"input": { "channel": "#alerts", "text": "Jira integration failed: {{vars.jiraErr.message}}" }
}}
],
"edges": [
{ "from": "call-jira", "to": "catch-errors", "type": "onError" },
{ "from": "catch-errors", "to": "log-failure" }
]
}Global run-level: retryPolicy
Set on the workflow (not a node) to retry the entire run from the beginning on failure:
{
"retryPolicy": {
"maxAttempts": 3,
"strategy": "exponential",
"delayMs": 5000,
"maxDelayMs": 60000,
"retryOn": ["NETWORK_ERROR", "TIMEOUT", "RATE_LIMITED"]
}
}Timeouts
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| timeout (workflow) | integer | optional | 1800000 | Total run timeout in ms. Default 30 minutes. Max 24 hours. Run fails with TIMEOUT error if exceeded. |
| timeout (node) | integer | optional | 30000 | Per-node timeout. Default 30s. Max 5 minutes for regular nodes, 24h for delay/approval nodes. |
Manual Run Operations
/api/workflows/:id/triggerStart a new run (manual trigger, always async)
/api/workflows/:id/executions/:execId/cancelCancel a running execution
/api/workflows/:id/executions/:execId/replayRe-run from the beginning with the same trigger payload
/api/workflows/:id/executions/:execId/resumeResume a paused run (after approval)
/api/workflows/:id/executions/:execIdGet full execution detail with node trace
/api/workflows/executions/:executionId/streamServer-Sent Events stream of live node execution events
Manual trigger
POST /api/workflows/wf_01HZ.../trigger
{
"payload": { "incidentId": "inc_01HZ..." }
}
// Response (202 Accepted — trigger is always asynchronous, never blocks for output)
{
"success": true,
"data": { "executionId": "run_01HZ..." }
}Live Execution Streaming
For long-running workflows, stream node execution events in real time using Server-Sent Events:
const es = new EventSource(
'https://acme.flowos.io/api/workflows/executions/run_01HZ.../stream',
{ headers: { Authorization: 'Bearer ' + token } }
)
es.addEventListener('node.started', (event) => {
const entry = JSON.parse(event.data)
console.log('[' + entry.nodeId + '] started')
})
es.addEventListener('node.completed', (event) => {
const entry = JSON.parse(event.data)
console.log('[' + entry.nodeId + '] completed in ' + entry.durationMs + 'ms')
})
es.addEventListener('node.failed', (event) => {
const entry = JSON.parse(event.data)
console.log('[' + entry.nodeId + '] failed: ' + entry.error)
})
es.addEventListener('execution.done', () => { es.close() })Run History & Retention
Run records and traces are retained for:
- •Free/Starter plans — 7 days
- •Professional plans — 30 days
- •Enterprise plans — 90 days (configurable up to 1 year)
After the retention period, runs are purged automatically. Export run data before expiry using the Analytics export API if long-term retention is needed.