VolcanicCore/docs/superpowers/specs/2026-08-28-volcaniccore-workflow-engine-design.md

VolcanicCore — N8N-style Workflow Engine Design

Status: Approved for planning Date: 2026-08-28

Purpose

VolcanicCore is a self-hosted workflow engine in the spirit of n8n. An operator draws a directed graph of nodes connected by edges in a browser canvas. Each workflow can be activated independently; when active, its trigger nodes subscribe to their sources (a NATS subject, an incoming HTTP webhook) and each event starts one execution.

The system is designed around two clean extension points — node types and connection providers — so that adding a new node kind (Slack, Postgres, S3) or a new connection kind (Kafka, Postgres, Redis) is a matter of dropping a new C# class in and rebuilding the container.

v1 nodes

• NATS Hook — trigger. Subscribes to a subject on a configured NATS connection; each message starts one execution.
• Webhook — trigger. Exposes POST /wh/<workflow-id>/<path>; the request body starts one execution. Threads an internal answerChannelId through the run.
• PythonTransform — action. Runs user Python 3 (IronPython) against the incoming items and returns transformed items.
• WebResponse — terminal action. Resolves the answerChannelId from a Webhook trigger, replies to the still-open HTTP request with the given body and status.
• NATS Publish — terminal action. Publishes the current item(s) to a subject on a NATS connection.

Non-goals (v1)

• Sub-workflows, error workflows, node-level retries.
• IF / switch / loop / wait / merge nodes. (Fan-in is designed for, but no merge node ships in v1.)
• Multiple users, RBAC. Single shared admin password only.
• Versioning of workflows, staged / draft edits, rollback.
• Workflow JSON import/export UI. (Graphs are persisted as JSON, but no dedicated importer.)
• Distributed / multi-worker execution.
• Clustering, HA, federation.

1. Architecture

Single Docker container, one ASP.NET Core 9 process, one SQLite file in /data. IronPython 3 runs in-process. No separate worker fleet.

┌─────────────────────────── Container ────────────────────────────┐
│  ASP.NET Core 9                                                  │
│  ├─ SPA (React + Vite + Tailwind + React Flow)   static wwwroot  │
│  ├─ REST + WebSocket   :8080 (behind Coolify/Traefik)            │
│  ├─ NodeRegistry (INodeType instances, self-registering)         │
│  ├─ ConnectionProviderRegistry (IConnectionProvider, ditto)      │
│  ├─ TriggerHost (activates/deactivates trigger nodes)            │
│  ├─ Executor (runs one workflow execution to completion)         │
│  ├─ WebhookRouter (holds open HTTP responses by answerChannelId) │
│  └─ SQLite  (workflows, executions, connections, admin config)   │
└──────────────────────────────────────────────────────────────────┘
        ▲
        │  :8080 → volcaniccore.<domain> via Coolify

Auth. Single admin password provided by env ADMIN_PASSWORD. Cookie session on successful login. If ADMIN_PASSWORD is unset at boot, the process refuses to start.

Connection secrets encryption. Env ENCRYPTION_KEY (base64-encoded 32 bytes) is the AES-GCM key for encrypting connection-provider config in SQLite. If unset at boot, a fresh key is generated and persisted to /data/.enc-key (owner-only read).

2. Modularity contracts

Two extension points, both discovered at startup via reflection over loaded assemblies.

2.1 INodeType

Every node kind implements this:

public interface INodeType {
    string Id { get; }                    // "nats.hook", "http.webhook", ...
    string DisplayName { get; }
    NodeCategory Category { get; }        // Trigger | Action | Terminal
    int InputCount { get; }               // 0 for triggers
    int OutputCount { get; }              // 0 for terminals
    string? ConnectionProviderId { get; } // e.g. "nats"; null if none
    JsonSchema ConfigSchema { get; }      // drives UI form generation
}

public interface ITriggerNode : INodeType {
    // Starts the trigger. Emitting an event on `sink` starts one execution.
    // Return value is disposed when the workflow is deactivated.
    Task<IAsyncDisposable> StartAsync(NodeContext ctx, TriggerSink sink, CancellationToken ct);
}

public interface IActionNode : INodeType {
    Task<IReadOnlyList<IReadOnlyList<Item>>> ExecuteAsync(
        NodeContext ctx,
        IReadOnlyList<IReadOnlyList<Item>> inputsByPort,   // indexed by input port
        CancellationToken ct);                             // return indexed by output port
}

NodeContext gives the node its config (deserialized from ConfigSchema), its resolved live connection (from the connection provider) if any, a logger scoped to the current execution, and the current answerChannelId if one is in scope.

Every action node in v1 has InputCount == 1 and OutputCount <= 1, so it always reads inputsByPort[0] and returns either [] (terminal) or [items] (single output). The per-port shape exists so that a future Merge node with InputCount == 2, or a future IF node with OutputCount == 2, drops in without breaking the executor or the storage format.

2.2 IConnectionProvider

One per connection type:

public interface IConnectionProvider {
    string Id { get; }                    // "nats"
    string DisplayName { get; }
    JsonSchema ConfigSchema { get; }      // fields, marking which are secret
    Task<object> OpenAsync(JsonElement config, CancellationToken ct);
    Task CloseAsync(object handle);
}

Nodes never construct raw NATS clients directly — they call NodeContext.GetConnection<T>() and get the pooled live instance the registry opened from the connection row. The registry keeps one live instance per connection row and closes it when the row's config changes or the row is deleted.

2.3 Adding new pieces

• New node = new class implementing IActionNode / ITriggerNode in VolcanicCore.Nodes (or an auxiliary assembly on the load path). Rebuild the container.
• New connection provider = one new class implementing IConnectionProvider in VolcanicCore.Providers. Rebuild the container.

Both surface in the UI automatically because the SPA fetches /api/node-types and /api/connection-providers at load and generates node-config and connection-config forms from ConfigSchema.

3. Data model between nodes

An execution passes lists of items between nodes:

public record Item(JsonElement Json, IReadOnlyDictionary<string,byte[]>? Binary = null);

Triggers produce the initial list (usually one item). Actions consume inputsByPort[i] and return one list per output port. Terminals return an empty outer list.

Fan-out. One output port with N downstream edges → each downstream node receives the same items and runs in parallel. Supported in v1.

Fan-in. More than one input edge feeding into the same input port of the same node → the node waits for all upstream branches to finish before running. Not shipped in v1 (no merge node in the 5), but the executor and storage are designed to support it — see §4.

4. Execution engine

One Executor per running execution, in-process, one C# Task. The scheduler is in-degree based from day 1 so fan-in drops in later without a redesign.

Trigger scoping. A workflow may declare zero or more trigger nodes. When the workflow is active, each trigger node subscribes/listens independently, and each event fires its own execution scoped to that one trigger. Activating a workflow with zero triggers is legal but has no effect on its own (the workflow can still be run manually via Test run).

Trigger fires → executor:

1. INSERT execution row with status = running, workflow_id, trigger_node_id, trigger, trigger_data.
2. Compute per-node in-degree from the workflow's edges (edges are { from_node, from_port, to_node, to_port }).
3. Record the trigger firing as a node_execution for the trigger node — inputs_json = [], outputs_json = [triggerItems], status = completed. Deliver triggerItems on the trigger's output edges.
4. Loop: pop any node that has received data on all its input ports and run it in a Task. On completion, look at its output edges; for each edge, deliver the corresponding output-port list to (to_node, to_port).
5. For each node run, INSERT node_execution with inputs_json, outputs_json, started_at, ended_at, status, error.
6. When no nodes are running and no nodes are ready, the execution ends. UPDATE execution.status, ended_at.

Failure semantics.

• A node throw is recorded on that node's node_execution row with status = failed.
• Failure propagates on the failed node's output edges. A downstream node ready-check treats an incoming "failure signal" the same as "delivered": once it has heard from all input ports and any of them signaled failure, the node is marked skipped_due_to_failure (without being run) and propagates failure on its own outputs. This makes any subgraph downstream of a failure unreachable, and — once merge nodes exist — makes a merge whose upstream branch failed itself fail.
• Whole execution status:
  • completed — every reachable node finished successfully.
  • partial — at least one branch failed, at least one succeeded.
  • failed — every branch failed, or the trigger itself could not produce items.

Concurrency. For v1, executions run with unbounded in-execution parallelism (fan-out arms run concurrently), but the count of concurrent executions across the process is capped by MAX_CONCURRENT_EXECUTIONS (default 32). If an execution can't start (webhook trigger, cap hit) it returns 503. NATS-trigger events that hit the cap wait on a bounded channel; if the channel fills, the oldest event is dropped and logged.

Crash safety. On startup, any execution.status == running is flipped to aborted. Best-effort recovery is out of scope for v1.

Retention. A background sweep every hour deletes execution rows (cascade to node_executions) older than EXECUTION_RETENTION_DAYS (default 7) OR beyond the last EXECUTION_KEEP_PER_WORKFLOW (default 500) per workflow, whichever is stricter.

Manual test run. UI button on the canvas → the same executor. A modal asks the user which node to start from (defaults to the sole trigger if there's exactly one; otherwise the user picks) and lets them paste JSON trigger_data. trigger is manual. Manual runs count against MAX_CONCURRENT_EXECUTIONS.

Single edge per input port (v1). The editor UI enforces that each input port accepts at most one incoming edge. True multi-edge fan-in requires a merge-node semantics decision (§10) and does not ship in v1.

5. Webhook + WebResponse mechanics

WebhookRouter is a singleton keyed by answerChannelId (a GUID) mapping to a TaskCompletionSource<WebResponsePayload> plus a CancellationTokenSource with a per-node timeout (default 30s, overridable per Webhook node config).

• Webhook trigger receives POST /wh/<workflow-id>/<path>. If the workflow is inactive or no Webhook node in it matches that path, the router returns 404. Otherwise it generates a fresh answerChannelId, registers it with the router, kicks off the execution, and awaits the TCS or the timeout.
• The answerChannelId lives on NodeContext as ambient per-execution state — it is not part of the item JSON. That keeps user Python transforms from needing to know about it.
• WebResponse node calls WebhookRouter.Resolve(answerChannelId, payload).
• Resolve is first-write-wins: if the router already has a result for that channel, the second call is a no-op and gets logged on the current node's node_execution. That way WebResponse can safely be a sibling of NATS Publishes in a fan-out — the first branch to reach WebResponse replies.
• If the execution finishes with no WebResponse call: 204 No Content.
• If the timeout fires first: 504 Gateway Timeout. The execution continues in the background but any later WebResponse call is a no-op.

6. Persistence (SQLite)

Single SQLite file at /data/volcaniccore.db.

admin_config       (key TEXT PRIMARY KEY, value TEXT)
connections        (id, provider_id, name, config_json_encrypted, created_at)
workflows          (id, name, graph_json, active BOOL, created_at, updated_at)
executions         (id, workflow_id, trigger, trigger_data_json,
                    status, started_at, ended_at, error)
node_executions    (id, execution_id, node_id, inputs_json, outputs_json,
                    status, started_at, ended_at, error)
audit_log          (id, at, actor, action, target_kind, target_id, detail_json)

• workflows.graph_json shape: { nodes: [{ id, type_id, name, position, config_json, connection_id? }], edges: [{ from_node, from_port, to_node, to_port }] }. Port indices are always present, even when v1 nodes only use port 0.
• node_executions.inputs_json is a JSON array of arrays — one entry per input port. In v1 always length 1.
• Migrations: single hand-rolled Migrator class, versioned via PRAGMA user_version, applied at boot.
• Encrypted columns use AES-GCM with ENCRYPTION_KEY. Nonce is prepended to the ciphertext.

7. Frontend

React 18 + Vite + TypeScript + Tailwind + React Flow + TanStack Query.

• Login — password prompt against POST /api/session.
• Workflows — list, create, delete, activate toggle.
• Editor — React Flow canvas. Left panel: node palette fetched from /api/node-types. Right panel: contextual config for the selected node (form auto-generated from ConfigSchema). Top-right actions: Test run, Activate, Save.
• Executions — per workflow, table of runs with status / duration; click a run → node-by-node view with per-port input and output JSON.
• Connections — list, create, delete. Provider picker → generated form. Secret fields stored write-only (never returned on GET).

8. Deployment

Coolify, single container. Multi-stage Dockerfile: pnpm build for the SPA into wwwroot, dotnet publish for the backend, final runtime image runs the .NET process. Data volume mounted at /data. Follows the preparing-dotnet-react-app-for-coolify skill for the compose file (named volume, expose: [8080], SERVICE_FQDN_APP substitution, no host ports).

Environment variables.

Var	Required	Default	Purpose
ADMIN_PASSWORD	yes	—	Admin login password
ENCRYPTION_KEY	no	auto-gen to /data/.enc-key	AES-GCM key for connection secrets
EXECUTION_RETENTION_DAYS	no	7	Sweep executions older than N days
EXECUTION_KEEP_PER_WORKFLOW	no	500	Also cap per-workflow row count
WEBHOOK_DEFAULT_TIMEOUT_SECONDS	no	30	Default per Webhook node
MAX_CONCURRENT_EXECUTIONS	no	32	Process-wide cap

9. Repository layout

VolcanicCore.sln
src/
  VolcanicCore.Api/          # ASP.NET Core host, endpoints, WebhookRouter
  VolcanicCore.Core/         # INodeType, IConnectionProvider, Executor, TriggerHost, Item
  VolcanicCore.Nodes/        # NatsHookNode, WebhookNode, PythonTransformNode,
                             # WebResponseNode, NatsPublishNode
  VolcanicCore.Providers/    # NatsConnectionProvider
  VolcanicCore.Persistence/  # SQLite, Migrator, repositories
frontend/                    # Vite React SPA
docs/superpowers/specs/…     # this design
docs/superpowers/plans/…     # implementation plans (next step)
Dockerfile
docker-compose.yml           # local dev + Coolify reference

10. Open decisions deferred to a later spec

• Merge-node semantics (concat vs zip vs first-only) — decide when adding the node.
• Retry / backoff policy at the node level.
• Multi-user + RBAC.
• Sub-workflows and error workflows.