FlowStone

Asset-first orchestration for the BEAM.

FlowStone is an orchestration library for Elixir that treats assets (data artifacts) as the primary abstraction. Execution order is derived from explicit dependencies, and materializations can be tracked and queried via PostgreSQL when enabled.

Status

FlowStone is in alpha. Core execution, persistence primitives, and safety hardening are implemented, but some higher-level platform features (e.g., a bundled UI, fully resumable approvals, richer DSL surface) are still evolving.

Installation

Add to your mix.exs:

def deps do
  [
    {:flowstone, "~> 0.2.0"}
  ]
end

Quick Start

Define a pipeline:

defmodule MyApp.Pipeline do
  use FlowStone.Pipeline

  asset :raw do
    execute fn _context, _deps -> {:ok, :raw} end
  end

  asset :report do
    depends_on [:raw]
    execute fn _context, %{raw: raw} -> {:ok, {:report, raw}} end
  end
end

Register and materialize (using the in-memory IO manager):

{:ok, _} = FlowStone.Registry.start_link(name: MyRegistry)
{:ok, _} = FlowStone.IO.Memory.start_link(name: MyMemory)

FlowStone.register(MyApp.Pipeline, registry: MyRegistry)

io = [config: %{agent: MyMemory}] # uses default :memory IO manager

FlowStone.materialize(:report,
  partition: ~D[2025-01-15],
  registry: MyRegistry,
  io: io,
  resource_server: nil
)

{:ok, {:report, :raw}} = FlowStone.IO.load(:report, ~D[2025-01-15], io)

Materialize an asset and its dependencies:

FlowStone.materialize_all(:report,
  partition: ~D[2025-01-15],
  registry: MyRegistry,
  io: io,
  resource_server: nil
)

Backfill across partitions:

FlowStone.backfill(:report,
  partitions: Date.range(~D[2025-01-01], ~D[2025-01-07]),
  registry: MyRegistry,
  io: io,
  resource_server: nil,
  max_parallel: 4
)

Execution Model

• If Oban is running, FlowStone.materialize/2 enqueues a job and returns {:ok, %Oban.Job{}}.
• If Oban is not running, FlowStone.materialize/2 executes synchronously and returns :ok or {:error, reason}.

Oban job args are JSON-safe. Non-JSON runtime wiring (servers, IO keyword options, etc.) is stored in FlowStone.RunConfig keyed by run_id and is resolved by the worker with application-config fallback.

Documentation

• Design overview: docs/design/OVERVIEW.md
• ADR index: docs/adr/README.md
• 2025-12-14 review notes: docs/20251214/review/README.md

Contributing

Contributions are welcome. Please read the ADRs first to understand the current decisions and constraints.

License

MIT