Observability

Traces, metrics, and logs for a iii system through the iii-observability worker.

Every call in a iii system passes through the engine: function invocations, trigger firings, and channel messages all flow across it. Because every worker communicates through the engine, it can trace work as it moves from one worker to the next, giving you observability across the whole system without each worker instrumenting itself. The automatically supplied iii-observability worker turns that information into OpenTelemetry traces, metrics, and logs. Do not declare it in either config.yaml or worker-compose.yaml.

This page is a quick tour. For the complete configuration and the full list of query functions, see the iii-observability worker docs.

OpenTelemetry support

iii-observability is OpenTelemetry-based: it produces distributed traces across worker hops, metrics, and structured logs, and it can export them to any OTel-compatible backend or keep them in-memory for local development. Sampling, retention, and exporter targets are configured on the worker; see the worker docs for the full set of options.

Logging

Worker code emits structured logs through the Logger from the observability SDK, which routes them into the same OpenTelemetry pipeline (so log lines correlate with the trace they happened in) rather than to raw stdout.

Node / TypeScript

import { Logger } from "@iii-dev/helpers/observability";

const logger = new Logger();

// each level takes a message and optional structured data
logger.debug("cache lookup", { key });
logger.info("processing link", { slug });
logger.warn("retrying upstream", { attempt });
logger.error("failed to persist", { slug, err });

Python

from iii_helpers.observability import Logger

logger = Logger()

# each level takes a message and optional structured data
logger.debug("cache lookup", {"key": key})
logger.info("processing link", {"slug": slug})
logger.warn("retrying upstream", {"attempt": attempt})
logger.error("failed to persist", {"slug": slug, "err": str(err)})

Rust

use iii_helpers::observability::Logger;
use serde_json::json;

let logger = Logger::new();

// each level takes a message and optional structured data
logger.debug("cache lookup", Some(json!({ "key": key })));
logger.info("processing link", Some(json!({ "slug": slug })));
logger.warn("retrying upstream", Some(json!({ "attempt": attempt })));
logger.error("failed to persist", Some(json!({ "slug": slug, "err": err.to_string() })));

You can also query recorded logs and emit one-off log lines through the observability worker’s triggers, for example:

# emit a log line from the CLI
iii trigger engine::log::info --json '{"message":"hello from the CLI"}'

# find it in the stored logs
iii trigger engine::logs::list | grep -C 10 hello

Traces

A trace captures the spans of a call as it moves across workers, the headline view for debugging a request end to end. Traces come from worker function invocations: a worker records a span each time it handles a call. Built-in engine::* functions are not traced by default (set III_OTEL_TRACE_BUILTINS=true to include them), so call a worker function first to produce a trace. The sandbox worker’s sandbox::run is an easy one to start with:

worker-compose.yaml

engine:
  workers:
    iii-sandbox: {}

Start this file with iii compose --up before calling the sandbox function. iii-sandbox is an engine-owned exception, so it belongs under the top-level engine.workers, not under containers.

# run something to produce a trace
iii trigger sandbox::run image=python lang=python code='print(1)'

# count the recorded traces (workers export spans a moment after the call,
# so re-run this until it reports a non-zero count)
iii trigger engine::traces::list | jq '.total'

# grab the most recent trace id and print its span tree
TID=$(iii trigger engine::traces::list | jq -r '.traces[-1].trace_id')
iii trigger engine::traces::tree trace_id=$TID

engine::traces::tree needs a real trace_id and traces export on a short delay. If engine::traces::list still reports 0 or the call above to get a trace_id resolves to null and fails then wait a few seconds and try again.

React to logs and spans

The observability worker hosts two event sources you can observe: observability::on-log, one event per log record, and observability::on-span, the ids of the traces that recorded spans in each window of about 300 ms. They replace the log and trace trigger types, which keep working.

// invoke alerts::log for every ERROR log record
worker.observe(
  {
    function_id: "observability::on-log",
    deliver: "input",
    filters: { "input.severity_text": { $eq: "ERROR" } },
  },
  "alerts::log",
);

Each event runs its targets in a trace of its own, and log records and spans in that trace are not events again. A target that logs while it handles an event does not trigger itself: the SDK loggers record the trace the handler runs in, and so does engine::log::* when the call passes no trace_id.

Activity outside that trace is bounded by depth. A span the engine records for a call made inside a chain of observers carries an iii.observe_depth attribute (set when the depth is above 0). These events take that depth, so a target that logs or makes calls on every event stops at observers.max_observe_depth.

Observers have their own metrics (iii.observer.activations.total, iii.observer.dropped.total, iii.observers, and more), and calls a worker makes to its own functions are counted in iii.invocations.total with dispatch="local". See Engine-collected metrics.

Health and clearing

Check engine health, or clear stored observability data while developing:

# overall health status
iii trigger engine::health::check | jq -r .status

# clear stored logs
iii trigger engine::logs::clear

# the logs are now empty
iii trigger engine::logs::list

# clear stored traces
iii trigger engine::traces::clear

# the trace count is now zero
iii trigger engine::traces::list | jq '.total'

Logs populate quickly. It is possible to call engine::logs::clear immediately followed by engine::logs::list and still see new logs that were generated in between the two calls.

Seeing it in the console

The console renders this observability data visually, traces, logs, and metrics for a running system, so you usually do not query the functions above by hand during development.