Commands: source / filter / sink

Author

Affiliation

Paolo Bosetti

University of Trento

Abstract

mads source, mads filter, and mads sink are the three generic plugin-loader agents: one shared implementation, compiled three times, that dynamically loads a .plugin shared library and drives it with the role-appropriate main loop. This page covers all three together, since they differ only in which base-class methods they call and a handful of role-specific options.

Related guides:

• Plugins — writing the plugin itself
• OTA Plugins — distributing .plugin files via the broker
• Security in MADS (for --crypto)
• Auto-discovery (for --room)
• Local transport and shared memory (for --transport and shm)

Synopsis

mads source [common options] [-p|--period ms] [plugin]
mads filter [common options] [-p|--period ms] [-b|--dont-block] [plugin]
mads sink   [common options] [-b|--dont-block] [plugin]

Description

All three are the same executable (src/main/plugin_loader.cpp) compiled under a different #define, each linking a different base class:

flowchart LR
  src["plugin_loader.cpp
(one source file)"]
  src -- "PLUGIN_LOADER_SOURCE" --> S["mads-source
Source<json>"]
  src -- "PLUGIN_LOADER_FILTER" --> F["mads-filter
Filter<json,json>"]
  src -- "PLUGIN_LOADER_SINK" --> K["mads-sink
Sink<json>"]
  S --> P1["get_output(out, blob?)"]
  F --> P2["load_data(in, topic, blob?)
then process(out, blob?)"]
  K --> P3["load_data(in, topic, blob?)"]

Whichever role, the loader: parses CLI options, initializes the agent, loads the named .plugin shared library via the pugg kernel, checks its protocol version, calls set_params() once with the merged settings, then drives the plugin’s methods from the main loop.

Important

Every loader refuses to load a plugin whose protocol version is older than the loader’s own minimum (currently 7). Use mads plugin --update to migrate an older plugin forward.

Common options

These apply to all three:

Flag	Effect
plugin (positional)	Path or bare name of the .plugin file to load. See Plugin resolution below.
-n, --name	Agent name (default: the plugin’s file name without extension); selects the .ini section to read.
-i, --agent-id	Tag added to the payload, to distinguish agents sharing the same name/section.
-d, --delay ms	Wait this long before the first message (works around the ZeroMQ “slow joiner” problem).
-o, --option key=value	Override a plugin setting from the CLI (repeatable); the value is coerced to bool/int/float/string automatically.
--silent	Suppress the live status line.
-s, --settings URI	Settings file/URI (tcp://host:port, the broker’s local ipc://path, or a local .ini path).
--settings-timeout ms	Timeout when reading settings from a broker (0 = no timeout, the default).
--transport mode	auto, tcp or ipc; see Choosing the transport.
-r, --room[=name]	Discover the broker via UDP instead of --settings (opt-in; default room mads).
-S, --save-settings file	Dump the resolved settings to an ini file.
--crypto, --keys_dir, --key_broker, --key_client, --auth_verbose	CURVE encryption for the broker connection.
-v, --version / -h, --help	Version / usage.

Role-specific options

	source	filter	sink
Default plugin	publish.plugin	bridge.plugin	feedback.plugin
-p, --period ms	Yes — sampling period; default 0 = free run	Yes — only used together with --dont-block	—
-b, --dont-block	—	Don’t block waiting for input; ask the plugin for output on every loop tick regardless	Don’t block on read
Loop body	get_output() on every tick	load_data() (if a message arrived, or always with --dont-block) then process()	load_data() whenever a message arrives
Remote control	threaded	inline	inline

Note

A filter with --dont-block is how a plugin acts as both filter and source — e.g. reacting to incoming messages when present, but also producing output on a timer when idle.

Plugin resolution

Resolution order is the same for all three:

1. a plugin path given on the command line — used regardless of the ini file;
2. otherwise, an attachment key in the agent’s ini section — the broker sends this plugin file Over-The-Air (OTA) at connect time (see OTA Plugins);
3. otherwise, the built-in default plugin for that role (table above).

Tip

For the same plugin on multiple architectures, give each architecture its own ini section (e.g. [my_plugin_x86], [my_plugin_arm64]) with its own attachment, launch each with -n matching that section, and set the same pub_topic in both so downstream consumers see one logical source.

Choosing the transport

An agent reaches the broker either over tcp, at the [agents] frontend_address/backend_address, or over the broker’s local ipc:// endpoints (see Local transport). The choice is made once, at startup, from the transport setting:

Value	Behaviour
auto (default)	Use ipc when the broker runs on this host, tcp otherwise.
tcp	Always use the tcp addresses, as before ipc existed.
ipc	Use ipc, or refuse to start if it is not possible.

The --transport option wins over the transport key of the agent’s own section, which wins over the one in [agents]:

[agents]
# transport = "auto"

[camera]
transport = "tcp"   # this one always goes through tcp

With auto, the agent asks the broker which ipc endpoints it offers (the transport command) and switches to them only if all of these hold:

• the agent’s libzmq supports ipc;
• the broker offers both the frontend and the backend ipc endpoints;
• the broker runs on the same host: its host token matches the agent’s;
• the socket files are visible from the agent, which rules out containers that share a hostname but not a filesystem.

Otherwise it silently uses tcp. Settings fetched over ipc (-s ipc:///tmp/mads-1000/broker-9092-settings.ipc) prove the broker is local, so the host check is skipped, and the data sockets follow over ipc. The transport in use is shown in the agent’s startup information.

mads doctor predicts which transport an agent started on a given host would use.

Shared memory

A producer that publishes large binary blobs (camera frames, point clouds) can keep them in shared memory instead of copying them through ZeroMQ and the broker. Only a small descriptor travels on the bus; consumers on the same host read the bytes in place. Enable it in the producer’s section:

Key	Default	Meaning
shm	false	Publish blobs through a shared-memory ring owned by this agent.
shm_slots	8	Number of slots in the ring.
shm_slot_size	16777216 (16 MiB)	Size of one slot, in bytes. Larger blobs are sent inline.

[camera]
pub_topic = "frames"
shm = true
shm_slots = 8
shm_slot_size = 8388608

Consumers need no setting. A filter or sink on the same host reads shared blobs directly; agents that do not opt in to zero-copy, mads record, and the C API receive an ordinary blob copy. The ring is lossy: a slow consumer may find its slot overwritten, and that message is then dropped and counted, never delivered half-written. A consumer on another host cannot read the ring and reports an error, so use shm only when every consumer of the topic is local.

Plugin zero-copy API (protocol P9)

Plugins built against plugin protocol 9 can skip the remaining copies. All the new methods have copying defaults, so a P8 plugin only needs a rebuild:

Plugin	Override	Effect
Filter, Sink	load_data_view(in, topic, BlobView blob)	Read the blob in place. The default copies it and calls load_data().
Source	supports_shared_output() + get_output_shared(out, BlobBuffer &buf)	Write the blob straight into a ring slot.
Filter	supports_shared_output() + process_shared(out, BlobBuffer &buf)	Same, for a filter’s output.

A BlobView is valid only for the duration of the call (for a filter, through the following process()). If an output blob does not fit in buf.capacity, the plugin sets buf.size to the size it needs and returns return_type::retry; the loader then calls get_output()/process() and sends that blob inline. See the shared-memory guide for a full example.

Gotchas

Warning

A filter that receives a message on the same topic it publishes to skips it and logs a warning, specifically to avoid self-feedback loops. If your filter’s output never seems to arrive downstream, check whether its pub_topic collides with its own sub_topic.

Warning

Two ini keys have deprecated spellings that still work but print a warning: dont-block → use dont_block, and high_watermark → use queue_size.

See also

mads · plugin · broker · Local transport and shared memory