Local transport and shared memory

Author

Affiliation

Paolo Bosetti

University of Trento

Abstract

Agents on the same machine as the broker can skip the network stack entirely: they talk to the broker over ipc sockets, and large binary blobs can stay in shared memory while only a small descriptor travels on the bus. This guide explains both mechanisms, when to use them, and how plugins can read and write blobs without copying them.

Context

MADS was born for distributed systems, so every agent talks to the broker over TCP. On a single machine, though, much of that work is wasted: an agent on the broker’s own host still goes through the loopback TCP stack, and a large binary payload, such as a camera frame (see Binary payloads), is copied into the ZeroMQ message, through the broker, and again into the receiving agent’s memory.

MADS offers two independent remedies for agents that share a host:

• the local transport: the broker also listens on ipc:// endpoints (Unix-domain sockets), and local agents use them instead of loopback TCP. This is automatic, needs no configuration, and benefits every message;
• the shared-memory data plane: a producer keeps its blobs in a ring of shared-memory slots, and only a small descriptor goes through the broker. Consumers on the same host read the bytes in place. This is opt-in, per producer, and pays off for large blobs.

Local transport

When the broker starts, it binds its frontend, backend and settings sockets twice: on their usual tcp:// addresses, and on ipc:// socket files in a per-user runtime directory ($XDG_RUNTIME_DIR/mads or /tmp/mads-<uid> on Unix, %LOCALAPPDATA%\mads\run on Windows), for example /tmp/mads-1000/broker-9090-front.ipc.

At startup each agent asks the broker which ipc endpoints it offers, and with the default transport = "auto" it switches to them when the broker runs on the same host and the socket files are visible. Otherwise it stays on tcp. You can force either transport with the transport key (in [agents] or in the agent’s own section) or the --transport option:

[agents]
transport = "auto"   # or "tcp", or "ipc" (fail if ipc is not possible)

The agent’s startup information shows which transport it ended up using, and mads doctor predicts it for any agent you would start on that host. The broker side is covered in detail in the broker manual, and the agent side in Choosing the transport.

ImportantSecurity

ip_whitelist does not apply to ipc peers, which have no IP address. The owner-only runtime directory is their access control. See Security in MADS.

Shared-memory data plane

The model

flowchart LR
  P["Producer agent
(shm = true)"]
  R[("Shared-memory ring
slots 0..N-1")]
  B["Broker"]
  C1["Consumer on
the same host"]
  C2["Agent that did
not opt in"]
  P -- "writes blob" --> R
  P -- "descriptor" --> B
  B -- "descriptor" --> C1
  B -- "descriptor" --> C2
  R -. "reads in place" .-> C1
  R -. "one copy" .-> C2

A producer with shm = true owns a ring of fixed-size slots in a named shared-memory segment. For every blob it publishes, it:

1. takes the next slot, round-robin, and writes the bytes into it;
2. publishes an ordinary MADS message whose JSON metadata carries a small descriptor: segment name, slot index, sequence number, size, and the producer’s host.

The broker forwards that message like any other: it is still payload-opaque. A consumer on the same host maps the segment once, then reads each blob straight from the slot.

When to use it

Shared memory helps when blobs are large and frequent, and when producer and consumers run on the same machine: camera frames, depth maps, point clouds, audio buffers. For small messages the descriptor costs about as much as the data, and the local transport alone already removes the TCP overhead. For consumers on other hosts it does not work at all (see below).

Configuration

Only the producer is configured, in its own section:

[camera]
pub_topic = "frames"
shm = true               # default false
shm_slots = 8            # default 8
shm_slot_size = 8388608  # bytes, default 16777216 (16 MiB)

Size the slots for your largest regular blob. A blob larger than shm_slot_size is not an error: it is sent inline, as an ordinary binary payload, with a one-time warning. Memory use is shm_slots × shm_slot_size per producer.

Shared memory is used only when the producer’s broker is on its own host (that is, when the producer is connected over ipc or local tcp). Otherwise the agent warns and publishes inline.

Lossy semantics and torn reads

The ring never blocks the producer. When all slots have been used, the oldest is overwritten, whether or not every consumer has read it. This is the right trade-off for live sensor data: a slow consumer should skip frames, not stall the camera.

Each slot carries a sequence counter, updated by the producer before and after it writes (a seqlock). The descriptor records the value the slot had when the blob was published. A consumer checks the counter before and after reading:

• same value: the blob is intact;
• different value: the producer has reused the slot meanwhile. The read is torn: the message is dropped and counted, never delivered half-written.

If your consumers see torn reads, give them more headroom with more slots (shm_slots), or make them faster.

Who can read shared blobs

Consumer	What it gets
Filter or sink on the same host, P9 plugin with load_data_view()	A read-only view into the slot, no copy.
Any other agent on the same host (older plugins, mads record, C API, Python)	An ordinary binary payload: the loader copies the slot once.
Any agent on another host	Nothing: a clear error that the shared-memory reference comes from a foreign host.
An agent on the same host but under another OS user, or in a container with its own /dev/shm (IPC namespace)	Nothing: it cannot map the segment (segments are created owner-only, mode 0600).

The consequence of the last two rows is the same-host, same-user rule: enable shm on a producer only if every consumer of its topic runs on the producer’s host, as the same OS user, and in the same container. A consumer that breaks the rule does not fail loudly: it drops every blob of the topic, reporting it only through rate-limited warnings (the first one names the cause, for example “permission denied … same OS user”). If you cannot guarantee the rule, leave shm off. There is no bridge that copies shared blobs to remote agents yet.

Recording and replaying

mads record does not opt in to zero-copy, so it receives the real bytes and stores them in the bag exactly as for an inline blob. A bag therefore never refers to a shared-memory segment that no longer exists, and mads play publishes its blobs as ordinary payloads (unless you set shm = true on the player’s own section, which makes it a shared-memory producer like any other).

Plugin zero-copy API

Plugin protocol P9 (mads_plugin v2.6-P9) adds two plain structs and a few virtual methods. All of them have defaults that copy, so a P8 plugin only needs to be rebuilt: mads plugin --update moves the pin and prints the opt-in notes.

struct BlobView   { const unsigned char *data; size_t size; };           // read-only
struct BlobBuffer { unsigned char *data; size_t capacity; size_t size; }; // a ring slot

Reading blobs in place

Filters and sinks override load_data_view(). The default implementation copies the view into a vector and calls load_data(), which is why old plugins keep working.

return_type load_data_view(json const &meta, std::string topic,
                           BlobView blob) override {
  // Valid only during this call (for a filter: through the next process()).
  cv::Mat frame(meta["height"].get<int>(), meta["width"].get<int>(), CV_8UC3,
                const_cast<unsigned char *>(blob.data));
  _mean = cv::mean(frame)[0]; // use it now, or copy what you need to keep
  return return_type::success;
}

Never keep blob.data after the call, and never write through it. After the call the loader re-checks the slot; if the producer overwrote it meanwhile, the result is discarded as a torn read.

Writing blobs into a slot

Sources (and filters) opt in by returning true from supports_shared_output() and implementing get_output_shared() (or process_shared() for filters). The loader hands over a ring slot as a BlobBuffer: write at most buf.capacity bytes into buf.data and set buf.size.

bool supports_shared_output() const override { return true; }

return_type get_output_shared(json &out, BlobBuffer &buf) override {
  cv::Mat frame;
  if (!_cap.read(frame) || frame.empty())
    return return_type::retry;
  size_t n = frame.total() * frame.elemSize();
  if (n > buf.capacity) {
    _pending = frame;        // keep it for get_output()
    buf.size = n;            // tell the host how much room was needed
    return return_type::retry;
  }
  std::memcpy(buf.data, frame.data, n);
  buf.size = n;
  out["width"] = frame.cols;
  out["height"] = frame.rows;
  out["format"] = "bgr8";
  return return_type::success;
}

The oversize contract matters: if the blob does not fit, write nothing, set buf.size to the size you need (so that it exceeds buf.capacity) and return return_type::retry without losing the data. The loader then calls the classic get_output() (or process()) in the same loop iteration and sends that blob inline.

Likewise, buf.size == 0 (no blob this time) must mean that nothing was written: the loader then hands the slot back untouched, and the blob previously published in it stays readable for slow consumers. Never scribble into buf.data unless you set buf.size to what you wrote.

A plugin that does not opt in still benefits when its agent has shm = true: the loader copies the get_output() blob into a slot, which costs one copy but saves all the ZeroMQ and broker copies downstream.

Housekeeping

On Linux and macOS a shared-memory segment outlives a producer that crashes. To reclaim them, every producer registers its segments under <runtime dir>/shm/ together with its pid, and a new producer unlinks the segments of dead owners when it starts. mads doctor lists the registry and counts the orphans, and mads doctor --fix removes them on demand.

On Windows the segments are named file mappings (Local\mads_...), which the kernel frees with their last handle: there is nothing to clean up.

Windows notes

• ipc uses AF_UNIX sockets, available since Windows 10. libzmq enables them only when it is built against the Windows 10 SDK (which provides afunix.h). With a libzmq built without them, the broker warns and serves tcp only, and agents fall back to tcp silently; mads doctor reports which case applies.
• Shared memory works on any supported Windows version, but the same-host rule applies: it is only used when the producer’s broker is local.

Summary

	Local transport	Shared-memory data plane
Scope	Every message between local agents and the broker	Blobs from producers with shm = true
Configuration	None (transport = "auto" default)	shm, shm_slots, shm_slot_size on the producer
Remote agents	Keep using tcp	Get an error: keep producers with remote consumers on shm = false
Loss	None	Lossy ring: slow consumers drop overwritten slots
Plugins	Unchanged	Optional P9 view API for zero-copy