MongoDB Replay

Author

Affiliation

Paolo Bosetti

University of Trento

Abstract

MADS v2.0.1 provides a new replay.plugin for replaying a stream of datasets from a MongoDB database. This can be useful for testing and debugging purposes.

Rationale

Before MADS v2.0.1, there was no built-in way to replay a stream of datasets from a MongoDB database. The only solution was to export the data to a CSV file and then use the plugin replay_plugin to read the CSV file and publish the datasets. This was not ideal, as it required extra steps and a custom plugin.

Furthermore, the replay plugin allowed to stream a single topic at a time, making it impossible to replay a complex system with multiple topics and interdependencies.

Now we have a better solution directly provided by the MADS installer, which allows to replay a stream of datasets straight from the MongoDB database, without the need for intermediate CSV files. This plugin can be launched with the usual mads source replay command, and it allows to replay multiple topics at the same time, preserving the original timestamps and interdependencies between datasets.

Tip

Just to be clear: no need to clone and compile a new plugin: the replay.plugin is now included in the MADS installer, and can be launched with the usual mads source replay command.

Usage

INI file section

as for any agent, you first need to vcreate a section in the mads.ini file for the replay.plugin, with the following additional fields:

[replay]
db_uri = "mongodb://localhost:27017"
db_name = "mads_test"                                      # Mandatory, a string
collections = ["perf_assess_1", "perf_assess_2"]   # Mandatory, array of strings
start_time = "2026-03-10T20:27:00Z"      # Mandatory, a string in ISO8601 format
end_time = "2026-03-10T20:27:10Z"        # Optional, a string in ISO8601 format
max_loop_duration_ms = 0
repeat = false
view_name = "mads_replay_view"   # default="": if specified mongo_replay creates
                                 # a view with this name 
reuse_view = false          # default=false: if true and view_name is specified, 
                            # mongo_replay will reuse the existing view

The above is pretty self-explanatory, but here are some details:

• db_uri is the URI of the MongoDB database to read from. It can be either a local or a remote database, and it can include authentication credentials if needed. This is optional, and if missing “mongodb://localhost:27017” is used as default.
• db_name is the name of the database to read from. This is mandatory, and it must be a string.
• collections is the list of collections to read from. This is mandatory, and it must be an array of strings. The plugin will read all the datasets from the specified collections, and publish them on the corresponding topics (i.e. the topic name will be the same as the collection name).
• start_time is the timestamp of the first dataset to read. This is mandatory, and it must be a string in ISO8601 format. The plugin will read only the datasets with a timestamp greater than or equal to this value.
• end_time is the timestamp of the last dataset to read. This is optional, and it must be a string in ISO8601 format. If specified, the plugin will read only the datasets with a timestamp less than or equal to this value. If not specified, the plugin will read all the datasets with a timestamp greater than or equal to start_time.
• max_loop_duration_ms is the maximum duration of a single loop in milliseconds. This is optional, and it must be a non-negative integer. If specified, the plugin will limit the duration of each loop to this value. This is useful whenever logged data may contain long periods of inactivity, which would otherwise cause the plugin to wait for a long time before publishing the next dataset. If not specified or set to 0, there is no limit on the loop duration.
• repeat is a boolean flag that indicates whether to repeat the replay after reaching the end of the specified time range. This is optional, and it must be a boolean value (true or false). - view_name is the name of the MongoDB view to create for the replay. This is optional, and it must be a string. If specified, the plugin will create a view with this name that filters the datasets according to the specified time range and collections. The plugin will then read from this view instead of directly from the collections. This can improve performance and reduce the load on the database, especially if the original collections are large. If not specified, a temporary view with a random name will be created and deleted at the end of the replay.
• reuse_view is a boolean flag that indicates whether to reuse an existing view if view_name is specified. This is optional, and it must be a boolean value (true or false). If set to true and the specified view already exists, the plugin will reuse the existing view instead of creating a new one. This can be useful if you want to run multiple replays with the same view, or if you want to create the view manually with specific settings.

Running the plugin

As usual, you can run the plugin with the mads source replay command, which will start the agent and publish the datasets on the corresponding topics. The plugin will read the datasets from the specified MongoDB database, and publish them with their original time differences, preserving the interdependencies between datasets from different collections.

So, for example, if you are replaying the collections signal_1 and signal_2, and thefirst colletion contains a data flow with one document every 100 ms, while the second collection contains a data flow with one document every 500 ms, the plugin will publish the datasets from signal_1 every 100 ms, and the datasets from signal_2 every 500 ms, preserving the original timing, although with timestamps translated to the current time (i.e. the timestamp of the first published dataset will be the current time, and the timestamps of the subsequent datasets will be shifted accordingly).