Plugins

Author

Affiliation

Paolo Bosetti

University of Trento

Abstract

MADS supports agents implemented as plugins in C++17. This guide explains how to create a plugin.

What are plugins

Originally, MADS only allowed to develop monolitic agents: this required to write the whole executable code, including the management of command line arguments and loading of serrings from the INI file (either local or loaded from the broker). This solution is still possible (see the guide) and allows maximum flexibility, but it requires a good knowledse of the MADS innards and is more tedious and error prone.

For this reason, we developed the plugin support: the common functionality of an agent are already available, and only the data management operations need to be implemented.

Behaviors of plugins

Remember that there are three behaviors for plugins:

• source: puts data into the MADS network
• filter: transforms data
• sink: consumes data

As a consequences, there are three MADS subcommands to load a plugin, depending on its behavior: mads source, mads filter, and mads sink. They all take a mandatory argument that is the name or the path to a proper plugin file.

A plugin file is a compiled dynamic library with the extension .plugin. The behavior of a plugin is rigid: a source plugin cannot be loaded by a filter agent.

How to create a plugin

The mads command provides an utility to generate a template for a new plugin:

> mads plugin -h

Usage:
  plugin [OPTION...] name

  -t, --type arg         Type of the plugin
  -d, --dir arg          Directory of the plugin
  -i, --install-dir arg  Directory where to install the plugin
  -o, --overwrite        Overwrite existing files
  -v, --version          Print version
  -h, --help             Print usage

Suppose that you want to create a source plugin named my_plugin: then the command shall be mads plugin -t source -d my_source my_source. This will create a CMake stub project in the folder my_source that, once compiled, will produce a my_source.plugin library, setting CMake to install that file in the default MADS prefix path, as given by mads -p. The command line option -i can be used to override that path.

How to configure and build

As suggested by the command output:

cd my_source
cmake -Bbuild 
# make your changes
cmake --build build -j6

These steps are iterative, and probably need to tweak with the CMakeLists.txt file for adding third party libraries and other source/header files. The last command produces in the build folder two notable files:

• my_source.plugin: the plugin proper;
• my_source: an executable generated by conditionally enabling the  main function at the end of the my_source.cpp file.

The second file can be used for checking that the plogin properly deals with the data, without the need of a MADS network available (broker etc.). Indeed, that executable does not even connect to the broker. Launch it as:

build/my_source

During development, you want to use this executable quite often to test what you are doing. You also have to tune the main function so that the dummy data are properly defined.

When the bare executable works, you can try to load the plugin with the proper agent: provided that the broker is running and the mads.ini file contains a proper [my_source] section, you load the plugin as:

mads source build/my_source.plugin

The agent name is the name of the plugin (i.e. my_source). If you need to override that name — for example, because you want to have different identical agents that load different settings — then you can use the -n command line option and match the proper mads.ini section.

Important

On MacOS, the plugin file can also be directly executed, so the my_source executable is not generated and the my_source.plugin can be both loaded from an agent or directly executed for testing.

How to install the plugin

Once the plugin is tested (both standalone and when loaded from an agent) it should be installed in the MADS tree. Remember that the command mads -i shows where plugins are expected to be installed (usually, /usr/local/lib). To install:

cmake --install build

The you can launch it as agent with:

mads source my_source.plugin

And when you are satisfield with the result, you can make it a service as explained here.

Writing a plugin

Concepts

All the complexity of writing a shared object (i.e. a dynamic library) that can be loaded at runtime as a plugin is demanded to the pugg and plugin libraries that are automatically loaded by CMake (via FetchContent). This makes relatively easy and fast to implement a plugin, effectively requiring the user to inherit from a plugin class and to override a handful of methods.

The mads plugin command creates a template cpp file that provides a commented skeleton class, which the user has to complete in a few points.

The list of methods to override depends on the type of plugin (source, filter or sink), and are detailed in the next section.

Mandatory methods

Source plugin

The methods to override are:

• void set_params(void const *params): receives a buffer containing a nlohmann::json object, which contains the settings loaded by the mads.ini file; it is its responsibility to locally store those data. This method is called once on startup.
• return_type get_output(json &out, std::vector<unsigned char> *blob = nullptr): provides the plugin loader with a new nlohmann::json object to be published on the network (optionally, a binary blob can be attached as well). This method is called repeatedly in the plugin loader loop.
• map<string, string> info(): returns a map of info that is printed at the end of the agent initialization; it is useful for explicitly showing the settings values as loaded by mads.ini via broker. This method is called once after set_params().

Note that set_params() should call the parent class method Source::set_params(params): this way, the attribute _params is filled with all the settings received from the broker.

The return value of get_output() can be:

• return_type::success: the output document out has been filled correctly
• return_type::warning: the user can set the _error string for adding a warning inside the out document (field ["warning"]["get_output"]); the rest of the document is filled as usual
• return_type::retry: nothing is published by the plugin loader, skip to next iteration
• return_type::error: nothing is published, the status line increments the counters; the user can set the _error string for publishing an error message to the agent_event special topic (field ["error"]["get_output"]).
• return_type::critical: the execution stops, the _error string is printed and a message is published on agent_event.

Filter plugin

The methods to override are:

• void set_params(void const *params): receives a buffer containing a nlohmann::json object, which contains the settings loaded by the mads.ini file; it is its responsibility to locally store those data. This method is called once on startup.
• return_type load_data(json const &input, string topic = ""): receives from the plugin loader the JSON payload last recieved on the given topic. This method should internally elaborate and store the received data. This method is called repeatedly in the plugin loader loop.
• return_type process(json &out): process the received data (internally stored) and provides back the plugin loader with a new nlohmann::json object to be published on the network. This method is called repeatedly in the plugin loader loop (after load_data()).
• map<string, string> info(): returns a map of info that is printed at the end of the agent initialization; it is useful for explicitly showing the settings values as loaded by mads.ini via broker. This method is called once after set_params().

Note that set_params() should call the parent class method Source::set_params(params): this way, the attribute _params is filled with all the settings received from the broker.

The return value of get_output() can be:

• return_type::success: the input or output documents have been filled correctly
• return_type::warning:
  • load_data(): the user can set the _error string for adding a warning inside the document that will be published (field ["warning"]["load_data"])
  • process(): the user can set the _error string for adding a warning inside the document that will be published (field ["warning"]["process"])
• return_type::retry:
  • load_data(): nothing is published by the plugin loader
  • process(): nothing is published by the plugin loader (but data has been loaded by load_data())
• return_type::error:
  • load_data(): nothing is published, the status line increments the counters; the user can set the _error string for publishing an error message to the agent_event special topic (field ["error"]["load_data"]). The process() method is not called.
  • process(): nothing is published, the status line increments the counters; the user can set the _error string for publishing an error message to the agent_event special topic (field ["error"]["process"]).
• return_type::critical: the execution stops, the _error string is printed and a message is published on agent_event.

Sink plugin

The methods to override are:

• void set_params(void const *params): receives a buffer containing a nlohmann::json object, which contains the settings loaded by the mads.ini file; it is its responsibility to locally store those data. This method is called once on startup.
• return_type load_data(json const &input, string topic = ""): receives the nlohmann::json object from the plugin loader and disposes of it accordingly. This method is called repeatedly in the plugin loader loop.
• map<string, string> info(): returns a map of info that is printed at the end of the agent initialization; it is useful for explicitly showing the settings values as loaded by mads.ini via broker. This method is called once after set_params().

Note that set_params() should call the parent class method Source::set_params(params): this way, the attribute _params is filled with all the settings received from the broker.

The return value of load_data() can be:

• return_type::success: data has been correctly elaborated.
• return_type::warning: the user can set the _error string, which is printed on console.
• return_type::retry: nothing is done.
• return_type::error: the user can set the _error string, which is printed on console and published on agent_event (field ["error"]["load_data"]).
• return_type::critical: the execution stops, the _error string is printed and a message is published on agent_event.

Settings

For each plugin, the list of supported settings in the corresponding section of the mads.ini file is decided by the developer. Any given setting is collectively received within a JSON object by the set_params() method, which has the responsibility for checking for correctness and providing a sensible default.

Starting from MADS v1.4.0, the plugin loaders can override the settings in the mads.ini file using the -o|--option command line switch, which may be repeated once for each settings to be overridden. For example, suppose that the INI section is:

[plugin_name]
key1 = "value"
key2 = 10.3
key3 = 7

Then, each option can be overridden by:

mads source my_source.plugin -o key1=test -o key2=10.4 -o key3=13

so the set_params() method will receive a JSON as:

{
  "key1": "test",
  "key2": 10.4,
  "key3": 13
}