[udf] Unlock Lua for user-defined functions

Author: amotl

CHANGES.rst

@@ -13,6 +13,7 @@ in progress
 - Tests: Add more test cases to increase mqttwarn core coverage to ~100%
 - Improve example "Forward OwnTracks low-battery warnings to ntfy"
 - [udf] Unlock JavaScript for user-defined functions. Thanks, @extremeheat.
+- [udf] Unlock Lua for user-defined functions. Thanks, @scoder.
 
 
 2023-04-28 0.34.0

Dockerfile

@@ -34,7 +34,7 @@ RUN --mount=type=cache,id=pip,target=/root/.cache/pip \
     true \
     && pip install --upgrade pip \
     && pip install --prefer-binary versioningit wheel \
-    && pip install --use-pep517 --prefer-binary '/src[javascript]'
+    && pip install --use-pep517 --prefer-binary '/src[javascript,lua]'
 
 # Uninstall build prerequisites again.
 RUN apt-get --yes remove --purge git && apt-get --yes autoremove

docs/configure/transformation.md

@@ -447,6 +447,19 @@ export its main entry point symbol, configure mqttwarn to use `functions = myclo
 and adjust its settings to use your MQTT broker endpoint at the beginning of the data
 pipeline, invoke mqttwarn, and turn off Kafka. It works!
 
+On the next day, after investigating if you need to migrate any other system components,
+you realize that there is an Nginx instance, which receives a certain share of telemetry
+traffic using HTTP, and processes it using Lua. One quick `mosquitto_pub` later, you are
+sure those telemetry messages are _also_ available on the MQTT bus already. Another set
+of transformation rules written in Lua was quickly identified, and, after applying the
+same procedure of inlining it into a single-file version, and configuring another mqttwarn
+instance with `functions = mycloud.lua`, you are ready to turn off your whole cloud
+infrastructure, and save valuable resources.
+
+After a while, you are able to hire back half of your previous engineering team, and,
+based on the new architecture, you will happily start contributing back to mqttwarn,
+both in terms of maintenance, and by adding new features.
+
 :::{note}
 Rest assured we are overexaggerating a bit, and [Kafka] can only be compared to [MQTT]
 if you are also willing to compare apples with oranges, but you will get the point that
@@ -467,6 +480,15 @@ available [OCI images](#using-oci-image).
 You can find an example implementation for a `filter` function written in JavaScript
 at the [OwnTracks-to-ntfy example tutorial](#owntracks-ntfy-variants-udf).
 
+#### Lua
+
+For running user-defined functions code written in Lua, mqttwarn uses the excellent
+[lupa] package. For adding JavaScript support to mqttwarn, install it using pip like
+`pip install --upgrade 'mqttwarn[lua]'`, or use one of the available
+[OCI images](#using-oci-image).
+
+You can find an example implementation for a `filter` function written in Lua
+at the [OwnTracks-to-ntfy example tutorial](#owntracks-ntfy-variants-udf).
 
 
 ## User-defined function examples
@@ -707,6 +729,7 @@ weather,topic=tasmota/temp/ds/1 temperature=19.7 1517525319000
 [Jinja2 templates]: https://jinja.palletsprojects.com/templates/
 [JSPyBridge]: https://pypi.org/project/javascript/
 [Kafka]: https://en.wikipedia.org/wiki/Apache_Kafka
+[lupa]: https://github.com/scoder/lupa
 [MQTT]: https://en.wikipedia.org/wiki/MQTT
 [Node.js]: https://en.wikipedia.org/wiki/Node.js
 [OwnTracks]: https://owntracks.org

docs/usage/pip.md

@@ -11,9 +11,9 @@ that.
 pip install --upgrade mqttwarn
 ```
 
-Add JavaScript support for user-defined functions.
+Add JavaScript and Lua support for user-defined functions.
 ```bash
-pip install --upgrade 'mqttwarn[javascript]'
+pip install --upgrade 'mqttwarn[javascript,lua]'
 ```
 
 You can also add support for a specific service plugin.

examples/owntracks-ntfy/mqttwarn-owntracks.lua

@@ -0,0 +1,28 @@
+--[[
+Forward OwnTracks low-battery warnings to ntfy.
+https://mqttwarn.readthedocs.io/en/latest/examples/owntracks-battery/readme.html
+--]]
+
+-- mqttwarn filter function, returning true if the message should be ignored.
+-- In this case, ignore all battery level telemetry values above a certain threshold.
+function owntracks_batteryfilter(topic, message)
+    local ignore = true
+
+    -- Decode inbound message.
+    local data = json.decode(message)
+
+    -- Evaluate filtering rule.
+    if data ~= nil and data.batt ~= nil then
+        ignore = tonumber(data.batt) > 20
+    end
+
+    return ignore
+end
+
+-- Status message.
+print("Loaded Lua module.")
+
+-- Export symbols.
+return {
+    owntracks_batteryfilter = owntracks_batteryfilter,
+}

examples/owntracks-ntfy/readme-variants.md

@@ -50,9 +50,9 @@ targets   = {'testdrive': 'http://localhost:5555/testdrive'}
 
 ### JavaScript
 
-In order to try that on the OwnTracks-to-ntfy example, use the alternative
-`mqttwarn-owntracks.js` implementation by adjusting the `functions` setting within the
-`[defaults]` section of your configuration file, and restart mqttwarn.
+In order to explore JavaScript user-defined functions using the OwnTracks-to-ntfy recipe,
+use the alternative `mqttwarn-owntracks.js` implementation by adjusting the `functions`
+setting within the `[defaults]` section of your configuration file, and restart mqttwarn.
 ```ini
 [defaults]
 functions = mqttwarn-owntracks.js
@@ -69,3 +69,25 @@ previous one, which was written in Python.
 The feature to run JavaScript code is currently considered to be experimental.
 Please use it responsibly.
 :::
+
+### Lua
+
+In order to explore Lua user-defined functions using the OwnTracks-to-ntfy recipe,
+use the alternative `mqttwarn-owntracks.lua` implementation by adjusting the `functions`
+setting within the `[defaults]` section of your configuration file, and restart mqttwarn.
+```ini
+[defaults]
+functions = mqttwarn-owntracks.lua
+```
+
+The Lua function `owntracks_batteryfilter()` implements the same rule as the
+previous ones, which was written in Python and JavaScript.
+
+:::{literalinclude} mqttwarn-owntracks.lua
+:language: lua
+:::
+
+:::{attention}
+The feature to run Lua code is currently considered to be experimental.
+Please use it responsibly.
+:::

mqttwarn/util.py

@@ -7,6 +7,7 @@
 import os
 import re
 import string
+import sys
 import threading
 import types
 import typing as t
@@ -204,6 +205,9 @@ def load_functions(filepath: t.Optional[t.Union[str, Path]] = None) -> t.Optiona
     elif file_ext.lower() in [".js", ".javascript"]:
         py_mod = load_source_js(mod_name, filepath)
 
+    elif file_ext.lower() in [".lua"]:
+        py_mod = load_source_lua(mod_name, filepath)
+
     else:
         raise RuntimeError(f"Unable to interpret module file: {filepath}")
 
@@ -285,3 +289,41 @@ def load_source_js(mod_name, filepath):
     javascript.eval_js(js_code)
     threading.Event().wait(0.01)
     return module_factory(mod_name, module["exports"])
+
+
+class LuaJsonAdapter:
+    """
+    Support Lua as if it had its `json` module.
+
+    Wasn't able to make Lua's `json` module work, so this provides minimal functionality
+    instead. It will be injected into the Lua context's global `json` symbol.
+    """
+
+    @staticmethod
+    def decode(data):
+        if data is None:
+            return None
+        return json.loads(data)
+
+
+def load_source_lua(mod_name, filepath):
+    """
+    Load a Lua module, and import its exported symbols into a synthetic Python module.
+    """
+    import lupa
+
+    lua = lupa.LuaRuntime(unpack_returned_tuples=True)
+
+    # Lua modules want to be loaded without suffix, but the interpreter would like to know about their path.
+    modfile = Path(filepath).with_suffix("").name
+    modpath = Path(filepath).parent
+    # Yeah, Windows.
+    if sys.platform == "win32":
+        modpath = str(modpath).replace("\\", "\\\\")
+    lua.execute(rf'package.path = package.path .. ";{str(modpath)}/?.lua"')
+
+    logger.info(f"Loading Lua module {modfile} from path {modpath}")
+    module, filepath = lua.require(modfile)
+    # FIXME: Add support for common modules, as long as they are not available natively.
+    lua.globals()["json"] = LuaJsonAdapter
+    return module_factory(mod_name, module)

setup.py

@@ -55,6 +55,9 @@
     "javascript": [
         "javascript==1!1.0.1; python_version>='3.7'",
     ],
+    "lua": [
+        "lupa<3",
+    ],
     "mysql": [
         "mysql",
     ],

tests/test_util.py

@@ -199,6 +199,41 @@ def test_load_functions_javascript_runtime_failure(tmp_path):
     assert ex.match("ReferenceError: bar is not defined")
 
 
+def test_load_functions_lua_success(tmp_path):
+    """
+    Verify that Lua module loading, including symbol exporting and invocation, works well.
+    """
+    luafile = tmp_path / "test.lua"
+    luafile.write_text("return { forty_two = function() return 42 end }")
+    luamod = load_functions(luafile)
+    assert luamod.forty_two() == 42
+
+
+def test_load_functions_lua_compile_failure(tmp_path):
+    """
+    Verify that Lua module loading, including symbol exporting and invocation, works well.
+    """
+    luafile = tmp_path / "test.lua"
+    luafile.write_text("Hotzenplotz")
+    with pytest.raises(Exception) as ex:
+        load_functions(luafile)
+    assert ex.typename == "LuaError"
+    assert ex.match("syntax error near <eof>")
+
+
+def test_load_functions_lua_runtime_failure(tmp_path):
+    """
+    Verify that Lua module loading, including symbol exporting and invocation, works well.
+    """
+    luafile = tmp_path / "test.lua"
+    luafile.write_text("return { foo = function() bar() end }")
+    luamod = load_functions(luafile)
+    with pytest.raises(Exception) as ex:
+        luamod.foo()
+    assert ex.typename == "LuaError"
+    assert ex.match(re.escape("attempt to call a nil value (global 'bar')"))
+
+
 def test_load_function():
 
     # Load valid functions file