MAS-Lab Open-Source Release and arXiv Paper

We have released MAS-Lab: A Specification-Driven Validation Framework for Reliable Multi-Agent Systems as open source on GitHub, together with an accompanying arXiv preprint.

Since late 2024 I’ve been working on agentic and multi-agent systems — with a focus on making them robust, observable, and evaluable, not just fast to prototype. I’m still amazed at how quickly an entire ecosystem has emerged, yet important gaps remain around robustness, observability, and evaluation — gaps that feel unnecessary, since comparable problems have been addressed for years in other engineering domains through established practices. Those limitations became especially clear on Aether: turning a working prototype into something we could run at scale, wired into protocols that keep evolving (MCP, A2A), observability, and the rest of the operational stack — bricks you can often skip in a local test, but that become unavoidable once you aim for production. And we still had to evaluate it rigorously enough to stand up in a scientific paper.

To address that, we started developing MAS-Lab. The project blog gives an overview of the tool and its different components. We also wrote a paper about these challenges and how we approached them — including the formalization based on Mealy state machines that sits at the heart of the runtime. All code is released as open source on GitHub.

MAS-Lab is built around four components:

Spec. A multi-agent workflow is defined declaratively — agents, roles, delegation patterns, tools, and policies — as a versioned specification rather than logic scattered across prompts and glue code. That gives the system a stable identity you can compare, benchmark, and evolve, independently of whichever framework or model implements each agent.

Runtime. The runtime enforces the specification during execution. It is grounded in a formal model based on Mealy state machines: actions pass through policy gates, state and external resources are mediated, and interactions are recorded with enough context to audit what happened. The design also supports multiplexing across stateless connections, so several agentic applications can share the same infrastructure.

Control. A deployment layer carries the same specification from local development through parallel benchmark runs to production. The aim is lifecycle continuity — not rebuilding the workflow separately for each stage, but promoting one declared system with the orchestration and governance hooks needed in each environment.

Lab. Labs provide repeatable experiment harnesses: vary one factor at a time while holding the spec and runtime fixed, and collect comparable traces and artifacts for evaluation, regression checks, and readiness reviews. They bundle the benchmark scaffolding and analysis tooling that most projects otherwise rewrite for every new scenario.

MAS-Lab is a large project — what we’re releasing today is only a starting point, and we plan to extend it and keep building on it. Big thanks to my colleagues Giovanna Carofiglio, Giulio Grassi, and Jacques Samain, who were instrumental in making it happen.