LLM-Assisted IoT Testing: Finding Conformance Bugs in Matter SDKs
LLM-Assisted IoT Testing: Finding Conformance Bugs in Matter SDKs
Xiaoyue Ma,Junming Chen,Lannan Luo,Qiang Zeng
TLDR
MatterGuard is presented, the first automated system for identifying bugs in Matter SDKs that violate the specification, and unlike traditional SDK testing approaches, which typically integrate testing code with the SDK code, Matter-Guard decouples the two, allowing the testing code to be reused across SDK versions.
Abstract
Matter is an IoT standard endorsed by hundreds of companies, designed to ensure interoperability between devices from various vendors. The Matter Software Development Kit (SDK) serves as the foundation for developing Matter devices, making bug discovery in Matter SDKs crucial. Given the extensive specification and the rapid evolution of Matter—five versions released in just two and a half years—the need for automated solutions is increasingly urgent. In this paper, we present MatterGuard, the first automated system for identifying bugs in Matter SDKs that violate the specification. Unlike traditional SDK testing approaches, which typically integrate testing code with the SDK code, Matter-Guard decouples the two, allowing the testing code to be reused across SDK versions. Furthermore, MatterGuard leverages a large language model to analyze the Matter specification and uses the extracted knowledge to guide the bug discovery process. In our evaluation across all five SDK versions, MatterGuard uncovers 109 bugs, demonstrating the effectiveness and scalability of our approach.