Conformance suite

The conformance suite is 49 protocol-level checks in 8 categories. It runs against any plugin, in any language, with no Macro Deck installation: it lives in MacroDeck.Plugin.Testing.Conformance, on the same MacroDeckTestHost as Testing plugins, and produces one report with no test-framework types in it.

Example

The plugin the template generates, run with no test project:

macrodeck-plugin new --name "Hello Deck" --id com.example.hello-deck --publisher Example -y
cd HelloDeck
macrodeck-plugin test --project src/HelloDeck

Macro Deck plugin conformance report (suite 1.2.0)
Plugin: com.example.hello-deck 1.0.0
Started: 2026-09-12T10:27:14.6935280+00:00, duration: 00:00:20.8027253
Passed: 25, Failed: 0, Skipped: 24
Conformant: yes

[PASS] MDC0101 The plugin id is a valid reverse-domain package id (Required)
[PASS] MDC0102 Every declared capability's local id is a valid declared-kind identifier (Required)
[SKIP] MDC0103 Every weather station instance id is a valid resource-kind identifier (Required)
    Reason:   This subject does not declare the weather capability.
[SKIP] MDC0104 The manifest declares a supported manifest version and a protocol range this suite satisfies (Required)
    Reason:   This subject has no manifest - only an artifact subject does.
...
[SKIP] MDC0502 A deadline that elapses produces TIMEOUT, and nothing arrives afterward (Recommended)
    Reason:   No declared action ran long enough, under a 300 ms deadline, to observe deadline enforcement.
...
[PASS] MDC0804 Reconnecting does not replay a burst of previously published events (Required)

Skips are normal: a check that has nothing to exercise says so and why. Run a packed artifact (--artifact) to also cover the manifest checks.

Running it

From	How
CLI	macrodeck-plugin test with --project, --executable or --artifact. Filter with --category, --check (both repeatable) and --required-only; --list-checks prints the vocabulary. See test.
Your tests	ConformanceRunner plus a ConformanceSubject, below. Neither depends on a test framework.

In a test project

One NUnit test per check, so each failure is reported under its own id:

using MacroDeck.Plugin.Testing.Conformance;
using NUnit.Framework;

[TestFixture]
public sealed class ConformanceTests
{
    private static readonly ConformanceRunner Runner = new();
    private static ConformanceReport? _report;

    public static IEnumerable<IConformanceCheck> Checks => Runner.Checks;

    [OneTimeSetUp]
    public async Task RunSuite()
    {
        await using var subject = ConformanceSubject.InProcess(builder => builder
            .UseLocalization(Strings.LocalizationCatalog)
            .RegisterIntegration<PluginIntegration>());

        _report = await Runner.RunAsync(subject);
    }

    [TestCaseSource(nameof(Checks))]
    public void Check(IConformanceCheck check)
    {
        var result = _report!.Results.Single(r => r.Id == check.Id).Result;

        if (result.Outcome is ConformanceOutcome.Skipped or ConformanceOutcome.Inconclusive)
        {
            Assert.Ignore(result.SkipReason);
        }

        if (check.Requirement == ConformanceRequirement.Required)
        {
            Assert.That(result.Outcome, Is.EqualTo(ConformanceOutcome.Passed),
                $"Expected: {result.Expected}\nActual: {result.Actual}");
        }
    }
}

Under xUnit, use [Theory]/[MemberData] and Assert.Skip/Assert.True; the runner and subject are unchanged.

Subject factory	Runs
ConformanceSubject.InProcess(Action<PluginHostBuilder> configure, PluginTestManifest? manifest = null)	The same delegate you give MacroDeckPlugin.CreatePlugin(), against source.
ConformanceSubject.Executable(PluginLaunchSpec spec)	A real built process.
ConformanceSubject.Artifact(string macroDeckPluginPath)	A packed .macroDeckPlugin, closest to what a user installs.

Pass ConformanceOptions to the ConformanceRunner constructor to filter:

Option	Default	Meaning
Categories	[]	Only these ConformanceCategory values; empty means all.
Ids	[]	Only these check ids; empty means all.
RequiredOnly	false	Only Required checks.
PerCheckTimeout	60 s	Hard limit per check, enforced even when a check ignores its own cancellation token.

The three filters intersect.

Outcomes and requirements

Outcome	Meaning	Blocks Conformant?
Passed	The subject satisfied the check.	No
Failed	The subject violated the check.	Only for a Required check
Skipped	A declared precondition (live session, manifest, external process, controllable clock) was not met, or the check does not apply to this subject.	No
Inconclusive	The suite’s own infrastructure could not reach a verdict, for a reason outside the subject’s control.	No

• Required checks decide ConformanceReport.Conformant. Recommended checks need cooperation no protocol rule guarantees (an action slow enough to time out, log output to observe); their failures are reported and counted but never make a subject non-conformant.
• ConformanceRunner evaluates every precondition before the check body runs.
• Skipped and Inconclusive always carry a non-empty reason (ConformanceCheckResult.Skip/Inconclusive reject an empty one). A skip is not a pass: a plugin with no weather capability is not proven to handle weather correctly.
• Every check id is a stable public contract. An id is never reassigned, so --check MDC0305 in CI or a documented suppression stays valid. A title may be clarified; what the id asserts does not change.

Guard below marks a regression guard: a check a plugin built on the .NET SDK (MacroDeck.Plugin.Hosting) structurally cannot fail, because PluginHostBuilder.Build() or CapabilityDispatcher already enforces it. Guards still run against the wire; they make the suite a real contract check for a plugin written in another language or on another SDK. Every other check has a counterexample proven by this repository’s own misbehaving fixture, except MDC0804, which is a generic proxy for a trigger that cannot be built generically.

Checks

Titles are shortened; macrodeck-plugin test --list-checks prints the full ones. Req: R Required, Rec Recommended.

Manifest and identifiers (MDC01xx)

MDC0104-MDC0107 read the manifest, so they skip for any subject but an artifact.

Id	Req	Checks	How to fix
MDC0101	R, guard	The id /_macrodeck/info reports matches the reverse-domain package-id grammar.	Use an id like com.example.my-plugin.
MDC0102	R, guard	Every declared capability LocalId on the wire matches the declared-id grammar.	Use lowercase kebab-case local ids.
MDC0103	R	Every id weather/instances returns matches the resource-id grammar; skips without weather.	Validate instance ids chosen at runtime.
MDC0104	R	manifestVersion is supported and a declared compatibility.protocol overlaps the suite’s protocol range.	Set a supported manifest version and protocol range.
MDC0105	R, guard	The manifest id equals the reported id (ordinal); also skips with no reported id.	Keep one id.
MDC0106	R	The manifest name and version equal PluginHealthReport.Name/Version (ordinal); also skips when neither is reported.	Do not override name or version at runtime.
MDC0107	R	icons/describe reports the media type the manifest icon’s extension implies, from the check’s own extension table; skips with no icon or an unmapped extension (MDP1003).	Serve the icon with its real media type.

Registration and negotiation (MDC02xx)

Id	Req	Checks	How to fix
MDC0201	R, guard	session.hello carries the negotiated version and session id, and session.welcome arrives exactly once.	Follow the handshake.
MDC0202	R, guard	session.hello to session.welcome takes at most ProtocolTimeouts.Handshake (10 s); skips with no completed handshake.	Answer the handshake promptly.
MDC0203	R, guard	Against a host with an unreachable protocol range, the subject exits cleanly or settles not-live; tested through the client-side pre-check.	Stop rather than retry forever.
MDC0204	R	Two starts of a self-registering subject against the same host and state directory register exactly once; skips unless in-process and self-registering.	Persist and reuse the credentials.
MDC0205	R	A managed subject makes zero registration calls; skips unless managed.	Never register when managed.
MDC0206	R	With no enrollment token, a self-registering subject pairs through POST /api/plugins/pairing, the redeemed verifier hashes to the sent challenge, and a second start opens a session without pairing again; skips unless in-process and self-registering.	Prove possession of the verifier, and persist the credential.

Capability serialization (MDC03xx)

Id	Req	Checks	How to fix
MDC0301	R, guard	Every declared kind is in CapabilityKinds.All.	Declare only known kinds.
MDC0302	R	Every declared version range has Minimum <= Maximum, and no negotiated capability was rejected.	Declare a range the host supports.
MDC0303	R, guard	Every kind but actions and variables declares exactly one capability, at local id provider (icon for icons); variables catalog operations do not make it provider-shaped.	Declare provider-shaped kinds once, at the documented id.
MDC0304	R, guard	At most MaxDeclaredCapabilities (512) capabilities are declared.	Declare fewer.
MDC0305	R	The actions/describe reply is within MaxMessageBytes (256 KiB) and deserializes into ActionCatalogPayload; skips without actions.	Keep action metadata small.
MDC0306	R	ui/describe deserializes into UiDescribePayload and no surface has an empty kind; skips without ui.	Name every surface’s kind.
MDC0307	R	No ui/describe surface has an empty sessionMode; skips without ui.	Set a session mode, since an unknown one is treated as exclusive.
MDC0308	R	The ui/describe reply is within 256 KiB and deserializes into UiDescribePayload; skips without ui.	Keep the surface list small.
MDC0309	R	For each action reporting ProvidesState, state returns an ActionStateResult whose present snapshot has a non-empty States list with non-empty unique ids and an ActiveStateId naming one of them; null passes.	Return a consistent snapshot, or null when nothing is known.
MDC0310	R	Every returned state id passes MacroDeckId.TryValidateLocalId(_, LocalIdKind.Declared), as a button persists it; skips without ProvidesState.	Use declared-id grammar for state ids.
MDC0311	R	variables/discover answers within 5 s with at most MaxVariableCatalogPageSize items, each with a known VariableType and a resource id without :: and within MacroDeckId.MaxResourceLocalIdLength, and any ContinuationToken non-empty; skips without variables or with supportsCatalog: false, and fails if describe cannot be answered.	Page and bound the catalog.
MDC0312	R	For each action reporting ProvidesIcon, a NoIcon snapshot has no Reference and an empty Version, and any other has a non-empty Version and a Reference with non-empty Type and Reference; HasValue: false passes.	Return an internally consistent icon snapshot.
MDC0313	R	A non-NoIcon snapshot with no Reference answers icon.content with a non-empty ContentHash and an image/* MediaType, reports its Version has moved on, or fails only with AssetTooLarge; skips without ProvidesIcon.	Serve the icon bytes you advertise.
MDC0314	R	For each eager variable declaring write, writing its current reading back through variables/set succeeds as an invocation with a status other than NotWritable or NotFound; skips when nothing declares write, inconclusive when every writable variable reads unavailable.	Declare write only on variables you accept writes for.
MDC0315	R	variables and declaredVariables together hold at most VariableLimits.MaxEagerVariablesPerProvider eager entries, counted across the whole plugin; skips without variables.	Move the rest into a catalog.

Duplicate ids (MDC04xx)

Id	Req	Checks	How to fix
MDC0401	R, guard	No (Kind, LocalId) pair repeats among declared capabilities.	Rename one.
MDC0402	R	No Id repeats in weather/instances; skips without weather.	Keep instance ids unique.
MDC0403	R	No two eager variables resolve to the same local id, or to the same identity derived from Name when neither declares one; catalog ids are out of scope; skips without variables.	Give each eager variable its own id.

Timeout and cancellation (MDC05xx)

Id	Req	Checks	How to fix
MDC0501	R, guard	One invocation gets exactly one reply for its correlation id; skips with nothing to invoke.	Reply once.
MDC0502	Rec	An action invoked with a short deadline (300 ms) that times out produces exactly one reply; skips when no action runs that long.	Stop after TIMEOUT.
MDC0503	R, guard	Cancelling an unknown correlation id, or an already answered one, produces no plugin message.	Ignore such cancels.
MDC0504	Rec	Cancelling an in-flight invocation that reports cancelled produces exactly one reply; skips when nothing stays in flight long enough.	Reply once to a cancel.
MDC0505	Rec, guard	40 concurrent invocations never push the reported in-flight count past MaxConcurrentInvocations (32), and all complete; skips with nothing to invoke.	Queue a burst, do not fail it.

Disconnect and reconnect (MDC06xx)

Id	Req	Checks	How to fix
MDC0601	R	After a retryable close, a new session opens and health reports ready again.	Reconnect after non-fatal closes.
MDC0602	R	That reconnect sends the right resumeSessionId in session.hello and gets resumed: true with the same session id.	Resume inside the window.
MDC0603	R	With a zero resume window, the reconnect gets a fresh session id and eventually becomes ready.	Make re-initialization idempotent.
MDC0604	R	A close with SupervisorShutdown (4004) leaves a managed subject not-live and a self-registering one live.	Exit on 4004 only when managed.

Health endpoint (MDC07xx)

Id	Req	Checks	How to fix
MDC0701	R	While session creation is held, /_macrodeck/health answers and /_macrodeck/ready does not; ready turns true once released.	Report ready only with a session.
MDC0702	R	/_macrodeck/info and /_macrodeck/diagnostics report the negotiated version and declared/accepted capability counts the host recorded.	Report the real session.
MDC0703	R, guard	An unmapped path under /_macrodeck/ answers 404.	Map nothing under the prefix (MDP2005).
MDC0704	R	A health probe at the base address the launcher expects succeeds.	Listen where you are told (MDP4002).

Bounded queues (MDC08xx)

Id	Req	Checks	How to fix
MDC0801	Rec	An invocation that logs still replies while draining is paused, so capability.result is pause-exempt, and its logs arrive after resuming; skips without log output.	Do not block on logging.
MDC0802	Rec	After a log flood while paused, a trailing Error still arrives and Dropped is honest; skips without enough log output.	Log through the SDK pipeline, not a private sink.
MDC0803	R	Every collected log event is within the protocol limits for message length, property count, property value length, source-context length and exception-chain depth; skips before any logs.	Trim oversized log events.
MDC0804	R	The published-event count is the same just before and just after a disconnect/reconnect cycle.	Do not replay events on reconnect.
MDC0805	Rec, guard	The MDC0505 burst, against variables/get on the first declared variable; skips without variables.	Queue a burst, do not fail it.

The report

ConformanceRunner.RunAsync returns a ConformanceReport; ConformanceReportWriter.ToText, ToJson and ToMarkdown render it, and the CLI’s --report Text|Json|Markdown selects the same writers.

Member	Type
SuiteVersion	string (currently 1.2.0)
PluginId, PluginVersion	string?
StartedAt	DateTimeOffset
Duration	TimeSpan
Results	IReadOnlyList<ConformanceCheckOutcome>, one per selected check, in id order
Passed, Failed, Skipped	int; Skipped counts Skipped and Inconclusive together
Conformant	bool

Each ConformanceCheckOutcome carries Id, Title, category, requirement and a ConformanceCheckResult: Outcome, Expected/Actual for a failure, SkipReason for a skip or inconclusive, Duration, and any Observations.

JSON is camelCase, indented, with string enums (trimmed):

{
  "suiteVersion": "1.2.0",
  "pluginId": "com.example.hello-deck",
  "pluginVersion": "1.0.0",
  "startedAt": "2026-09-12T10:27:59.604661+00:00",
  "duration": "00:00:21.5385029",
  "results": [
    {
      "id": "MDC0101",
      "title": "The plugin id is a valid reverse-domain package id",
      "category": "manifestAndIdentifiers",
      "requirement": "required",
      "result": {
        "outcome": "passed",
        "expected": null,
        "actual": null,
        "skipReason": null,
        "duration": "00:00:00.0055956",
        "observations": [{ "label": "plugin id", "detail": "com.example.hello-deck" }]
      }
    }
  ],
  "passed": 25,
  "failed": 0,
  "skipped": 24,
  "conformant": true
}

Markdown (trimmed):

# Macro Deck plugin conformance report

Suite version: `1.2.0`
Plugin: `com.example.hello-deck` `1.0.0`
Conformant: **yes**
Passed: 25 - Failed: 0 - Skipped: 24

| Id | Title | Category | Requirement | Outcome | Detail |
|---|---|---|---|---|---|
| MDC0101 | The plugin id is a valid reverse-domain package id | ManifestAndIdentifiers | Required | PASS |  |
| MDC0103 | Every weather station instance id is a valid resource-kind identifier | ManifestAndIdentifiers | Required | SKIP | This subject does not declare the weather capability. |

See also

• Testing plugins - MacroDeck.Plugin.Testing, which the suite is built on.
• macrodeck-plugin test - options and report formats.
• ADR 0026 - why the suite is framework-agnostic and its ids are a stable contract.
• Plugin protocol - the wire contract the checks assert.
• Analyzers - the compile-time counterparts.