Supporting interoperable probabilistic defense structures through shared attack eval datasets

[johannhof]

As noted in various discussions (most recently in #97), it's unlikely that security challenges around WebMCP can be addressed purely through deterministic specification rules. Browser and/or agent developers will have to deploy custom probablistic defense architectures, such as improved model understanding, "critic" models that prevent misaligned actions, or sophisticated tool security architectures like CaMeL.

This risks being a departure from the traditional Web specification security workflow: New implementers of the specification would not be able to use some of the lessons learned by early adopters and the community in delivering a secure feature.

One thing we can (and should) do is write very good and precise security considerations. However, there are limits to that, particularly in that it is hard to quantify what e.g. "should be resistant to prompt injections" means in practice.

As such, I think this group should embrace the default approach by agent makers to quantify the performance of an agentic system: evaluations. A robust dataset of prompt injection attacks that any implementer of WebMCP MUST (or SHOULD?) prevent with >99% recall would be a good and pragmatic step forward in interoperably enabling security for WebMCP.

Obviously there are a lot of open questions here around both maintenance and architecture for running these evals.

cc @jpagnucco @bvandersloot-mozilla @victorhuangwq

[victorhuangwq]

thanks for raising this. I agree that deterministic spec rules and well documented considerations might never be fully sufficient. evals does (at this moment in time) seem like a pragmatic path forward.

a few thoughts here:
I think a publicly available "eval set" would be useful, both as a security baseline, and also as a way to build cross-vendor developer confidence in webmcp. kind of akin to wpt tests almost.

I would frame the eval as the following three possible layers:

1. sample site(s) - sites that implementing webmcp
2. evaluations - human readable description of what should happen given a specific task on those sites
3. common evaluator system ("critics") - that evaluates the evals

1 and 2 feels close to necessary for this to be meaningful. 3 is worth discussing as well - a fair evaluator can help ensure consistency, but it could encode implicit model preferences at odds of being not vendor neutral (maybe more of a BYOM system?).

on broader framing, I would strong encourage this to be perceived as part of a swiss cheese model - one layer of defence amongst many. Passing the evals shouldn't be conflated with being secure.

some other open questions worth considering:

1. scope beyond security. Security is definitely a pressing concern, but this have broader utility for correctness and capabilities.
2. any thoughts on coordinated disclosure model, especially if it's a via common vector (e.g. spec related etc)?

[johannhof]

1 and 2 feels close to necessary for this to be meaningful. 3 is worth discussing as well - a fair evaluator can help ensure consistency, but it could encode implicit model preferences at odds of being not vendor neutral (maybe more of a BYOM system?).

The nice thing about security-related evaluations is that rating can often be done deterministically - there's a bad type of information to leak or a bad tool call to make which can often just be validated via checking a regex of proposed actions or having an event listener trigger some test failure function. Maybe this is what we should go for in the initial iteration.

on broader framing, I would strong encourage this to be perceived as part of a swiss cheese model - one layer of defence amongst many. Passing the evals shouldn't be conflated with being secure.

Yes, I agree, one question is how do we give other ecosystem implementers the confidence that this is a safe technology for them to implement without building up their own swiss cheese architecture for years to pass these evals. Ideally we could define a deterministic / HITL security model for the spec that would always pass these evals, and then have increased autonomy for agent providers that are willing to ensure that their agents have enough security mechanisms to still pass.

any thoughts on coordinated disclosure model, especially if it's a via common vector (e.g. spec related etc)?

Yes this is a hard question we'll have to figure out - I would assume that the spec evals are generally "lagging" real world findings by quite a bit of time.