Module 02 · Evaluation and policy
runnableEvals & English Governance
Write the rule in English, then test the route.
Runnable v0.1 slice available on GitHub — architecture and scope below.
Problem
Each model platform has its own eval screens, scoring terms, and policy controls. Teams end up comparing results by hand, while governance rules live in documents that do not map cleanly to provider settings.
Where it sits
Sits across model onboarding and routed model traffic. It is meant to bring hyperscaler evaluation results together and translate plain-English rules into the provider toolsets that enforce them.
This module is designed for teams that already use evaluation features in Foundry, Bedrock, Vertex, or other platforms. It does not replace those systems; it is meant to collect their results into a readable comparison.
A rule such as where a workload may run or what content checks it needs can be written in English, reviewed by a human, and mapped to the relevant provider toolsets. Scores are intended to run when a model is onboarded and again against traffic passing through Smart Router.
How it fits
Evals observes the Smart Router path rather than becoming another gateway. Portable evaluation evidence can join Data Lineage egress records on the same request/trace id.
Capabilities
- Intended dashboard for evaluation results from multiple providers
- Plain-English governance rules with reviewable provider mappings
- Onboarding scores before a model enters routing choices
- Ongoing scoring against sampled or selected routed requests
- Request and trace ID links back to router activity
- Readable evidence for technical and non-technical reviewers
Other modules
- Smart RouterModel access gets scattered across provider keys, endpoints, deployment rules, and separate UIs. Even a VS Code assistant and a home chatbot can end up repeating the same integration work.
- Stage GateAI-assisted projects can jump from an appointment straight into generated code. The missing middle—prerequisites, requirements, decisions, and clear advance criteria—is usually spread across documents and understood by only a few people.
- Shadow VersioningOne project can have its brief in Confluence, approvals in SharePoint, code in Git, and evidence in a testing system. Humans see one piece of work, but the tools expose unrelated histories and technical version language.
- Context EngineAn agent working across 15 repositories can spend more tokens rediscovering the codebase than answering the question. Every tool builds its own partial picture, then throws much of that work away.
- Data LineageOnce a prompt crosses a gateway, it can be hard to explain which provider, region, model, or internal service handled it. Existing lineage tools cover data estates well, but AI request paths often need a smaller, trace-linked view.
- AI-DORAAI coding agents inflate commit counts and PR volume, hiding code churn, review bottlenecks, and rework. Traditional DORA dashboards track deployment frequency without telling you if half the AI-written lines were reverted two weeks later.