AI agents
Procurement questions for an AI-ready development platform
During a demonstration, a development platform looks like a coherent whole: someone gives an instruction, tools are called, and a plausible answer appears. Procurement has to separate that picture into individual commitments. Which ones does the supplier actually accept? Which will your own team have to fulfil? Looking behind the interface is worthwhile before a successful demonstration turns into a dependency that is difficult to unwind.
Establish what is actually being supplied
In my open-source project architecture-kit, I describe contracts, data structures and verification requirements for portable agent systems. It is a specification and verification kit, not an operated platform. It supplies neither a model loop nor executing services or a database. That distinction should also be visible in a commercial proposal, rather than disappearing behind the word integration.
Selecting a profile in the kit identifies applicable requirements. It does not install a service or grant authority. The procurement equivalent is straightforward: for each promised capability, ask which component implements it, who operates that component and what evidence demonstrates that it works. An architecture diagram alone cannot answer those questions, however well its boxes fit together.
Useful contracts are not a running service
A well-defined tool contract makes expectations discussable: which inputs are accepted, which outputs should appear, and what constitutes an error? In architecture-kit, synthetic examples and negative contract tests help examine those expectations. They do not establish that an installation is running, that a security boundary exists, or that the system delivers a particular level of performance in real conditions.
For a purchasing decision, I would request a small evidence matrix. Each important requirement should sit alongside its implementation, its owner and an appropriate piece of evidence. A capability can explicitly remain open. The problem is not an acknowledged gap; it is quietly turning an example into a supposedly delivered property that your operations team later has to create.
Good instructions are not a performance guarantee
My open-source project unix-agent-skills collects short, independently loadable instructions for agents working on Unix-like systems. They cover subjects such as tool selection, verification and security boundaries. Examples describe when an instruction applies and when it does not. Reusable guidance makes working practices inspectable without requiring a particular supplier or adopting that supplier's entire platform as a prerequisite.
The documented status also names limitations: no model comparison had been run for the rewritten instructions, and certain platform-specific paths had not been exercised on their target platform. This is an important procurement lesson. Good documentation and passing routing cases are different evidence from demonstrated improvements in everyday work. Ask what claim each check actually supports, rather than treating all validation as interchangeable.
Separate access to knowledge from correct answers
With mcp-nixos-rust, I maintain a public Rust fork of an existing MCP server. It makes information from the NixOS ecosystem queryable by AI clients, including package and configuration sources and package history. This can ground infrastructure assistance in accessible sources instead of relying exclusively on a model's recollection. Acknowledging the upstream project is part of describing that contribution honestly.
An accessible source still does not guarantee a correct conclusion. Ask how the platform exposes freshness, unavailable sources and contradictory results. The distinction between external queries and functions requiring local tools matters too. A demonstration on the supplier's machine does not establish that the same capability will be available inside your restricted environment, with your actual permissions and dependencies.
Buy the operating responsibility as well
The decisive question is often not whether an agent can complete a task once, but who restores its usefulness after something changes. Dependencies, permissions and information sources evolve. A proposal should identify who investigates failures, who approves updates and which evidence must be renewed after an environment changes. Without those assignments, responsibility can fall into the gap between procurement and engineering.
I would prefer a bounded trial using your own tasks to a broad feature catalogue. Agree in advance which outcomes count and which uncertainties are acceptable. Also record which configurations and operating instructions remain available if you change suppliers. The result is a decision about an operable service, rather than a persuasive image of what that service might eventually become.
Questions for your team or vendor
- Which offered capabilities are implemented today, and which currently exist only as specifications or examples?
- What evidence demonstrates compatibility with our actual platform and permissions?
- How will unavailable, outdated or conflicting knowledge sources become visible to us?
- Which performance claims are supported by real tasks rather than synthetic checks alone?
- Who owns operation, failure investigation and renewed approval after changes?
- Which configurations and operating instructions can we retain when changing suppliers?