I think it's interesting to add what they use it for and why its hard.
What they use it for:
- It's about automated testing against third party services.
- It's not about replicating the product for end users
Why using external services is hard/problematic
- Performance: They want to have super fast feedback cycles in the agentic loop: In-Memory tests. So they let the AI write full in-memory simulations of (for example) the slack api that are behaviorally equivalent for their use cases.
- Feasiblity: The sandboxes offered by these services usually have performance limits (= number of requests per month, etc) that would easily be exhausted if attached to a test harness that runs every other minute in an automated BDD loop.
I think it's interesting to add what they use it for and why its hard.
What they use it for:
- It's about automated testing against third party services.
- It's not about replicating the product for end users
Why using external services is hard/problematic
- Performance: They want to have super fast feedback cycles in the agentic loop: In-Memory tests. So they let the AI write full in-memory simulations of (for example) the slack api that are behaviorally equivalent for their use cases.
- Feasiblity: The sandboxes offered by these services usually have performance limits (= number of requests per month, etc) that would easily be exhausted if attached to a test harness that runs every other minute in an automated BDD loop.