Test your actions

Test with AI, on the Actions tab, shows whether AI agents pick the right action for what your users ask, before you publish. It sends realistic requests to real AI models with your actions exactly as AI apps will see them, and shows where agents get it right, where they mix actions up, and how to fix the descriptions that confuse them.

It’s on Launch, Growth and Scale. When you set up a new app on one, it runs right after Gephyr’s AI picks your actions. Nothing reaches your API: a test stops at the action each agent picks (after at most one look-up, answered with made-up data: see below).

The first test run

The first time, Gephyr’s AI writes about 24 realistic requests from your actions and tests them straight away: most are done with one action, some are easy to mix up, and a few are things your app can’t do. Each is something one call can do, and Gephyr keeps only tests whose inputs your actions accept, so every test can pass. It takes about a minute.

Your tests

Open Tests · Edit under the results to see them. Each test is something a user would ask, plus the action an agent should pick for it.

• Add a test adds your own. Pick the right action, or Nothing (your app can’t do this) for requests agents should say they can’t help with.
• Optionally, fill in the inputs the request states exactly, like limit = 5 for “show my last 5 workouts”. Leave an input empty to skip it, or type * when it just needs a value. Inputs you leave out aren’t checked. A value the action wouldn’t accept (one that isn’t among an input’s choices, or is over its limit) can’t be saved, since no agent could pass that test: the form says which input to change.
• Write more with AI adds another set. An app can have up to 40 tests.

How a test run works

Gephyr AI sends every test to two different AI agents, Agent 1 and Agent 2, each on a model from a different AI company. They’re smaller than most AI apps use, so a pass there is a strong sign the bigger models will pick right too. A run uses your saved actions, including changes you haven’t published yet, listed one tool per action even if your app lets agents search. Save your changes on the Actions tab before you test.

A test passes when the agent picks the right action, with inputs that fit it and the inputs the test states. For a request your app can’t do, it passes when the agent picks nothing or calls request_capability.

Agents often look something up before they change it, as a person would: for “Change pet 332’s category to Cats”, an agent may read the pet first. So when a test expects a change (an action that isn’t Read) and the agent first picks one of your other Read actions, it gets a short, made-up sample answer for that read and one more turn, and that next pick is the one checked. Only one look-up is allowed. Nothing is sent to your API, and Every test notes where it happened, such as “looked up get_pet first”. The same goes for a test of something your app can’t do: an agent may read the record first (“Change jenny22’s email” → look up the profile), and it’s then checked on whether it says the app can’t do it. A test that expects a Read action is checked on the first pick.

If an agent is busy, Gephyr waits and asks it again a few times. If it still can’t be reached, those tests show as Not checked and don’t count either way.

Reading the results

• A score for each agent, such as “27 of 30”.
• Each action’s result next to it in the list, such as “✓ Agents: 6 of 6 right”, or a warning with the action it was mixed up with.
• Mix-ups: an action picked instead of another, nothing picked when an action was right, or an action used for something your app can’t do, with the requests that caused them.
• Suggest a better description: for an action agents mixed up, Gephyr’s AI writes a clearer description that says when to use it and how it differs from the action it was confused with. Use this description saves it as an unpublished change. Test again to confirm it, then publish. Once your app is live, Gephyr AI can also rewrite descriptions for actions agents send wrong inputs to.
• Every test shows each request on each agent.

If you’ve changed your actions since the last run, the results say so. Test again to see how agents do now.

When agents pick the right action but fill its inputs wrongly, the fix is usually in your inputs and docs rather than the description: see Make your API agent-ready.

Allowance

Every plan includes, per workspace each calendar month (UTC):

A run where no agent could be reached isn’t counted, and nor is a set the AI couldn’t be reached to write, or that came back with no test that fits your actions. Writing tests with AI is limited to 10 a minute from one address. Deleting an app doesn’t give back the runs it used. Tests never count as calls. Before your workspace has a plan, the Actions tab shows what Test with AI does and which plans include it.

What’s sent where

Your actions (names, descriptions and inputs, plus the field names a Read action returns, in a made-up look-up answer), your app’s name, description and agent notes, and your tests go to the AI models through Gephyr’s AI providers. No end user’s data is sent, and nothing is sent to your API.

Something missing or wrong? Email hello@gephyr.ai.