How do you add agent testing to a CI/CD pipeline?

Why agents belong in CI

If an agent only gets tested by hand before a release, integration bugs surface in front of users. Putting the agent in continuous integration means every commit is checked against the services it touches, and a regression shows up as a failed check on the pull request instead of an incident later.

The shape of a good pipeline step

1. Provision a fresh sandbox seeded with realistic starting data
2. Run the agent end to end across representative scenarios
3. Assert on final state, not just the agent's response
4. Inject a failure in at least one scenario and confirm recovery
5. Tear the sandbox down so nothing leaks into the next run

Keep runs isolated and fast

Each run needs its own clean environment, or one test quietly depends on state another left behind and your suite goes flaky. A sandbox that resets in seconds is what makes a fresh environment per run practical. Wire it into your runner with GitHub Actions or the CI system you already use, and gate merges on the result.

A GitHub Actions step

Provision a fresh sandbox for the job, run the agent suite against it, and fail the build on any regression:

yamljobs:
  agent-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - name: Run agent suite against a fresh sandbox
        env:
          SANDBOX_URL: ${{ secrets.SANDBOX_URL }}
          SANDBOX_TOKEN: ${{ secrets.SANDBOX_TOKEN }}
        run: pytest tests/agent -q   # non-zero exit fails the PR

The suite resets the sandbox per scenario, so a flaky ordering bug cannot pass by borrowing state from an earlier test.

Validating a whole change, not just your scenarios

Your own suite catches the regressions you thought to write, not the ones you did not. A complementary approach is to deploy the change into an isolated environment, wire its outbound calls to replicas of the real services, and drive the actual user flows end to end.

The idea is that instead of asserting on outputs, you run the full flow against connected service replicas and check what actually happened, the steps taken, the state left behind, and the side effects produced. Because the runner is deterministic and the replicas reset between runs, a failed check points at a specific broken flow rather than flaky infrastructure, which is what makes it practical to block a merge on. Arga Labs supports this if you are already using it for the earlier testing stages.