How do you test a GitHub agent before production?
Use a throwaway GitHub repo or organization, then test the states that block automation: branch protection rules, merge conflicts, required reviews, and stale refs. These are the cases where an agent thinks it merged when it did not.
Start with the official test environment
Spin up a dedicated test repository, or better, a test organization, configured with the same branch protection and required checks your real repos use. The GitHub REST API and Building GitHub Apps docs cover the permissions model that makes the test honest.
Where agents actually break
The happy path is the easy part. An agent that can do the simple version of the task will still fall over on the cases below, and these are the ones worth building tests around:
- A push to a protected branch that the agent assumed would succeed
- A merge conflict the agent needs to detect instead of forcing
- A pull request that cannot merge because required reviews are missing
- A stale base branch, where the agent acts on an out-of-date ref
- Secondary rate limits during a burst of API calls
How to test it properly
Give the agent a realistic starting state, let it run the full task end to end, then check the state it left behind rather than the response it returned. Run the same scenario twice to confirm it does not duplicate work or corrupt state on a retry. Then deliberately inject a failure partway through, a rejected write or a rate limit, and confirm the agent recovers instead of charging ahead as if nothing happened.