Let's say you want to update and test your web app with a CMS regularly and you want to automate this:
Do you prefer n8n like workflows or AI workflows where you describe your issue in detail with specific steps?
Commit
d3071e9
It depends on the technology the app is built with. But I use Laravel Dusk for this kind of thing.
And if you need to build something like that:
- Update to minor version
- ⬎Test critical paths with test suite
- if test fails -> AI Fix ⬏
- Create Branch
- Commit Change
- Open a PR
Github Actions + Claude Code (or codex, or devin, etc, etc).
- Use a trigger that detects a minor release.
- Create a branch and update the packages.
- Run your test suite.
(this is all a simple script of gh Action)
- If you detect a failure, call the AI to pass it the error (or…
So you would not just use an AI Prompt:
"Update all packages to latest minor version, after that run the test suite with npm run test. If we have failing tests fix them and rerun the test suite. If you are done create a branch, commit it and open a PR so I can review"?
But…
Not necessarily. That approach is great for local environments, but for CI/CD… I wouldn’t recommend it.
When using agents in automated pipelines, it’s important to separate the deterministic from the probabilistic.
What’s already 100% deterministic doesn’t need an LLM. Running update commands, creating Git branches, running tests, and opening pull requests are predictable tasks that cost $0. Putting them inside a prompt means paying for tokens and latency just for a model that decides to execute commands that a script can run reliably.
Also, if you give an agent an open-ended “fix this and rerun the tests until it’s green” prompt, when it hits a tricky bug, it might modify or delete your tests instead of fixing the actual code, just to satisfy the prompt. By controlling the loop externally, you run the tests, ensure the test files remain untouched, and set a strict cap on retries before aborting.
Plus, an autonomous agent exploring the repo from scratch every week burns a ton of context just to run a routine update... and if it fails halfway through, debugging a prompt execution is way harder than reading a failed GitHub Actions step.
commit d3071e9
Okey got it! I agree in every point. Thank you for the detailed answer :)