Flaky vs broken: how we decide if an install really failed
We install every agent skill, plugin and MCP server we track in Claude Code, Codex, Grok Build, Cursor and Pi every night. In our first week we called things "broken" or "fixed" off a single night. Then we noticed 13 of our 14 "fixed" items had simply flip-flopped. So we changed the rules. Here they are, in plain words.
When does SkillMaxx call an install broken?
Only after it fails two nights in a row, after one automatic retry in a fresh sandbox each night. A single bad night is marked pending, not broken.
What does "flaky" mean on SkillMaxx?
An install is flaky if it flip-flops between pass and fail across the last three runs, or only passes on retry. Flaky installs are labeled, never counted as broken or fixed, and never picked as the standout of the day. In week 2026-46, 13 add-ons were flaky.
What if an install only failed because of a missing API key?
Our sandbox deliberately has no accounts or keys. When the log shows a missing key or credential, we list it as needs your key instead of broken. That was 25 installs in week 2026-46. It will probably work once you add yours.
When is something "really fixed"?
Only when it passes after two or more bad nights. A pass after one bad night is just the flake resolving itself.
Why so strict?
Because "tested nightly" is only worth something if the results are boring most days. Shouting about noise trains people to ignore the real breakage, and it isn't fair to tool authors whose installs failed for reasons that had nothing to do with them.
Tell us if we got it wrong
These rules are new, and we're a small project. If you think a tool is labeled wrong, or the rules should be different, email us. We'll rerun it and reply.
FAQ
When does SkillMaxx call an agent plugin install broken?
Only after it fails two nights in a row, with one automatic retry in a fresh sandbox each night. One bad night is shown as pending.
What is a flaky install?
One that flip-flops between pass and fail across the last three nightly runs or only passes on retry. SkillMaxx labels it flaky and doesn't count it as broken or fixed.
Does a missing API key count as a failure?
No. SkillMaxx tests without any keys, so failures caused by missing credentials are listed separately as needs your key.
SkillMaxx Weekly
The week's install scorecard across all 5 agents, plus new skills, plugins and MCP servers and whatever changed in the official docs. Every Tuesday. (The daily report stays on the site, no daily emails.)
Loading the signup form… or subscribe on Beehiiv
Form not loading? Subscribe on Beehiiv →