mirror of
https://github.com/Bubberstation/Bubberstation.git
synced 2025-12-17 05:03:28 +00:00
a7d1017da6
Rerun flaky tests automatically, and create issue reports for them (#71519) Adds a new workflow that will try to automatically detect and rerun flaky tests, and create an issue report for them. The detection mechanism is heuristic: if exactly ONE job fails in the CI Suite, then it is assumed to be flaky, and will be rerun. If the next run succeeds, then it will create an issue report for that flaky test if one does not already exist. It will do its best to create a unique but consistent identifier, aided by PRs like #71515. You can find an example here: https://github.com/Mothblocks/ss13-workflow-testing/issues/20. Maintainers can also rename the issue if they wish, it will still be able to find it. While there is a chance for this mechanism to go wrong and create bogus issue reports, it IS possible to easily disable actions, I did it for the stale one just a bit ago. Most likely, this mechanism going wrong is going to be the result of randomness leaking in tests, like random human names, so this can be solved in the tests themselves. I find it extremely unlikely, but in the worst case scenario where this happens often, we can add a way for maintainers to edit the issue report and include a regex to match for runtimes. Just an idea. Includes a few large-ish downloaded logs from past failures that are interesting in unique ways. These are used for tests of the title generator. Co-authored-by: Mothblocks <35135081+Mothblocks@users.noreply.github.com>