Rerun flaky tests automatically, and create issue reports for them (#71519)
Adds a new workflow that will try to automatically detect and rerun
flaky tests, and create an issue report for them.
The detection mechanism is heuristic: if exactly ONE job fails in the CI
Suite, then it is assumed to be flaky, and will be rerun. If the next
run succeeds, then it will create an issue report for that flaky test if
one does not already exist. It will do its best to create a unique but
consistent identifier, aided by PRs like #71515. You can find an example
here: https://github.com/Mothblocks/ss13-workflow-testing/issues/20.
Maintainers can also rename the issue if they wish, it will still be
able to find it.
While there is a chance for this mechanism to go wrong and create bogus
issue reports, it IS possible to easily disable actions, I did it for
the stale one just a bit ago. Most likely, this mechanism going wrong is
going to be the result of randomness leaking in tests, like random human
names, so this can be solved in the tests themselves. I find it
extremely unlikely, but in the worst case scenario where this happens
often, we can add a way for maintainers to edit the issue report and
include a regex to match for runtimes. Just an idea.
Includes a few large-ish downloaded logs from past failures that are
interesting in unique ways. These are used for tests of the title
generator.
Co-authored-by: Mothblocks <35135081+Mothblocks@users.noreply.github.com>
* Fix github not announcing new prs
* Resets github_webhook_processor.php to upstream version, with our minor modification
* Adds the whole pull_request_hooks folder that we also missed from upstream
Co-authored-by: Kyle Spier-Swenson <kyleshome@gmail.com>
Co-authored-by: GoldenAlpharex <jerego1234@hotmail.com>
