mirror of
https://github.com/Bubberstation/Bubberstation.git
synced 2025-12-31 12:01:47 +00:00
4552de36fa
Adds a new workflow that will try to automatically detect and rerun flaky tests, and create an issue report for them. The detection mechanism is heuristic: if exactly ONE job fails in the CI Suite, then it is assumed to be flaky, and will be rerun. If the next run succeeds, then it will create an issue report for that flaky test if one does not already exist. It will do its best to create a unique but consistent identifier, aided by PRs like #71515. You can find an example here: https://github.com/Mothblocks/ss13-workflow-testing/issues/20. Maintainers can also rename the issue if they wish, it will still be able to find it. While there is a chance for this mechanism to go wrong and create bogus issue reports, it IS possible to easily disable actions, I did it for the stale one just a bit ago. Most likely, this mechanism going wrong is going to be the result of randomness leaking in tests, like random human names, so this can be solved in the tests themselves. I find it extremely unlikely, but in the worst case scenario where this happens often, we can add a way for maintainers to edit the issue report and include a regex to match for runtimes. Just an idea. Includes a few large-ish downloaded logs from past failures that are interesting in unique ways. These are used for tests of the title generator.
40 lines
1.2 KiB
JavaScript
40 lines
1.2 KiB
JavaScript
import { strict as assert } from "node:assert";
|
|
import fs from "node:fs";
|
|
import { extractDetails } from "./rerunFlakyTests.js";
|
|
|
|
function extractDetailsFromPayload(filename) {
|
|
return extractDetails(
|
|
fs.readFileSync(`tests/flakyTestPayloads/${filename}.txt`, {
|
|
encoding: "utf8",
|
|
})
|
|
);
|
|
}
|
|
|
|
const chatClient = extractDetailsFromPayload("chat_client");
|
|
assert.equal(
|
|
chatClient.title,
|
|
"Flaky test create_and_destroy: /datum/computer_file/program/chatclient hard deleted 1 times out of a total del count of 13"
|
|
);
|
|
assert.equal(chatClient.failures.length, 1);
|
|
|
|
const monkeyBusiness = extractDetailsFromPayload("monkey_business");
|
|
assert.equal(
|
|
monkeyBusiness.title,
|
|
"Flaky test monkey_business: Cannot execute null.resolve()."
|
|
);
|
|
assert.equal(monkeyBusiness.failures.length, 1);
|
|
|
|
const shapeshift = extractDetailsFromPayload("shapeshift");
|
|
assert.equal(
|
|
shapeshift.title,
|
|
"Multiple errors in flaky test shapeshift_spell"
|
|
);
|
|
assert.equal(shapeshift.failures.length, 16);
|
|
|
|
const multipleFailures = extractDetailsFromPayload("multiple_failures");
|
|
assert.equal(
|
|
multipleFailures.title,
|
|
"Multiple flaky test failures in more_shapeshift_spell, shapeshift_spell"
|
|
);
|
|
assert.equal(multipleFailures.failures.length, 2);
|