Rerunning test suites with webdriverio and a failed suite reporter

UI browser tests can be flaky, and it can be really frustrating to have to re-run a test suite because 1/50 tests failed and most likely, it'll work on the next run. webdriverio offers retries but only at the individual spec level as seen on their docs which doesn't help you re-run all the specs in 1 file. webdriverio lets you hook into the test reporter using Custom Reporters, and we can create a tool that lets us re-run the specs that failed

This is a solution proposed by an answer to this Github issue for webdriverio. Since so many things are pluggable in the framework, we can hook into the test reporter, find all the tests that failed, get their filenames, and then re-run webdriverio with those file names.

finding all the tests that failed:

you can get the right info by responding to the test:fail and end events.

You can collect all the test failed events....

this.on('test:fail', (data) => {
      this.failures.push(data);
    });

Then build a report at the end containing all the failed test files.

this.on('end', () => {
  console.log(this.failures);

});

When each test fails, you'll get a test result object that looks like this:

{ cid: '0-0',
  uid: 'test1spec0',
  event: 'test:fail',
  title: 'test1',
  pending: false,
  parent: 'testfile1',
  type: 'test',
  file: '',
  err: 
   { matcherName: '',
     message: 'Failed',
     stack: 'Error: Failed\n    at <Jasmine>\n    at UserContext.it (/path/to/testfile1.js:25:7)\n    at <Jasmine>',
     passed: false,
     expected: '',
     actual: '' },
  duration: 15294,
  runner: 
   { '0-0': 
      { maxInstances: 1,
        browserName: 'chrome',
        chromeOptions: [Object] } },
  specs: 
   [ '/path/to/testfile1.js' ],
  specHash: '3ad3fe8fe4af1ca17782a5b3fdc65e54' }

You can save it as JSON if you want to parse it with jq or another tool. You can also just grab the info you need, like all the file names with tests the failed:

// Save as JSON
let dir = path.resolve(this.outputDir);
let filepath = path.join(dir, 'failed-suite-reporter.json');
mkdirp.sync(dir);
fs.writeFileSync(filepath, JSON.stringify(this.failures));

// Save filenames
dir = path.resolve(this.outputDir);
filepath = path.join(dir, 'failed-specs-to-rerun.txt');
const specs = _.uniq(_.flatMap(this.failures, (f) => f.specs)).join(',');
fs.writeFileSync(filepath, specs);

That txt file will look like /path/to/testfile1.js,/path/to/testfile2.js for example.

Re-running failed tests

At this point, we have a listing of all the test files that had failed tests, and those results are saved to a file. Now we'll need a wrapper script that runs our wdio suite, and then re-runs it if it fails.

You can add a script to your repo, and link it to the npm test shortcut.

wdio --spec all_my_specs/**
test_exit_code=$?

if [[ $test_exit_code != 0 ]]; then
  failed_spec_files='./output/failed-specs-to-rerun.txt'
  wdio --spec $(cat $failed_spec_files);
  exit $?
fi

exit $test_exit_code

Something to be aware of:

If you're using another test reporter already that puts a bunch of files in an outputDir, such as the JUnit Reporter, re-running the tests will also overwrite the files from the first run.

That means, if you have 50 specs and 1 of them failed, and you re-run the 1 that failed, your JUnit report will only show the 1 test. Depending on how you use your test report, you might have to make a copy of the files in your outputDir before re-running the tests again.

tl;dr here's a snippet of what you need in a failed test suite reporter.

const events = require('events');
const path = require('path');
const fs = require('fs');
const mkdirp = require('mkdirp');
const _ = require('lodash');

// Pulled from this very helpful Github comment
// https://github.com/webdriverio/webdriverio/issues/2521#issuecomment-367190751
class FailedSuiteReporter extends events.EventEmitter {
  static get reporterName() {
    return 'FailedSuiteReporter';
  }

  constructor(baseReporter, config, options = {}) {
    super();
    this.failures = [];
    this.outputDir = options.outputDir || './output/';

    this.on('test:fail', (data) => {
      this.failures.push(data);
    });

    this.on('end', () => {
      const specs = _.uniq(_.flatMap(this.failures, (f) => f.specs)).join(',');

      let dir = path.resolve(this.outputDir);
      let filepath = path.join(dir, 'failed-suite-reporter.json');
      mkdirp.sync(dir);
      fs.writeFileSync(filepath, JSON.stringify(this.failures));

      dir = path.resolve(this.outputDir);
      filepath = path.join(dir, 'failed-specs-to-rerun.txt');

      fs.writeFileSync(filepath, specs);
      console.log('Wrote report to ', filepath);
    });
  }
}

/**
 * Expose Custom Reporter
 */
exports = module.exports = FailedSuiteReporter;

Wrapping up

There’s no support for re-running failed suites in webdriverio out of the box which can make re-running flaky test suites take a lot longer. However, using Custom Reporters, we can get the information we need in order to re-run tests on our own. You can grab that test reporter above and include it in your project to start re-running only the test suites that failed.

A few other helpful resources:

• Seeing errors with Chrome crashing randomly when running your tests in Docker? Check out this post to see how to configure Chrome for docker.

• Sometimes, you just need to step through your tests with a debugger. See here for how to connect it to VSCode