UI browser tests can be flaky, and it can be really frustrating to have to re-run a test suite because 1/50 tests failed and most likely, it'll work on the next run. webdriverio offers retries but only at the individual spec level as seen on their docs which doesn't help you re-run all the specs in 1 file. webdriverio lets you hook into the test reporter using Custom Reporters, and we can create a tool that lets us re-run the specs that failed
This is a solution proposed by an answer to this Github issue for webdriverio. Since so many things are pluggable in the framework, we can hook into the test reporter, find all the tests that failed, get their filenames, and then re-run webdriverio with those file names.
finding all the tests that failed:
you can get the right info by responding to the test:fail
and end
events.
You can collect all the test failed events....
this.on('test:fail', (data) => {
this.failures.push(data);
});
Then build a report at the end containing all the failed test files.
this.on('end', () => {
console.log(this.failures);
});
When each test fails, you'll get a test result object that looks like this:
{ cid: '0-0',
uid: 'test1spec0',
event: 'test:fail',
title: 'test1',
pending: false,
parent: 'testfile1',
type: 'test',
file: '',
err:
{ matcherName: '',
message: 'Failed',
stack: 'Error: Failed\n at <Jasmine>\n at UserContext.it (/path/to/testfile1.js:25:7)\n at <Jasmine>',
passed: false,
expected: '',
actual: '' },
duration: 15294,
runner:
{ '0-0':
{ maxInstances: 1,
browserName: 'chrome',
chromeOptions: [Object] } },
specs:
[ '/path/to/testfile1.js' ],
specHash: '3ad3fe8fe4af1ca17782a5b3fdc65e54' }
You can save it as JSON if you want to parse it with jq
or another tool. You can also just grab the info you need, like all the file names with tests the failed:
// Save as JSON
let dir = path.resolve(this.outputDir);
let filepath = path.join(dir, 'failed-suite-reporter.json');
mkdirp.sync(dir);
fs.writeFileSync(filepath, JSON.stringify(this.failures));
// Save filenames
dir = path.resolve(this.outputDir);
filepath = path.join(dir, 'failed-specs-to-rerun.txt');
const specs = _.uniq(_.flatMap(this.failures, (f) => f.specs)).join(',');
fs.writeFileSync(filepath, specs);
That txt file will look like /path/to/testfile1.js,/path/to/testfile2.js
for example.
Re-running failed tests
At this point, we have a listing of all the test files that had failed tests, and those results are saved to a file. Now we'll need a wrapper script that runs our wdio suite, and then re-runs it if it fails.
You can add a script to your repo, and link it to the npm test
shortcut.
wdio --spec all_my_specs/**
test_exit_code=$?
if [[ $test_exit_code != 0 ]]; then
failed_spec_files='./output/failed-specs-to-rerun.txt'
wdio --spec $(cat $failed_spec_files);
exit $?
fi
exit $test_exit_code
Something to be aware of:
If you're using another test reporter already that puts a bunch of files in an outputDir
, such as the JUnit Reporter, re-running the tests will also overwrite the files from the first run.
That means, if you have 50 specs and 1 of them failed, and you re-run the 1 that failed, your JUnit report will only show the 1 test. Depending on how you use your test report, you might have to make a copy of the files in your outputDir
before re-running the tests again.
tl;dr here's a snippet of what you need in a failed test suite reporter.
const events = require('events');
const path = require('path');
const fs = require('fs');
const mkdirp = require('mkdirp');
const _ = require('lodash');
// Pulled from this very helpful Github comment
// https://github.com/webdriverio/webdriverio/issues/2521#issuecomment-367190751
class FailedSuiteReporter extends events.EventEmitter {
static get reporterName() {
return 'FailedSuiteReporter';
}
constructor(baseReporter, config, options = {}) {
super();
this.failures = [];
this.outputDir = options.outputDir || './output/';
this.on('test:fail', (data) => {
this.failures.push(data);
});
this.on('end', () => {
const specs = _.uniq(_.flatMap(this.failures, (f) => f.specs)).join(',');
let dir = path.resolve(this.outputDir);
let filepath = path.join(dir, 'failed-suite-reporter.json');
mkdirp.sync(dir);
fs.writeFileSync(filepath, JSON.stringify(this.failures));
dir = path.resolve(this.outputDir);
filepath = path.join(dir, 'failed-specs-to-rerun.txt');
fs.writeFileSync(filepath, specs);
console.log('Wrote report to ', filepath);
});
}
}
/**
* Expose Custom Reporter
*/
exports = module.exports = FailedSuiteReporter;
Wrapping up
There’s no support for re-running failed suites in webdriverio out of the box which can make re-running flaky test suites take a lot longer. However, using Custom Reporters, we can get the information we need in order to re-run tests on our own. You can grab that test reporter above and include it in your project to start re-running only the test suites that failed.
A few other helpful resources:
-
Seeing errors with Chrome crashing randomly when running your tests in Docker? Check out this post to see how to configure Chrome for docker.
-
Sometimes, you just need to step through your tests with a debugger. See here for how to connect it to VSCode
75 Comments. Leave new
Where to write the below piece of code?
if [[ $test_exit_code != 0 ]]; then
failed_spec_files=’./output/failed-specs-to-rerun.txt’
wdio –spec $(cat $failed_spec_files);
exit $?
fi
exit $test_exit_code
Please tell me
Hi @logakarthick:disqus
You can create a script and use that to run wdio and the retries. You can call it test.sh with
wdio --spec all_my_specs/**
test_exit_code=$?
if [[ $test_exit_code != 0 ]]; then
failed_spec_files='./output/failed-specs-to-rerun.txt'
wdio --spec $(cat $failed_spec_files);
exit $?
fi
exit $test_exit_code
Then you can run ./test.sh
Thanks @disqus_QeVdWrMuwK:disqus .
I getting this error “Error: Error: more than one config file specified” while running this ./test.sh
As I website owner I believe the content material here is really good , appreciate it for your efforts. manoto
My colleague shared your article with me and I found it very useful after reading it. Great article, it helped me a lot. I also hope to make a beautiful website like your blog, hope you can give me some advice, my website:
gate io coins
I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?
Wonderful post! We will be linking to this great article on our site. Keep up the great writing willhill tv
Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me?
I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.
Your article made me suddenly realize that I am writing a thesis on gate.io. After reading your article, I have a different way of thinking, thank you. However, I still have some doubts, can you help me? Thanks.
Can you be more specific about the content of your article? After reading it, I still have some doubts. Hope you can help me. https://accounts.binance.com/fr/register?ref=T7KCZASX
Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me? https://accounts.binance.com/bg/register?ref=V2H9AFPY
Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me? https://accounts.binance.com/bg/register?ref=WTOZ531Y
I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article. https://www.binance.com/pt-PT/register?ref=JHQQKNKN
Your article gave me a lot of inspiration, I hope you can explain your point of view in more detail, because I have some doubts, thank you. 20bet
The Elitepipe Plastic Factory in Iraq serves as a catalyst for infrastructure development, providing the market with superior HDPE, uPVC pipes, and fittings that contribute to the growth and success of various sectors. Elitepipe Plastic Factory
Thank you very much for sharing, I learned a lot from your article. Very cool. Thanks. nimabi
Elitepipe Plastic Factory’s fittings are designed to be user-friendly, facilitating easy installation and ensuring hassle-free maintenance throughout their lifespan. Elitepipe Plastic Factory
hello!,I like your writing very so much! proportion we keep up a correspondence extra approximately your post on AOL? I need an expert in this space to unravel my problem. May be that is you! Taking a look forward to see you.
Your article helped me a lot, is there any more related content? Thanks! https://www.binance.com/lv/join?ref=V2H9AFPY
Thanks for sharing. I read many of your blog posts, cool, your blog is very good. https://www.binance.info/uk-UA/join?ref=W0BCQMF1
Your point of view caught my eye and was very interesting. Thanks. I have a question for you. https://accounts.binance.com/bg/register?ref=T7KCZASX
I admire how this blog promotes kindness and compassion towards ourselves and others We could all use a little more of that in our lives
Can you be more specific about the content of your article? After reading it, I still have some doubts. Hope you can help me.
Your article helped me a lot, is there any more related content? Thanks!
I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.
I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.
Thanks for sharing. I read many of your blog posts, cool, your blog is very good.
Can you be more specific about the content of your article? After reading it, I still have some doubts. Hope you can help me.
Mygreat learning I’m often to blogging and i really appreciate your content. The article has actually peaks my interest. I’m going to bookmark your web site and maintain checking for brand spanking new information.
Your article helped me a lot, is there any more related content? Thanks!
أنابيب الري بالتنقيط في العراق في مصنع إيليت بايب، يتم تصميم أنابيب الري بالتنقيط لدينا لتوفير توصيل فعال للمياه للتطبيقات الزراعية. تم تصميم هذه الأنابيب لتقليل هدر المياه وزيادة إنتاج المحاصيل، مما يعكس التزامنا بتطوير تكنولوجيا الري في العراق. باعتبارنا مصنعًا رائدًا وموثوقًا، يضمن مصنع إيليت بايب أن تكون أنابيب الري بالتنقيط لدينا ذات أعلى جودة، مما يساهم في نجاح الممارسات الزراعية. اكتشف حلول الري بالتنقيط لدينا على elitepipeiraq.com.
Henof I truly appreciate your technique of writing a blog. I added it to my bookmark site list and will
Pink Withney Good post! We will be linking to this particularly great post on our site. Keep up the great writing
dodb buzz I like the efforts you have put in this, regards for all the great content.
SocialMediaGirls For the reason that the admin of this site is working, no uncertainty very quickly it will be renowned, due to its quality contents.
Aroma Sensei You’re so awesome! I don’t believe I have read a single thing like that before. So great to find someone with some original thoughts on this topic. Really.. thank you for starting this up. This website is something that is needed on the internet, someone with a little originality!
Mangaclash Nice post. I learn something totally new and challenging on websites
Your article helped me a lot, is there any more related content? Thanks!
Clochant very informative articles or reviews at this time.
Thanks for sharing. I read many of your blog posts, cool, your blog is very good.
Your article helped me a lot, is there any more related content? Thanks!
I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.
Thanks for sharing. I read many of your blog posts, cool, your blog is very good.
Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me?
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
gluco6 reviews : https://gluco6reviews.usaloves.com/
Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me? https://accounts.binance.com/register?ref=P9L9FQKY
Thanks for sharing. I read many of your blog posts, cool, your blog is very good.
Your article helped me a lot, is there any more related content? Thanks!
Your point of view caught my eye and was very interesting. Thanks. I have a question for you.
Noodlemagazine For the reason that the admin of this site is working, no uncertainty very quickly it will be renowned, due to its quality contents.
Your point of view caught my eye and was very interesting. Thanks. I have a question for you.