Flaky tests

What's a flaky test?

It's a test that sometimes fails, but if you retry it enough times, it passes, eventually.

Automatic retries and flaky tests detection

On our CI, we use rspec-retry to automatically retry a failing example a few times (see spec/spec_helper.rb for the precise retries count).

We also use a home-made RspecFlaky::Listener listener which records flaky examples in a JSON report file on master (retrieve-tests-metadata and update-tests-metadata jobs), and warns when a new flaky example is detected in any other branch (flaky-examples-check job). In the future, the flaky-examples-check job will not be allowed to fail.

This was originally implemented in: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13021.

Problems we had in the past at GitLab

Time-sensitive flaky tests

Array order expectation

Feature tests

Capybara viewport size related issues

Capybara JS driver related issues

PhantomJS / WebKit related issues


Return to Testing documentation