top of page
Writer's pictureMarie Cruz

Run Tests in Parallel using CircleCI



One of the strategies out there to speed up your tests is to run them in parallel especially if you have a large codebase with lots of tests! Fast feedback loop is crucial to speed up development work and if you have automated tests running for a long time, it slows down everyone's work. In this post, I would like to quickly show you how to run your tests in parallel using CircleCI.


CircleCI has a feature to split your tests in parallel by executing them on multiple executors. You can read more about this feature in their documentation, Running Tests in Parallel. In this specific example, I will show how to split your Cypress tests but this will apply to other tests too. This tutorial will assume you are using a paid version of CircleCI as the free version only includes 1 container to execute from. Of course, if you are using the Cypress Dashboard already, then you don't need to follow along. 😀


Initial CircleCI config file

My initial CircleCI config file looks like the one below. Without going into too much technical detail, I have set up two jobs. One to install the project's dependencies and the other to run the tests. These jobs are then called sequentially on a single workflow called `build_and_test`.




Setup Parallelism

Let's first setup the parallelism level on your config file. This parallelism key defines how many independent executors CircleCI will create to run your tests. To do this, you just need to add this key before calling the steps to run your tests. Depending on your CircleCI's pricing plan, the parallelism value can be increased to your requirements.



Easy right? That's the first step! Now in order to split the actual tests, the next step that we need to do is use CircleCI CLI (Command Line Interface) to call additional commands to split our tests.


Split Tests using CircleCI CLI

The commands that we will use are `circleci tests glob` and `circleci tests split`. If you are not familiar with glob, a glob is basically a pattern that will match a list of filenames to a specific pattern that you specify.


The split command by default will split your tests according to a list of filenames but you can also use other strategies such as splitting it by timing data or file size. For this tutorial, I will use the default way.

Image showing four executors each containing its own set of split tests.
Credit: CircleCI

By using these two commands, you can split your tests into multiple executors that are independent of each other as you can see from the image above.


Now, let's look at how we can achieve this in our configuration file.



There's a lot going on at the above code so I will explain it line by line.


First, we create a new temporary directory called `cypress/tmp`. This is needed because this is where the files that are split by CircleCI will be stored.


Next, using the glob command, I am trying to match all filenames that follows the pattern `cypress/integration/examples/*.spec.js` and then we are piping the output to `circleci tests split`. Wait, piping? what is that?! Piping basically means you are redirecting the output to another command 😊


CircleCI will then do its own magic and split the tests and then we'll move the split files to the `cypress/tmp` folder. For this to work, you need to modify your Cypress configuration to point at this specific folder when you run the tests.


Then finally, just call the command that you use to run your tests. To avoid scenarios where the executors don't have tests to run, I added an error handling command that will check if the `cypress/tmp` folder is empty. If it's not empty, then continue and run the tests and that is pretty much it! 🎉


Wrapping up

In this post, I showed one way of running tests in parallel using CircleCI. Applying test parallelism regardless of the technical approach is always great news because it will speed up your build times and even in most cases, cut the times nearly half.


For additional resources on how others have used the parallelism feature, you can check out the following blogs:

2,484 views10 comments

10 Comments


Akhilesh Sinha
Akhilesh Sinha
Feb 15, 2022

Hi Marie,


I am also getting errors after implementing the test split. Command: mkdir cypress/tmp

mv $(circleci tests glob cypress/integration/Folder1/Folder1_1/*/*/*.feature | circleci tests split) cypress/tmp || true echo $(circleci tests glob "cypress/integration/Folder1/Folder1_1/*/*/*.feature") npx cypress run --parallel --record --key $AUTOMATION_KEY --spec cypress/integration/Folder1/Folder1_1/*/*/*.feature Error: You passed the --parallel flag, but we do not parallelize tests across different environments. This machine is sending different environment parameters than the first machine that started this parallel run. The existing run is: https://dashboard.cypress.io/projects/XXXXX/runs In order to run in parallel mode each machine must send identical environment parameters such as: - specs - osName - osVersion - browserName - browserVersion (major) This machine sent the following parameters: { "osName": "linux", "osVersion": "Debian - 9.11", "browserName": "Electron", "browserVersion": "93.0.4577.82", "specs": [ list of 31 specs [*.feature files]

] }

https://on.cypress.io/parallel-group-params-mismatch
Like

Shaheda Hussain
Shaheda Hussain
Jan 18, 2022

Hi Maria,

Thank you for the information. I followed the exact same steps, however the result I am getting is that all tests are running across all 4 machines instead of dividing the tests. Is there something I might be missing? I have 42 tests in total and I see them running across all 4 machines

Like
Akhilesh Sinha
Akhilesh Sinha
Feb 15, 2022
Replying to

Hi Shaheda, Are you still facing the issue or it's working for you now?


Like

jorge.quiros
jorge.quiros
May 11, 2021

I don't know much about docker or Circle CI but: - Shouldn't you copy your specs files, config or anything alike from your Cypress project to the Cypress container? or are you doing that in the circle CI GUI job itself? Thanks!

Like
Marie Cruz
Marie Cruz
May 16, 2021
Replying to

Hi Jorge, for this demo since the project directory is mounted already in CircleCI, I didn’t need to copy any files on top. However, if I want to run my tests from inside a custom docker container, then I would need to copy the files as noted from my previous post here https://www.mariedrake.com/post/using-docker-to-run-your-cypress-tests

Like
bottom of page