Mesmer Accessibility Review

Last year, I was given the opportunity to trial an accessibility tool called Mesmer for doing accessibility audits on mobile devices. Having created an introductory course about accessibility testing, which you can find out more about on this post, Introduction to Accessibility Testing, I'm always trying to learn new tools out there which can help us with making our products accessible. In this blog post, I'll be giving an overview as to what Mesmer does and my honest review on its accessibility feature.

Overview

Mesmer uses RPA (Robotic Process Automation) that crawls your mobile application on iOS and Android devices to speed up accessibility testing amongst other things. It crawls different screens of your application and reports any accessibility issues that do not meet the standard WCAG (Web Content Accessibility Guidelines) rules. The robots that crawl the mobile application are trained using deep learning automation meaning they are trained to distinguish objects on the screen and how to interact with it.

By default, Mesmer automatically runs the accessibility checks against a subset of rules listed under WCAG version 2.1 Level AA. It does not automatically cover all the rules so performing additional screen reader or keyboard accessibility testing is still required to have a better audit or generate a VPAT (Voluntary Product Accessibility Template). Additionally, for inclusive design, it’s also recommended to test with real users to ensure that you catch a wide array of accessibility issues.

Overall, Mesmer does provide a quick leg up on many of the most common accessibility issues that can be added to every build verification run to help developers resolve issues before an app is released. This allows teams to spend more time on the accessibility issues that can only be detected by exploring the application manually.

How does it work?

Upload app build

The first step is to upload a test build of your mobile application. If it requires a user login, then this should also be provided so that the crawler knows what credentials to use.

Screen showing where to upload your app build

Start crawl

Once you have a test build, you can start crawling the build and provide custom configurations such as how long the crawl time should be and on which mobile device. Once these are set, the bots will navigate different screens up until the crawl time you specified.

Screen showing the different screens from the Mesmer test app that the bots have crawled

Accessibility issues

Any screens that have accessibility issues reported will have the accessibility icon at the bottom of the screen.

Various screens from your test application with accessibility issues

To see what the accessibility issues are, you need to click the accessibility button found on the right side of the screen.

Screen showing what the accessibility violations are for a specific object

It highlights which object has the violation, which rule it has failed to meet and more information about this rule. In this example, the label for the input field has a low contrast ratio so it will be difficult for users with low vision to see this label.

If you want to view all the accessibility violations, you can skip selecting a screen and just click the accessibility button on the right side of the screen. This will display all the violations, how many objects have the violation as well as additional information about the different objects such as its attributes, which will be useful for your developers and testers.

Accessibility Report

What I found really useful is the accessibility report that you can generate from Mesmer. The reports can be downloadable as PDF or Google Slides. Mesmer app test report is an example report that you can access.

This report can be shared easily among your team and will also ensure that everyone is informed of what accessibility issues their mobile application have. For example, if you need to inform your leadership team of progress on accessibility, then the report is a great way to start sharing vital information.

Example report from Mesmer showing a summary of how many accessibility issues it found

The report also includes screenshots of screens where the issues were found as well as the object containing the violation.

Example report showing the screens and object containing accessibility issues

As mentioned previously, Mesmer checks the accessibility rules against a subset of rules from WCAG 2.1 Level AA. However, if you need to modify or disable some of the rules, then this is also possible by creating your custom version.

Screen showing how to create a new version of accessibility compliance

Custom configurations include disabling some rules, changing its severity status (e.g. blocker, critical, warning, info), adding any remediation notes and customising the best practices and even updating the links to point to other external accessibility sources.

Screen showing how to update existing accessibility rules

Once the changes are made to this custom version and set as the default, the changes will also reflect on the accessibility reports.

One Report for Both Automated and Manual Testing

While Memser may not automate 100% of the WCAG standard, it does allow testers to mark screens and objects with violations found in manual testing that bring standard remediation steps and best practices. This creates a single source of truth for all the known accessibility violations for a given build and provides developers with additional valuable information like UI control classes, Android fragments / activities, and accessibility attributes.

Plus, it saves a lot of time when taking screenshots, screencasts or writing the same remediation note, over and over again.

CI / CD Integration

Mesmer also has a very robust CLI that allows you to integrate it into your CI/CD pipelines. Jenkins, CircleCI, TravisCI, and GitHub Actions are documented as CI integration examples. The CLI documentation covers all the bases and provides very flexible output both formatted and raw JSON for easy integration into other reporting tools.

Wrapping up

Accessibility testing should be embedded right from the start and nowadays, there are so many tools that can help us achieve this. One of the tools is provided by Mesmer. The ability to crawl different screens quickly just like an actual user would and spot basic accessibility issues can allow teams to fix accessibility issues earlier on. Tools such as Mesmer Accessibility help us in developing and testing accessible products faster. The accessibility reports are a great bonus which can help you and your teams communicate what issues your mobile application have. A gentle reminder though that only a subset of rules are validated against which means actual accessibility testing is still very much important.

If you want to give it a try and get a free accessibility report, please use this Mesmer referral link with promo code as Marie.

Massive thanks to Dan Merritts for being patient with me and showing me how to use Mesmer's accessibility feature 😊