Gray-Box Mobile Accessibility Testing
I hope we can all agree that mobile phones, and their apps, are a critical component in most people’s lives – interestingly, so critical that in some instances people will go hungry instead of running out of phone credit(1); possibly equating their mobile with safety and security as in “Maslow’s Hierarchy of Needs.”
Quite frankly as such a critical piece in people’s lives the ability for anyone (regardless of disability; or age) to access, and use, their mobile applications, as and when they need, is critically important.
To date, ensuring the accessibility of a mobile application (in whatever form) has, for some at least, been far from easy. With different Accessibility requirements, mainly set at the lower code level, needed on different platforms; along with different testing methods, skills, knowledge, tools and Assistive Technologies (AT).
Explaining the current issues with mobile accessibility testing
Meet fictional Bill; he’s spent the last 5 years testing native mobile applications. He’s responsible for User-Interface (UI) testing. His is a complicated role. In the past he has mainly coded “white–box” tests, adding them into larger groups or “test–suites”. Each test-suite coded in specific computer languages (swift; java). Once satisfied, he has then had to then ensure that those test-suites are included along with the application’s own source code in a specially built “test” variant, for execution.
It’s even more complicated when he’s been asked to test the accessibility of an application. As he’s not an accessibility expert, he has had to rely on third-party code libraries for accessibility checking, and ask that they be included in the source code – then handle all the expected push- back, as you can imagine!
He’s has also had the additional head-ache of having to deal with several sets of UI requirements for accessibility – effectively one per platform. Bill has often complained to colleagues “there’s only one for web; so why can’t there just be one set of requirements for Mobile?”
Over the past couple of years Bill, like many testers, has transitioned to tools which allow mobile phones to be operated, controlled, and tested from the outside – “over-the-wire” so to say. Using these tools Bill has dramatically cut his need to code the low-level “white–box” tests. Instead, he concentrates on writing higher-level tests, which can be used, as Bill describes “amazingly,” on all platforms.
Bill longs for the day he can use these similar higher-level tests for accessibility checking too… To Bill, this just seems like the smart way to proceed.
The purpose of this paper is to consider a new method of mobile accessibility testing, applying so-called “gray–box” testing, which only uses higher-level, cross-platform, tests; and the widely- used testing frameworks / tools used to run them – and does not require any additional code to be “baked–in” a specially built test application!
In addition, this method, being at the higher level, means a single set of higher-level requirements – such as WCAG 2.1 – can be used to assess the accessibility of any mobile UI (be it a native, hybrid or mobile web application).
The hope is to make Bill, and all other mobile testers, very happy – as testing accessibility should now be much easier, and more in-line with the way they work!
So, what is gray-box testing? Let’s say you have a box, which is now sitting on the pre-scan conveyor belt of an airport luggage scanner…
The security guard looks at the outside of the box whilst it rolls up to the scanner. What they are doing is black-box testing: testing of a device, program or system whose workings are completely unknown.
The box rolls into the scanner, and the guard looks at an x-ray representation of what is in the box. That guard is now doing gray-box testing – testing of a device, program or system whose workings are now partially known.
When that box is flagged for opening, and the guard opens the box – then they are doing white- box testing – testing of a device, program or system whose workings are completely known.
Although, just as an FYI, in the context of software testing – white-box testing generally comes before gray-box and black-box testing, as it is done on the source code, rather than via the UI of a compiled application.
Why use gray-box testing over white-box testing…
As we indicated in Bill’s story earlier, in a mobile testing context, white-box solutions generally consist of a set of separate iOS or Android tests which get “baked in” along with the source code during code compilation in order to create a testable version of an application.
Earlier, we also referred to white-box tests as being “lower-level.” This was really due to the fact that they are intended for inclusion within the source code itself. At this lower code level, white- box tests can be used to create (instantiate) objects available within the code itself, for testing. As such, white-box tests have access to:
- each UI component, and its attribute values, in the language of the UI Framework(s) used to create it; and
- specific code-level knowledge about the broader application.
Access to the low level code is of course good, but, when thinking about accessibility testing in particular, the problem is that a good deal of home-grown accessibility testing knowledge / skill is needed in order to obtain meaningful / actionable results.
Developers particularly need to be cognisant of code-level requirements for ensuring the inclusion of mechanisms that expose additional text descriptions, amongst other things, to AT users. As such, developers need to aware of accessibility best practices on iOS(2), and Android(3), which cover platform specific accessibility mechanisms. Things like:
- accessibilityLabel: A succinct label that identifies the accessibility element, in a localized string.
- accessibilityHint: A brief description of the result of performing an action on the accessibility element, in a localized string.
- accessibilityTrait: The combination of accessibility traits that best characterize the accessibility element.
- accessibilityFrame: The frame of the accessibility element, in screen coordinates.
- android:contentDescription XML attribute when labelling static elements;
- the setContentDescription()(4) method for labelling dynamic elements.
Testers, on the other hand, need to understand methods for testing the as-implemented accessibility features on the different platforms. Generally testing them through platform specific testing frameworks like XCUITest(5) on iOS, and Expresso(6) on Android.
And, all of this required knowledge for developers and testers also really dictates at least some appreciation of the ways different ATs on different platforms deal with coded accessibility features.
In teams which do not possess such deep accessibility knowledge, or skills, third-party accessibility testing code libraries are an option.
However, in more recent times we’ve seen the general market’s appetite for such third-party solutions shrink; driven by factors such as security concerns – often associated with the need for these third-party code libraries to be “baked–in” to source code.
In light of this, a need has emerged for low-entry-bar accessibility testing which can be achieved from outside the mobile-application-under-test; and, here, is where gray-box testing comes into the picture; especially as gray-box accessibility testing has been used successfully for some time in a web context.
So, how does gray-box testing work? Firstly in a web context…
To get a high-level view we’ll look at how Selenium, as a widely used gray-box testing tool, is used when testing web UI content.
Why? This will lay a good foundation for our later discussions on mobile UI testing.
This paper was posted by Alistair Garrison from Level Access in ICT Accessibility Symposium 2018