Automation and Accessibility Integration (AAI) (Total Control 8+)
Accessibility is a feature in Android that map on-screen UI elements to underlying nodes, the node can represent an UI element (e.g. button) or a group of UI elements or layout of certain elements. A node (element, group or layout) can be identified by a node ID (represent in string of hex). We integrate Accessibility, TC scripting framework and UI Automator library to achieve the following goals:
- Coordinate independent makes the script more portable with different resolutions and multiple brands.
- Synchronous API will wait until the screen is repaint, make the script simpler, do not need to guess the time to sleep. Synchronous API will not slow down the multi-device execution.
- Can retrieve the string from the app with ease, instead of using error-prone OCR.
The simplest case of AAI:
- Click "OK" on the screen if found, far better than click(100, 100) for specific resolution: devices.clickSync("OK")
- Enter text into text entry, position is specified, enter into nth position on multiple text entry lines.
devices.inputTextSync([position], "text") // Enter text, the position is used for multiple inputs
- Run or restart application, without query, it will return on screen refresh; with query, it will match the query after screen is refreshed.
devices.runAppSync(<package name>, [query])
devices.restartAppSync(<package name>, [query])
For more complex scripting, you need to locate node ID or list of node ID by query the accessibility nodes. UI Automator in Java provides "UiSelector" and "BySelector" in UiDevice.findObject() or findObjects() to locate nodes, it can be complex for multiple conditions:
new UiSelector().className("android.widget.TextView").text("OK")
We created a simple query language, that is shorter and portable since need to send this to lots of devices, the above query can be rewritten in:
"C: android.widget.TextView||T:OK"
The query syntax can contain "!" for not, ">", "<" for greater than or less than, "*" for wild card match and "/<regexp>/" for regular expression. It can match package name, class name, resource ID, text, description, child count and input type.
Use UiElement.findObject() to return an object of a node if found, since the object has the device object stored, to execute you do not need to specify device like other action/movement commands.
var obj = UiElement.findObject(device, "T:OK");
if (obj) {
obj.clickSync();
}
If you have multiple devices, it will return array of objects in class UiElementArray, an action will cause all devices to react, the code below will click all devices with text of "OK":
var objs = UiElement.findObject(devices, "T:OK");
if (objs) {
objs.clickSync();
}
You can also retrieve the information from the app, e.g. clock with hour, minute and second:
var hour = UiElement.findObject(devices, "R:" + app + ":id/timeHour").getText();
var min = UiElement.findObject(devices, "R:" + app + ":id/timeMinute").getText();
var sec = UiElement.findObject(devices, "R:" + app + ":id/timeSecond").getText();
print("The current time is " + hour + ":" + min + ":" + sec);
Alternatively, you can retrieve the information from "UI Explorer" to find the query syntax of a node (not the most optimize query). e.g.
To offload the complexity of JavaScript and CPU utilization of Total Control, all the search is conducted in the devices.
The program on the device to parse query, locate nodes and actions to the nodes is called "FindNode". device.sendAAi() and devices.sendAai() are way to communicate with FindNode with one or list of devices. The JS object will be translated to JSON before sending to device, it will return the output. If error is encountered, the return is null, the lastError() contains error message. UiElement is developed based on sendAai().
The command is separated to "preAction" and "postAction" and "postActions", "preAction" is before the search/count is conducted, "postAction" and "postActions" are the action toward the output of the search.
"query" contains the query syntax for the search.
There are 14 different keys to compose a query, they are:
- C: Class name (S)
- R: Resource ID (S)
- D: Description (S)
- T: Text (S)
- IT: Input type (I/S)
- CC: Child count (I/S)
- ID: Node ID returned by other queries (S)
- BI: Bound in: [x, y] or [left, top, right, bottom], if x or y is -1, it will be ignored
- IX: Index (I)
- OX: Offset on x (I)
- OY: Offset on y (I)
- TP: Template (S)
- ON: One node selection (S)
- LT: Line number (I)
- LB: Line number (I)
A simple query example, to obtain the text of Model name, use X offset of 1:
>> device.sendAai({query:"T:Model name||OX:1", postAction:"getText"})
{retval: 'Galaxy S10+'}
FindNode can even detects the fixed icons on the top/bottom of the screen:
>> device.sendAai({query:"LB:-1", postAction:"getText"})
{retval: ['Chats','Calls','Contacts','Notifications']}
The following 3 commands, doing the same thing, click on the "Calls" text:
>> device.sendAai({query:"LB:-1||T:Calls", postAction:"click"})
{retval: true}
>> device.sendAai({query:"LB:-1||IX:1", postAction:"click"})
{retval: true}
>> device.sendAai({query:"LB:-1||T:Chats||OX:1", postAction:"click"})
{retval: true}
Click "Contacts" icon:
>> device.sendAai({query:"LB:-1||T:Contacts||OY:-1", postAction:"click"})
{retval: true}
>> device.sendAai({query:"LB:1||IX:2", postAction:"click"})
{retval: true}
>> device.sendAai({query:"T:Contacts||IX:-1||OY:-1", postAction:"click"})
{retval: true}
Please read FindNode User Guide for more information.