FindNode

FindNode is a shell program on top of the core Selector package, the purpose of FindNode is to find the intended one or more UI elements (or Accessibility nodes) and extract information or perform action on them.

Selector sits on top of Accessibility and UI Automator, each UI element or container of UI elements is identified by node or sets of nodes, each node has its ID that is unique on the current screen. Selector provide various ways to search for one or multiple nodes. once the nodes are acquired, you can do interesting things such as obtain the texts/images or perform action such as click on button or entering the text on the text field, all without coordinates. This allow one automation script to run on different resolutions which is not possible using coordinates. The coordinates are acquired during the runtime.

For example:

JS API: devices.click("OK") instead of devices.click(100, 200). OK query with click will be sent to all devices.

MDCC: Users click a button on the main device, the unique query of the button (e.g. "T:OK") will be sent to other devices, to search for the node and click. The intention is to provide various ways to locate nodes without using screen coordinates, there is no page up/down but "scrollToView" to locate a node, this way, the same script can run on different phones with different resolutions and screen size.

FindNode uses the JSON construct of Appium to carry out the JSON commands, the JSON format is:

{"cmd": "action", "action": "findnode", "params": {…}}

Every FindNode commands are carried out in "params" object.

Return value can have one of two types, successful run:

{"status":0, "value":"{…}"}

Or failure:

{"status":13,"value":"{<error message>}"}

TC provide "device.sendAai()" or "devices.sendAai()" to communicate with FindNode:

  • Send the "params" object.
  • Generate timeout error if FindNode does not return in certain time period.
  • For certain commands that takes longer than the default timeout, FindNode will extend the time to avoid timeout error.
  • Handle return value automatically (null on error, non-null on value).
  • For multiple devices "devices", each device will be done in threads.

The params object contains 3 types of properties: query, preAction and postAction(s). The commands offered are quite rich, you can write a simple automation with them.

Presumably you can do most of the task by using "device.sendAai()" and "devices.sendAai()", FindNode is a "handler", another handler "invoke" can use Java Reflection to access methods in UiDevice, UiObject2, AccessibilityNodeInfo and InteractionController.

Limitations: This is the version 1 of AAI, it comes with some limitations:

  • Support portrait mode only, landscape mode will be introduced in the next version.
  • Not everything will be able to identify at this time, for instance horizontal scrolling or certain line mode.
  • More features will be included.

Query

Majority of the FindNode usages is about query, it allows users to locate intended UI elements via different kind of useful queries on the Accessibility nodes. The entire screen is composed by many nodes, a node can be a smallest UI element or a container of smaller nodes, many are invisible. The entire screen is a tree structure starting from the single root node. Query will be performed to obtain the nodes that meet the criteria, the intent is to reduce the large number of nodes to one or few intended nodes, users can obtain information or apply actions to the nodes.

We developed a small query language (for lack of better word) that is portable among devices and powerful enough to locate most of the nodes. The format is JSON:

{query:"<key>:<value>||<key>:<value>||…"}

For historically reason, although we are using "||" to separate different criteria, "||" is AND relationship, that means it needs to meet all criterion to become a "matching node". FindNode does not offer OR relationship but the underlying Selector package provides that capability. "templates" use that construct.

Query String

There are 14 different keys, they are:

  • C: Class name (S)
  • R: Resource ID (S)
  • D: Description (S)
  • T: Text (S)
  • IT: Input type (I/S)
  • CC: Child count (I/S)
  • ID: Node ID returned by other queries (S)
  • BI: Bound in: [x, y] or [left, top, right, bottom], all integers, if x or y is -1, it will be ignored
  • IX: Index (I)
  • OX: Offset on x (I)
  • OY: Offset on y (I)
  • TP: Template (S)
  • ON: One node selection (S)
  • LT: Line number (I)
  • LB: Line number (I)

All these queries are optional, you can pick and choose any field, if you are doing the query on the screen. The "(S)" stands for String, "I" integer, "[x, y]" array. I/S can accept both integer and String. String is more powerful.

The keys are case insensitive, we use uppercase to differentiate keys from values.

Special Characters in Query

For number such as inputType or childCount, it can either accept number or string:

  • <number>, "<number>" or "=<number>": Match exacting the number.
  • "> <number>": Match number greater than <number>.
  • "< <number>": Match number less than <number>.

String can be one of the following:

  • "<string>": Exact match on full text, case sensitive.
  • "(?i)<string>": Match in case insensitive.
  • "*<string>": Match string at the end, e.g. "*Group".
  • "<string>*": Match string at the beginning. e.g. "android.widget.*".
  • "*<string>*": Match any substring "*widget*".
  • "/<regex>/: Match any regular expression. E.g. "/ImageView|TextView/"

"!" is use at the beginning of string as NOT. "!0" = not zero. "C:!/Layout/", non-layout classes.

For simplicity in examples, the standard prefix of {"cmd": "action","action": "findnode", "params":{…}} will be omitted, we will list the examples in the "param" object and use JavaScript object notations to reduce number of double of quotes.

Shortcuts

Two shortcuts are available for class name (C) and resource ID (R), prefix with "." for substitutions of commonly use prefixes:

  • Class name (C): Prefix "." as substitution of "android.widget.". E.g. ".Button" = "android.widget.Button".
  • Resource ID (R): Prefix "." as substitution of "<package name>:id/". E.g. ".text_connect" = "com.sigma_rt.totalcontrol:id/text_connect".

Bound In (BI):

Bound in takes screen coordinate or bounds and return the nodes based on the criteria below. This mainly for screen manipulation such as UI Explorer or debugging purposes.

  • [x, y]: Will return the nodes with bounds contain (x, y).
  • [x, -1] or [-1,y] where one of them is -1: if x = -1, will ignore x and return nodes with bounds that contains y, similarly for y = -1, it will only match x. As if drawing a horizontal/vertical line on the y and x location, the nodes that touch the line will be returned.
  • [x1, y1, x2, y2]: Will match the nodes bounds within the rectangle specified by (x1, y1) - (x2, y2).

The last 2 modes are supported in 8.0-u40.

Example:

{query:"BI:[-1, 1000]||IX:-1", postAction:"click"}
{query:"BI:[0, 0, 1000, 1000]", postAction:"getBounds"}

Index (IX), Offset (OX/OY) and Line (LT & LB)

These features are useful:

Index: IX:<number>. Return a node from a resulting node list based on the position starting from zero. IX can be negative value in which case the position is in reversed order (-1 is the last node).

Offset: OX:<integer> and OY:<integer>. Use the offset to find neighboring node base on resulting node, OX find horizontal position, positive integer moves right, negative integer moves left. Similarly for OY, positive integer moves down, negative integer moves up. If both options are present, the options will apply based on the position on the query. For example: "OX:1||OY:2" will apply horizontal before vertical offset where "OY:2||OX:1" will apply vertical before horizontal offset. The order is important to locate asymmetrical arrangement of the nodes.

Line: LT:<Integer> and LB:<Integer>. For query with line mode, FindNode will analyze the screen, trying to locate the UI elements outside of scrollable region, line mode will group UI elements into a series of lines, each line can have one or more nodes. Many applications have fixed UI elements on the top and bottom of the screen, this can be easily located by using query such as "LT:1||IX:2" (2nd node of the top line) or "LB:1||T:Chats" bottom of the line, with Chats as text.

  • For non-scrollable apps such as calculator, LT and LB will always return null.
  • Between bottom of the status line and top of the scrollable region is considering as top region, LT starts from 1, "LT:1" is first line. LT:-1 is last line in that region.
  • Between bottom of the scrollable region and top of the navigation bar is considering as bottom region. Similarly, LB starts from 1, LB:1 is first line. LB:-1 is last line in that region (also the last line of the application).
  • Using out of bound line number will get null. (E.g. 2 lines at the bottom, LB:3 will get the null).
  • Line inside in the content will be supported in the future.
  • If one node height is similar to 2 vertical nodes height on the same line, the lower node will be placed in next line.

This can lead to complex query, e.g.

{query:"LB:-1||IX:2||OX:1||OY:-1"}

For line mode, the "intersectY" of the first node is applied of every line, usually it is fine, but in the below cases, it will return all nodes:

So "LT:1" will return all 6 nodes.

For non-scrollable applications, "LB" will return null but it is not hard to get the nodes:

You can use text as anchor:

>> device.sendAai({query:"T:Speed||ON:last", postActions:["intersectX", "getText"]})
{count: 2, list: [{count: 4},{retval: ['Speed','Video','VPN','Map']}]}
>> device.sendAai({query:"T:Speed||ON:last||OY:-1", postActions:["intersectX", "getIds"]})
{count: 2, list: [{count: 4},{count: 4, ids: ['181ce','19c15','1b65c','1d0a3']}]}

Template (TP)

For lack of better name, we developed several predefined templates that is built-in FindNode, the default template is "more".

  • all: no constrain, return all nodes.
  • more: "all" option with layout nodes removed. This is the default if no template is defined or empty query.
  • basic: "more" option with child count of zero.
  • anyText: search for nodes with content in text or description.
  • textInput: set with input type > 0 or class with ".EditText", attempt to find text input line.
  • findText: find certain text in text or description, the text to search will need to be defined in JSON "text". Example:

Example:

{query:"TP:findText", text:"OK", postAction:"click"}

Single Node Selection (ON)

There are query you need multiple nodes (e.g. retrieve stock quotes), there are times single node is needed, for example, click a node or enter a text, FindNode offers "IX" to pick a node from a specific position, ON provides multiple ways to pick a node.

These are the options:

  • first: Take the first node from the matching list (Similar to IX:0).
  • last: Take the last node from the matching list (Similar to IX:-1).
  • min: Pick the smallest node (based on area) from the matching list.
  • top: For overlap nodes, pick the top most node. Useful for "BI" query with x and y coordinate.

ON and IX have to be mutually exclusive.

Query Precedence

Below are the query precedences:

  1. All nodes or line mode (LT/LB).
  2. Template (TP).
  3. Index (IX) or one node selection (ON).
  4. Offset OX and OY.
  5. If multiple nodes, sort.

"elements" property

"elements" accept an array of node IDs, the node IDs must be a valid node ID present on the screen or it will generate error. When "elements" is specified, the search will not be performed, however, IX, OX, OY, ON will still work. It is useful for debugging purpose and for scripts with multiple operations are applied to the same nodes. Each query with "elements" will traverse all nodes to locate corresponding node, try to put multiple IDs in the array instead of one at a time.

device.sendAai({elements:["1234a", "5678b"], postAction:"getNodes"})

var ids = device.sendAai({query:"C:.Button"}).ids;
var retval = device.sendAai({elements:ids, postAction:"getNodes"});

Synchronous actions

Great efforts have been put in to ensure the actions are synchronous, when a "click" is called, not only ensure the click is pressed synchronously, it will wait for certain "events" to ensure the rendering is "started" to happen, cannot ensure the rendering is completed. FindNode offers several ways to performed a click:

  • "click": it should work most of the time.
  • "clickForNewWindow": if click opens a new window and takes time to render, use this action.
  • "waitSelector": if the new window takes very long time to complete the rendering, use this command to check if the node is available, add postAction:"click", will click when the node is found and ready.
  • "waitQuery" can be included in postActions, wait for node to appear before actions are carried out.

Popup window

There are 2 types of popup windows, they appear the same but they are implemented differently:

  • The popup window changes the root node of node tree, this will be handled correctly (e.g. Whatsapp).
  • The popup window created a new branch in the node tree, the resulted nodes will be accumulated nodes of main window and popup window, may likely find the wrong nodes. FindNode will monitor the events when this happens, will correctly predict the top branch of the node tree when popup window is opened and closed. When you start FindNode, make sure the application is not in the popup mode (Google Mail, Google search).

For second type of popup window, FindNode can identify some of the popup windows (Many Google applications use this type of popup), If the popup window breaks the FindNode, it will exhibit with one the following behaviors:

  • Nodes not found (even "TP:all") in query.
  • Partial nodes are displayed in UI Explorer.
  • Accumulate the nodes from popup window and the main window. UI Explorer will show a lot of bounds that do not make sense (nodes from main window).

When this happens, close (remove from memory) and reopen application, if popup windows does not work, please contact support.

This version, the horizontal scrolling (display new page by swipe left and right) will not work (e.g. Slack), it will be available in next version.

Testing

To find the nodes via Query is not very difficult, you need is "UI Explorer" and Terminal:

  • Node ID is important.
  • Terminal: Type: device.sendAai({query:"…"}) to find if it matches the node. There are many search options, there is more than one way to find nodes.
  • UI Explorer:
    • Find bounds from different nodes.
    • Find class name, resource ID, description and text. Can help you to develop query.
    • The "Code" in UI Explorer is a best effort to locate a node, not the most optimize way. May break if the node has no assigned resource ID or text (use index "IX" is problematic), refer to postAction "getUniqQuery" for more information.
  • Terminal: When in doubt, use device.sendAai({elements:["<ID>","<ID>"…], postAction:"getNodes"}), this will help you to obtain more information.
  • Start with one device using "device.sendAai" change to multiple devices via "devices.sendAai", you can search the devices by using Device.searchObject() to locate the devices you need (e.g. all devices, devices in a group, specific devices).

Query in Action I

The most basic query is "{}" or in TC:

var output = device.sendAai({})

This returns most of the nodes on the screen and the default return is a list of nodes ID:

{count: 89, ids: ['80006cbe','7440','7bc2','7f83','8344','8705','8ac6','8e87','1f6e7',…]}

You can do something like:

{query: "CC:!=0"} or {query:"CC:>0}: Non-zero child count.

{query:"IT:>10000"}, accept custom input type value that is greater than 10 thousand.

{query:"CC:!=0||IT:>10000"}, find nodes with non-zero child count AND input type is greater than 10,000.

{query:"TP:textInput||IX:2", postAction:"setText", input:"Hello"}, find third text field and enter "Hello" in the text field.

{query:"T:Input text here", postAction:"setText", input:"Hello"}, some empty text field has a label, search the text label and enter "Hello".

{query:"C:.TextView||T: Contacts"}

{elements:["abcd", "123e", "234f"], query:"IX:1||OY:1", postAction:"click"}

Query in Action II

Example 1: Finding nodes on the bottom of the screen:

>> device.sendAai({query:"LB:-1", postAction:"getText"})
{retval: ['Chats','Calls','Contacts','Notifications']}

By using offset (OX), index (IX) or query such as "T:", you can locate almost every node, below 3 examples on click "Contacts" icon:

>>  device.sendAai({query:"LB:1||IX:2", postAction:"click"})
{retval: true}
>> device.sendAai({query:"LB:-1||T:Contacts||OY:-1", postAction:"click"})
{retval: true}
>> device.sendAai({query:"LB:-1||IX:0||OX:2||OY:-1", postAction:"click"})
{retval: true}

The purpose of the line mode is to control the icons on the top/bottom. The red counter intersects with Chats icon will be removed. So for the line of icons, it is 4 nodes instead of 5 nodes:

>>  device.sendAai({query:"LB:1"})
{count: 4, ids: ['14735','17080','18ac7','1a50e']}

If you want the counter, use other query (IX:-1 is to prevent finding other "Chats" on the screen), the second line below will work since offset will not ignore intersected nodes:

>>  device.sendAai({query:"T:Chats||IX:-1||OY:-1||OX:1", postAction:"getText"})
{retval: '1'}
>> device.sendAai({query:"LB:-1||T:Chats||OY:-1||OX:1", postAction:"getText"})
{retval: '1'}

Example 2: Retrieve information from About phone:

To obtain the information:

>> device.sendAai({query:"T:Model name||OX:1", postAction:"getText"})
{retval: 'Galaxy S10+'}

Alternatively, you can use UiElement class:

>> var obj = UiElement.findObject(device, "T:Model name||OX:1")
UiElement: 3d012
>> obj.getText()
Galaxy S10+

Example 3: Assuming you want to first page stock tickers from Yahoo Finance:

device.sendAai({query:"R:.ticker", fields:"T", postAction:"getNodes"}).list.forEach(
    function(p) {
    print(p.text + ":" + 
        device.sendAai({query:"OX:1||T:" + p.text, fields:"D", 
            postAction:"getNodes"}).list[0].description);
})
INTC:53.14
WMT:142.00
XOM:65.93
BIDU:146.53
PEP:173.23

Example 4: Enter text in Skype text box:

Before entering text:

After text is input:

This can be done by using in one query command:

var input = "Hello";
device.sendAai({query:"LB:1||TP:textInput", postActions:[aaix("setText", input), "addQuery('OX:2')", "click"]});

aaix("setText", input) will return 'setText("Hello")', much better than 'setText("'+input+'")', useful for multiple arguments.

The following function to find the person, click, send a message and return back. The 2 "back" keys are required, the first back key is to dismiss the keyboard, second back key is to go back to main page.

function sendAai(device, obj) {
    var retval = device.sendAai(obj);
    if (retval == null) {
        throw "Error on: " + JSON.stringify(obj) + ":" + lastError();
    }
    print("$ " + JSON.stringify(obj) + "\n" + JSON.stringify(retval));
    return retval;
}

function sendSkype(device, name, text) {
    sendAai(device, {query:"T:" + name, preAction:"scrollToView", postAction:"click"});
    sendAai(device, {query:"TP:textInput", postActions:[aaix("setText", text), "addQuery('OX:2')", "click", "sendKey('back')", "sendKey('back')"]})
}

sendSkype(device, "John", "I will get back to you");

Example 5: On the first example, you can use "T:<text>" and offset to click the icons, some applications do not provide the text:

Since this is display on the last line, we can use "LB:-1" to return the nodes, then use "IX" to locate a single node from the list of nodes, to click the search icon:

device.sendAai({query:"LB:-1||IX:2", postAction:"click"})

Example 6: In certain chat window, it shows the name (or phone number).

>> device.sendAai({query:"LT:1"})
{ids: ['1637ca','163f4c','165d54','166115','1664d6']}

To go back:

>> device.sendAai({query:"LT:1||IX:0", postAction:"click"})
{retval: true}

To obtain the name:

>> device.sendAai({query:"LT:1||IX:1", postAction:"getText"})
{retval: '86278'}

Example 7: Locate and click with ease.

To click the arrow, use:

>> device.sendAai({query:"T:Channels||OX:1", postAction:"click"})
{retval: true}

Query in Action III

Example 1: Consider the following calculator in UI Explorer:

To use this calculator can be very simple:

function calc(device, app, resultQuery, formula) {
    device.runAppSync(app);
    formula.split("").forEach((n)=>device.clickSync(n));
    return device.aaiGetText(resultQuery)[0].text;
}
var result = calc(
    Device.getMain(),
    "com.dalviksoft.calculator",
    "R:*result",
    "1230÷567="
);
print(result);

This will give you the result of 1230÷567.

To retrieve all the digit buttons:

>> device.sendAai({query:"T:/[0-9]/||C:.Button", preAction:"doSort", postAction:"getText"})
{retval: ['7','8','9','4','5','6','1','2','3','0']}

Retrieve digit 5-9, sort is based on the location of the nodes:

>> device.sendAai({query:"T:/[5-9]/||C:.Button", postActions:["sort","getText"]})
{count: 2, list: [{retval: true},{retval: ['7','8','9','5','6']}]}

To retrieve 5 buttons of arithmetic operations + DEL buttons, there are several ways to do it, one way is to use bounds on x:

function getNodes() {
    var ret = device.sendAai({query:"T:DEL||C:.Button", postAction:"getNodes", fields:"B"});
    if (ret == null) {
        print("Error:" + lastError());
        return null;
    }
    var rect = new Rect(ret.list[0].bounds);
    var ret = device.sendAai({query:"BI:[" + rect.centerX() + ",-1]||C:.Button"})
    if (ret == null) {
        print("Error:" + lastError());
        return null;
    }    
    return ret.ids;
}

You can do the operation 2+3 here manually, not to search every time to save CPU, use "elements", "=" does not have string representation, so use OX:1 from "0".

var opIds = getNodes();
var numIds = device.sendAai({query:"T:/[0-9]/||C:.Button", preAction: "doSort"}).ids;
device.sendAai({elements:numIds, query:"IX:-2", postAction:"click"});
device.sendAai({elements:opIds, query:"IX:2", postAction:"click"});
device.sendAai({elements:numIds, query:"IX:8", postAction:"click"});
device.sendAai({query:"T:0||C:.Button||OX:1", postAction:"click"});

Alternatively, you can enter text on the text field. Can use UiElement to do that:

var obj = UiElement.findObject(device, "TP:textInput");
var equal = UiElement.findObject(device, "T:0||C:.Button||OX:1");
obj.setText("2+3");
equal.clickSync();
var result = obj.getText();
print(result);

Example 2:

Use "checked" to manipulate the above checkboxes.

Change Sunday from on to off, see the value before and after:

>> device.sendAai({query:"D:Sunday||C:.CheckBox", postActions:["getChecked", "setChecked(false)", "getChecked"]})
{count: 3, list: [{retval: true},{checked: false, retval: true},{retval: false}]}

"intersectX" will return all nodes with Y coordinate intersect with matching node, in this case all checkboxes. Use "setChecked()" to set all nodes to true/false (on/off), it only applies to checkable nodes, none checkable nodes will be ignored. Only click checkbox if the value need to be changed.

>> device.sendAai({query:"D:Sunday", postActions:["intersectX","getIds","setChecked(true)"]})
{count: 3, list: [{count: 7},{count: 7, ids: ['9c52d','9c8ee','9ccaf','9d070','9d431','9d7f2','9dbb3']},{changedCount: 7, retval: true}]}

Example 3: The order of OY and OX is important, below is the developer option:

"Disable adb authorization timeout" does not have node on the right side so apply offset on X first will get null:

>> device.sendAai({query:"T:Disable adb authorization*||OX:1||OY:1"})
null
>> lastError()
Offset out of range

If apply offset on Y first will get proper node:

>> device.sendAai({query:"T:Disable adb authorization*||OY:1||OX:1"})
{count: 1, ids: ['40e9c4']}
>> device.sendAai({query:"T:Disable adb authorization*||OY:1||OX:1", postAction:"click"})

Example 4: You can use one AAI command to enter emoji symbol, here is an example of Messenger:

>> device.sendAai({query:"LB:-1||TP:textInput||OX:1", postActions:["click", "addQuery('OY:1')","intersectX", "addQuery('IX:3')", "click", "addQuery('OY:1')", "intersectX", "addQuery('IX:0')", "addQuery('OX:2||OY:3')", "click", "sendKey('back')"]})
{count: 11, list: [{retval: true},{count: 1},{count: 5},{count: 1},{retval: true},{count: 1},{count: 7},{count: 1},{count: 1},{retval: true},{retval: true}]}

In order to click an emoji, 3 clicks are required:

query:"LB:-1||TP:textInput||OX:1" - This select the icon on the right of textfield in last row (blue face icon).

"click" – Click on the node to open the page.

"addQuery('OY:1')" – Use "OY" to skip to the next line. In this case, it will likely land into emoji icon, just to be safe, we use the following way to ensure the emoji icon is found.

"intersectX" – Will return the nodes on that line:

"addQuery('IX:3')" – Pick the 4th node in that line (emoji icon).

"click" – Click the node to change the page.

"addQuery('OY:1')" – Go to next line.

"intersectX" – Return the nodes on that line:

"addQuery('IX:0')" – Find the first node (top left) of the emoji as reference node.

"addQuery('OX:2||OY:3')" – Interest on the 4th line and 3rd node.

"click" – Click the node to enter the selected emoji.

"sendKey('back')" – Send the Back key.

Wondering which node or nodes are selected or generated, open UI Explorer and add "getIds" to find out the node ID in UI Explorer.

preAction, postAction and postActions

The main purpose of the FindNode develop query to find nodes, the "preAction" happens before search "postAction" and "postActions" happen after the search or nodes made available by preAction, the actions performed will be on the resulting nodes. For action that needs one node (e.g. "click"), the first node in the resulting node will be used (ON, IX, OX and OY can change that). There are many commands in pre and post actions, some commands require additional arguments (the arguments will be enclosed in parenthesis). Currently, the following types of arguments are supported:

  • "String" or ‘String': String enclosed by single quote or double quote.
  • true/false: Boolean value.
  • Number: Number with or without decimal (depending on the types of actions).

A few examples:

postAction:"getNodes('C,R')"

If you want to use variable in the action commands, use one of the following 3 options:

var input = "Hello"
>> "setText('" + input + "')"
setText('Hello')
>> 'setText("' + input + '")'
setText("Hello")
>> aaix("setText", input)
setText("Hello")

postActions:[aaix("setText", input), "addQuery('OX:2')", "click"]

preAction

"preAction" is used one of the following conditions, if the resulting output is one or mode nodes, it will proceed to the postAction otherwise the output will return to the caller.

  • Add more conditions to the existing query (e.g. getCount or doSort).
  • Obtain nodes that cannot be done by query (e.g. getFocus).
  • Action or information that is not related to nodes or query (e.g. waitForWindowUpdate, sendKey).


bgQuery:*

These commands are for background query, this is used by TC to support "addQueryListener()", will invoke callback when the matched is found.



getFocus

Return the node receiving focus (usually text field) or null with error if no node gains focus. If node is found, it will proceed to postAction.

Return: null if no focus is found otherwise procced to postAction(s).

Example: To enter the next line from the existing line

{preAction:"getFocus", query:"OY:1", postAction:"setText('Hello')"}


getCount

Return the number of matched nodes instead of the list of the node IDs. It should be in the postAction, however, getting is count is faster than the list of node IDs, this command is used to determine the count or node search. Return: Return the number of matching nodes, no postAction(s) will be performed.

Return: Return the number of matching nodes, no postAction(s) will be performed.

Example:

>> device.sendAai({preAction:"getCount", query:"R:.ticker"})
{count: 6}


scrollToView

Accept query string (in "query"), will try to match the query by scrolling up/down, when one or multiple nodes are found, it will proceed to postAction, if the match cannot be found, it will return an error. The scroll will use page up/down with the maximum of 10 pages, the scroll may take some time, this command will automatically lengthen the timeout in "sendAai" until it is done. Several options to determine how to the scroll is peformed, "showOnScreen" in postAction can be used to ensure the node is fully visible. On failure, to go back to the original location, run scrollToView in reverse direction.

preAction:"scrollToView()"

Arguments:

<from-direction> can be one of the following, if not specify, the default is "fromCurrentDown".

"currentDown": Starting from current location and search by scrolling down.

"currentUp": Starting from current location and search by scrolling up.

"topDown": Fast scroll to the top of page and search by scrolling down.

"bottomUp": Fast scroll to the bottom of page and search by scrolling up.

Return: Will return null only if the node cannot be found otherwise it will proceed to postAction or postActions.

Example:

{query:"T:Bob", preAction:"scrollToView('fromTop')", postActions:["showOnScreen", "getNodes('all')", "click"]}


waitForWindowUpdate

Waits for a window content update event to occur. If a package name for the window is specified, but the current window does not have the same package name, the function returns immediately.

waitForWindowUpdate(<package name>, <timeout>)

waitForWindowUpdate(<timeout>)

Arguments:

"packageName: <string>" (Optional): If no package name is specified, the current package name is used.

"timeout:<integer>" (Optional): Wait until timeout is expired in ms. If no timeout is specified, 3000ms is used. Maximum timeout limits to 20 seconds.

Return:

true if a window update occurred, false if timeout has elapsed or if the current window does not have the specified package name.



waitSelector

With this mode, if search on the specified "query" is failed, it will retry every 500ms until the nodes are found and proceed, if the timeout expired, it will return error.

waitSelector(<timeout>)

Arguments:

timeout: timeout in ms for the search. "timeout" has a limit of no more than 20 seconds.

Example:

device.sendAai({query:"T:Done", preAction:"waitSelector(10000)"})

postAction and postActions

After the search (or nodes from preAction), if the search cannot be found, an error will be generated. If preAction:"doCount" is specified, the number of matches will be return. The matching nodes will be available for "postAction" or "postActions", these commands will take the resulting nodes and perform actions on them, certain actions only accept one node, the first node will be applied. "postAction" will take one command and return the output as a JSON object. "postActions" will take an array of commands and return an array of output as a JSON object. Some commands can work with multiple nodes (M), some commands generate more nodes (intersect) and if commands only work in one node (O), the first node will be used. For single node actions, if first node is not desire, use addQuery with ON, IX to change that. The postActions order is important.

device.sendAai({query:"T:OK||IX:-1", postAction:"click"});
device.sendAai({query:"R:.tickers", postActions:["refresh", "getNodes('T,D')"]});
device.sendAai({query:"R:.tickers", postActions:["getNodes", "sortX", "getNodes"]});

See example above to see postAction/postActions in action.



getIds (M)

This is the default postAction if no postAction is defined, this command will return a list of matching nodes and counter of matching nodes:

Return: count and array of IDs.

Example:

>> device.sendAai({})
{count: 26, ids: ['7708','d8a2','e7a6','ef28','824b','860c','89cd','8d8e','914f','9510','98d1','9c92','a053','a414','a7d5','ab96','af57','b318','b6d9','ba9a','be5b','c21c','c5dd','c99e','cd5f','d120']}


getNodes (M)

This is used to retrieve the information of nodes. The fields can determine what to display. "getNodes" will refresh the node before retrieving information.

getNodes

getNodes(<fields>)

Arguments:

fields: The optional "fields" is used to define the fields to return, multiple fields in "fields" are separated by comma. The valid field identifiers are:

  • P: Package name (S)
  • C: Class name (S)
  • R: Resource ID (S)
  • D: Description (S)
  • T: Text (S)
  • IT: Input type (I)
  • CC: Child count (I)
  • RI: RangeInfo, provide more information about the type, min, max, current of widget such as SeekBar (slider), use "setProgress" command to change the value.
  • BP: All boolean properties in the node.
  • B: [left top][right bottom] define node area. (S)

Return: Count and array of node information.

>> device.sendAai({query:"T:/[0-9]/", postAction:"getNodes('R,T')"})
{count: 10, list: [{id: '98d1', resourceId: '.button_seven', text: '7'},{id: '9c92', resourceId: '.button_eight', text: '8'},{id: 'a053', resourceId: '.button_nine', text: '9'},{id: 'a7d5', resourceId: '.button_four', text: '4'},{id: 'ab96', resourceId: '.button_five', text: '5'},{id: 'af57', resourceId: '.button_six', text: '6'},{id: 'b6d9', resourceId: '.button_one', text: '1'},{id: 'ba9a', resourceId: '.button_two', text: '2'},{id: 'be5b', resourceId: '.button_three', text: '3'},{id: 'c5dd', resourceId: '.button_zero', text: '0'}]}


getBounds (M)

Return the bounds (format: [left, top, right, bottom]) of the nodes, it returns 2 arrays one with all the IDs the other one with all the bounds.

Return: The array of bounds, each bound is an array of 4-tuples: [x1, y1, x2. y2].

>> device.sendAai({query:"T:/[0-9]/", postAction:"getBounds"})
{bounds: [[18,1158,370,1521],[370,1158,722,1521],[722,1158,1074,1521],[18,1521,370,1884],[370,1521,722,1884],[722,1521,1074,1884],[18,1884,370,2247],[370,1884,722,2247],[722,1884,1074,2247],[18,2247,370,2610]], count: 10, ids: ['98d1','9c92','a053','a7d5','ab96','af57','b6d9','ba9a','be5b','c5dd']}


refresh (M)

The accessibility nodes information is cached, if the screen has been updated, the cache information may not be accurate, use "refresh" to force to reread the nodes information.

Return: Return count and true (always return true)

Example:

>> device.sendAai({query:"T:/[0-9]/", postAction:"refresh"})
{count: 10, retval: true}


click & longClick (O)

Various kinds of clicks on the node, this is synchronous click, will make sure screen is starting to refreshed when the control is returned.

  • click: click in the center of the node.
  • longClick: hold and click longer (500ms) to simulate a long click.
  • click2: click at the random location in the node.

Return: true or false to indicate if the operation is successful.

Example:

>> device.sendAai({query:"T:5", postAction:"click"})
{count: 1, retval: true}


clickForNewWindow (O)

Usually "click" is sufficient but if the click will open a new window and take time to reach steady state, use this command. This command wait for more stringent events than "click". If this command does not work, use "waitSelector". Do not use this command if the click does not create a new window.

clickForNewWindow

Return: true or false to indicate if the operation is successful.

Example:

device.sendAai({query:"T:Scan Now", postAction:"clickForNewWindow"})


showOnScreen (O)

This will ensure the node will appear fully on the screen, if associated UI element is partially displayed, it will scroll until the UI element is fully appeared on the screen. Depending on how the UI is designed, it may move to the top of the container.

Return: true to indicate the screen has been scrolled, false to indicate full node is found on screen, nothing is done.



setText (O)

If the node is text field, it will enter required "input" string value.

setText(<input string>)

Arguments:

Input string: By default the text field will be cleared before entering the string, if the first letter is "+", it will add on to the existing string. For multiple lines of text field, use "\n" to skip to next line.

Return: true or false to indicate if the operation is successful.

Example:

>> device.sendAai({query:"TP:textInput||IX:1", postAction:"setText('Hello')"})
{count: 1, retval: true}
>> device.sendAai({query:"TP:textInput||IX:1", postAction: "setText('+ World')"})
{count: 1, retval: true}


sort/sortX/sortY

Sort the nodes based on the bounds:

  • "sort": Sort y then x, from top left to bottom right.
  • "sortX": Ignore Y and sort X from left to right.
  • "sortY": Ignore X and sort Y from top to bottom.


reduceNodes (M)

This is the core functionality of AAI, reduceNodes accepts a list of node IDs, the intent to reduce the number of nodes to visible nodes. "TL", "BL", "OX/OY" and "intersect" all use the reduceNodes. Note that the reduceNodes will pick smaller node if bigger nodes are fully contain the smaller node, if ".Button" node is enclosed smaller ".TextView" node, the ".TextView" node will be selected over ".Button" node, the "click" on the .TextView node will still work. See UI Explorer "Optimize".

Return:

retval: true/false to indicate if the reduceNodes has reduced any nodes.

count: number of nodes (if false, maintain the same nodes as original nodes). Use "getIds" to retrieve the resulting node IDs.

Example:

>> device.sendAai({}).count
217
>> device.sendAai({postAction:"reduceNodes"}).count
63

For the above screen, try to get the nodes that intersect with , the reduceNodes reduce the nodes from 29 to 3.

>> var rect = new Rect(device.sendAai({query:"T:Schedule", postAction:"getNodes", fields:"B"}).list[0].bounds)
>> device.sendAai({query:'BI:[-1,'+rect.centerY()+']', postActions:["getIds", "reduceNodes", "sortX", "getIds"]})
{count: 4, list: [{count: 29, ids: ['7fbc','873e','8aff','9281','9642','a185','b089','b80b','bf8d','c34e','c70f','45b03','ce91','d613','d9d4','dd95','146b1','e156','e517','e8d8','ec99','28205','285c6','28987','28d48','2a78f','2ab50','2af11','2b2d2']},{count: 3, retval: true},{retval: true},{count: 3, ids: ['28987','2ab50','2af11']}]}


setProgress

To set the value of the slider bar, the min, max, type, value can be obtained by "getNodes(‘RI')",

setProgress(<number>)

Arguments:

: Integer or decimal to set the value.

Return:

true/false: if the operation is successful.

Example: Set the slider to 50%:

>> device.sendAai({query:"C:.SeekBar", postActions:["getNodes('RI')", "setProgress(50)", "getNodes('RI')"]})
{count: 3, list: [{count: 1, list: [{id: '4cf6e', rangeInfo: {current: 17, max: 100, min: 0, type: 0, typeString: 'int'}}]},{retval: true},{count: 1, list: [{id: '4cf6e', rangeInfo: {current: 50, max: 100, min: 0, type: 0, typeString: 'int'}}]}]}


getChecked (O) & setChecked (M)

These commands apply to nodes with "checkable" nodes. Typically, toggle controls are checkable, checkboxes or radio buttons, example of class can be ".Switch" or ".Checkbox".

setChecked(true|false)

getChecked

Arguments:

"getChecked" will return true or false of the node checked status. Generate error if node is not checkable.

"setChecked(true|false)", since there is no permission to set the checked, it will click the node to toggle the value if value needs to be changed. It accepts multiple nodes, will ignore non-checkable nodes.

Returns: getChecked return true or false on the state of the checkable node. setChecked returns "changedCount", how many checkable nodes have been changed.



addQuery (O|M)

addQuery perform subquery on the resulting nodes, this way you can derive nodes from addQuery and act upon them. For instance, you can enter text and click send (e.g. addQuery('OX:2')) in one query. Only LT and LB are not supported. You can add as many queries as possible.

addQuery(<query string>)

Arguments:

Query string: Exact same query as main query line.

Returns: Number of matching nodes.

Example: Most chat program needs to type in text and click send button, with addQuery, you can do it in one command:

device.sendAai({query:"TP:textInput", postActions:["setText('Hello')","addQuery('OX:2')","getIds","click"]})

In Skype, if you want to click the "Calls" icon without using "LB":

>> device.sendAai({query:"TP:anyText||IX:-1", postActions:["intersectX", "addQuery('T:Calls||OY:-1')", "click"]})
{count: 3, list: [{count: 4},{count: 1},{retval: true}]}


intersectX/intersectY (O)

These commands make the query much more interesting, these two are only commands that generate more nodes. They take a node as a reference and return all the nodes that intersect horizontally (intersectX) and vertically (intersect). The node can be any node on the intersect path, it will make use of the reference node bounds (top/bottom for intersectX, left/right for intersectY) to search for intersect nodes. This command will change the internal matching list, so you can use "addQuery" to set constraints or action to act upon them. This command will take one node and generate more nodes, see "Query in Action III" above for more examples.

Return:

true /false: if the operation is successful.

Example:

>> device.sendAai({query:"TP:textInput", postActions:["getIds", "intersectX", "getIds"]})
{count: 3, list: [{count: 1, ids: ['379ea6']},{count: 5},{count: 5, ids: ['37845f','379ea6','37a9e9','37c06f','37d6f5']}]}


waitQuery

This command is useful to wait for certain node to appear on the screen before proceed to other post actions. This command waits the query and timeout specified.

waitQuery(<query string>, <timeout>)

waitQuery(<query string>)

Arguments:

query string: Exact same query as main query line.

timeout: Optional, if not specified,

Returns: return true if match is found, generate error if not found.

Example:



sleep

This command will wait on the milliseconds specified, it is useful for simple wait before actions.

sleep(<time>)

Arguments:

time: Specify time to wait in milliseconds.

Returns: return true



getText (O|M)

Simpler version of "getNodes('T')", much easier to parse if you do not intent to know the node ID. It will detect one or multiple nodes.

Return: return an array if multiple nodes are found and return text if one node is found.

Example:

>> device.sendAai({query:"LB:2", postAction:"getText"})
{retval: ['Chats','Calls','Contacts','Notifications']} 
>> device.sendAai({query:"LB:2||IX:1", postAction:"getText"})
{retval: 'Calls'} 

Commands for both preAction and postAction/postactions

These commands applied to both preAction and postAction(s). For preAction, where node is required, use "elements" to supply the node.



sendKey

sendKey accepts key code or meta state and send the key to the screen (not node), the UI element with the focus will receive the keycode and meta, examples are text field or virtual keyboards. sendKey also offers shortcut, a text string some of the common key codes. Not all key codes will generate characters, several special key codes will bring in new window such as go back, appswitch or go to home screen. The key codes and meta states are displayed in Android KeyEvent class.

The shortcuts is nothing but a mapping to different key codes, following are all the shortcuts (case insensitive): "home", "back", "backspace", "enter", "appswitch" (switch to last app), for old applications, "search" and "menu".

sendKey(<key code>)

sendKey(<key code>, <meta state>)

sendKey(<shortcut>)

Arguments:

key code and meta state: are in integer, "shortcut" is in string. Regarding the back key, some time you need to send 2 back keys to go back to previous page, first back key is to dismiss the keyboard (Sigma is invisible on screen but the input frame is visible on top of the window), the second back key is to go back to previous screen.

Return: return the status of the operation in true/false.

Example:

tcConst contains key code and meta state information. For example, the following 3 examples will get the same result:

sendKey('back')
sendKey(4)
sendKey(tcConst.keyCodes.BACK)

This will type "A":

>> device.sendAai({preAction:aaix("sendKey", tcConst.keyCodes.A, tcConst.keyCodes.META_SHIFT_ON)})
{retval: true}


getQuery/getUniqQuery (O)

Accept one node and attempt to generate a query to match the node (or multiple nodes). The query returned from "getQuery" can match multiple nodes, useful to obtain the tubular information from the applications. The query returned from "getUniqQuery" will match only one node, useful for actions such as click. This is a rudimentary at this point, we will develop better query string in the future:

  • Will use the first node of the matching list (query or element) as the reference node.
  • The output is not fully optimized, for instance, it does not use offset and template.
  • If multiple nodes are found, it will include "IX" to identify the node, this is error prone especially for applications that do not use resource ID, IX can be large number, any change in UI may impact the accuracy.
  • UI Explorer's "Code" button use this command to display the query string.
  • For text and description, if the length is longer than 30 characters, it will list the first 30 characters and add "*" at the end. E.g. If the text of the node is "The quick brown fox jumps over the lazy dog", the query generated will be: "T:The quick brown fox jumps over*".
  • If text is changing (e.g. clock), create a query to ignore text use "true" on the first argument, to ignore description, use "true" on the second argument.

getUniqQuery/getQuer - Default is to use text and description as part of the query.

getUniqQuery/getQuery(<true/false> to ignore text) – true to ignore text in search query.

getUniqQuery/getQuery(<true/false> to ignore text, <true/false> to ignore description) – true on second argument to ignore description in search query.

Arguments:

true or false: to ignore text in query.

true or false: to ignore description in query.

Return: return query string.

>> device.sendAai({elements:["123554"], preAction:"getUniqQuery"})
{query: 'T:INTC'}
>> device.sendAai({query:"T:INTC", postAction:"getUniqQuery"})
{query: 'T:INTC'}
>> device.sendAai({query:"T:INTC", postAction:"getUniqQuery(true)"})
{query: 'C:.TextView||R:.ticker||CC:0||IX:0'}
>> var ret = device.sendAai({query:"R:.ticker", postAction:"getQuery(true)"})
{query: 'C:.TextView||R:.ticker||CC:0'}
>> device.sendAai({query:ret.query, postAction:"getNodes('T,D')"})
{count: 5, list: [{id: '411a4', text: 'INTC'},{id: '3ec1a', text: 'WMT'},{id: '3c690', text: 'TGT'},{id: '3a106', text: 'AAPL'},{id: '37b7c', text: 'PEP'}]}
>> device.sendAai({query:ret.query, postAction:"getText"})
{retval: ['INTC','WMT','TGT','AAPL','PEP']} 

Support

Please send email to support@sigma-rt.com for support.