FindNode

FindNode is a shell program on top of the core Selector package, the purpose of FindNode is to find the intended one or more UI elements (or Accessibility nodes) and extract information or perform action on them.

Selector sits on top of Accessibility and UI Automator, each UI element or container of UI elements is identified by node or sets of nodes, each node has its ID that is unique on the current screen. Selector provide various ways to search for one or multiple nodes. once the nodes are acquired, you can do interesting things such as obtain the texts/images or perform action such as click on button or entering the text on the text field, all without coordinates. This allow one automation script to run on different resolutions which is not possible using coordinates. The coordinates are acquired during the runtime.

For example:

JS API: devices.click("OK") instead of devices.click(100, 200). OK query with click will be sent to all devices.

MDCC: Users click a button on the main device, the unique query of the button (e.g. "T:OK") will be sent to other devices, to search for the node and click. The intention is to provide various ways to locate nodes without using screen coordinates, there is no page up/down but "scrollToView" to locate a node, this way, the same script can run on different phones with different resolutions and screen size.

FindNode uses the JSON construct of Appium to carry out the JSON commands, the JSON format is:

{"cmd": "action", "action": "findnode", "params": {…}}

Every FindNode commands are carried out in "params" object.

Return value can have one of two types, successful run:

{"status":0, "value":"{…}"}

Or failure:

{"status":13,"value":"{<error message>}"}

TC provide "device.sendAai()" or "devices.sendAai()" to communicate with FindNode:

  • Send the "params" object.
  • Generate timeout error if FindNode does not return in certain time period.
  • For certain commands that takes longer than the default timeout, FindNode will extend the time to avoid timeout error.
  • Handle return value automatically (null on error, non-null on value).
  • For multiple devices "devices", each device will be done in threads.

The params object contains 3 types of properties: query, preAction and postAction(s). The commands offered are quite rich, you can write a simple automation with them.

Presumably you can do most of the task by using "device.sendAai()" and "devices.sendAai()", FindNode is a "handler", another handler "invoke" can use Java Reflection to access methods in UiDevice, UiObject2, AccessibilityNodeInfo and InteractionController.

Limitations

This is the version 1 of AAI, it comes with some limitations:

  • Support portrait mode only, landscape mode will be introduced in the next version.
  • Not everything will be able to identify at this time, for instance horizontal scrolling or certain line mode.
  • Horizontal scrolling or certain popup windows are not supported (less nodes or more nodes).
  • Does not support multi-threaded, each execution has to be mutually exclusive: MDCC (object), terminal (or script), UI Explorer and part of REST API. This will be resolved in subsequent versions.

Query

Majority of the FindNode usages is about query, it allows users to locate intended UI elements via different kind of useful queries on the Accessibility nodes. The entire screen is composed by many nodes, a node can be a smallest UI element or a container of smaller nodes, many are invisible. The entire screen is a tree structure starting from the single root node. Query will be performed to obtain the nodes that meet the criteria, the intent is to reduce the large number of nodes to one or few intended nodes, users can obtain information or apply actions to the nodes.

We developed a small query language (for lack of better word) that is portable among devices and powerful enough to locate most of the nodes. The format is JSON:

{query:"<key>:<value>||<key>:<value>||…"}

For historically reason, although we are using "||" to separate different criteria, "||" is AND relationship, that means it needs to meet all criterion to become a "matching node". FindNode does not offer OR relationship but the underlying Selector package provides that capability. "templates" use that construct.

Query String

There are 14 different keys, they are:

  • C: Class name (S)
  • R: Resource ID (S)
  • D: Description (S)
  • T: Text (S)
  • IT: Input type (I/S)
  • CC: Child count (I/S)
  • ID: Node ID returned by other queries (S)
  • BI: Bound in: [x, y] or [left, top, right, bottom], all integers, if x or y is -1, it will be ignored
  • IX: Index (I)
  • OX: Offset on x (I)
  • OY: Offset on y (I)
  • TP: Template (S)
  • ON: One node selection (S)
  • LT: Line number (I)
  • LB: Line number (I)

All these queries are optional, you can pick and choose any field, if you are doing the query on the screen. The "(S)" stands for String, "I" integer, "[x, y]" array. I/S can accept both integer and String. String is more powerful.

The keys are case insensitive, we use uppercase to differentiate keys from values.

Special Characters in Query

For number such as inputType or childCount, it can either accept number or string:

  • <number>, "<number>" or "=<number>": Match exacting the number.
  • "> <number>": Match number greater than <number>.
  • "< <number>": Match number less than <number>.

String can be one of the following:

  • "<string>": Exact match on full text, case sensitive.
  • "(?i)<string>": Match in case insensitive match in regular expression.
  • "*<string>": Match string at the end, e.g. "*Group".
  • "<string>*": Match string at the beginning. e.g. "android.widget.*".
  • "*<string>*": Match any substring "*widget*".
  • "/<regex>/: Match any regular expression. E.g. "/ImageView|TextView/"

Version 11 – For "T:", Prefix "ci:" for case insensitive string match. E.g. "T:ci:JOHN Doe" or "T:ci:*DOE" or "T:ci:John*" or "T:ci:*ohn*"

"!" is use at the beginning of string as NOT. "!0" = not zero. "C:!/Layout/", non-layout classes.

For simplicity in examples, the standard prefix of {"cmd": "action","action": "findnode", "params":{…}} will be omitted, we will list the examples in the "param" object and use JavaScript object notations to reduce number of double of quotes.

Shortcuts

Two shortcuts are available for class name (C) and resource ID (R), prefix with "." for substitutions of commonly use prefixes:

  • Class name (C): Prefix "." as substitution of "android.widget.". E.g. ".Button" = "android.widget.Button".
  • Resource ID (R): Prefix "." as substitution of "<package name>:id/". E.g. ".text_connect" = "com.sigma_rt.totalcontrol:id/text_connect".

The "." notation can only be used for normal matching and "!" matching, other types of matching (e.g. regular expression and wildcard) needs to use full form.

Bound In (BI):

Bound in takes screen coordinate or bounds and return the nodes based on the criteria below. This mainly for screen manipulation such as UI Explorer or debugging purposes.

  • [x, y]: Will return the nodes with bounds contain (x, y).
  • [x, -1] or [-1,y] where one of them is -1: if x = -1, will ignore x and return nodes with bounds that contains y, similarly for y = -1, it will only match x. As if drawing a horizontal/vertical line on the y and x location, the nodes that touch the line will be returned.
  • [x1, y1, x2, y2]: Will match the nodes bounds within the rectangle specified by (x1, y1) - (x2, y2).

The last 2 modes are supported in 8.0-u40.

Example:

{query:"BI:[-1, 1000]||IX:-1", postAction:"click"}
{query:"BI:[0, 0, 1000, 1000]", postAction:"getBounds"}

Index (IX), Offset (OX/OY) and Line (LT & LB)

These features are useful:

Index: IX:<number>. Return a node from a resulting node list based on the position starting from zero. IX can be negative value in which case the position is in reversed order (-1 is the last node).

Offset: OX:<integer> and OY:<integer>. Use the offset to find neighboring node base on resulting node, OX find horizontal position, positive integer moves right, negative integer moves left. Similarly for OY, positive integer moves down, negative integer moves up. If both options are present, the options will apply based on the position on the query. For example: "OX:1||OY:2" will apply horizontal before vertical offset where "OY:2||OX:1" will apply vertical before horizontal offset. The order is important to locate asymmetrical arrangement of the nodes.

Line: LT:<Integer> and LB:<Integer>. For query with line mode, FindNode will analyze the screen, trying to locate the UI elements outside of scrollable region, line mode will group UI elements into a series of lines, each line can have one or more nodes. Many applications have fixed UI elements on the top and bottom of the screen, this can be easily located by using query such as "LT:1||IX:2" (3nd node of the top line) or "LB:1||T:Chats" bottom of the line, with Chats as text.

  • For non-scrollable apps such as calculator, LT and LB will always return null.
  • Between bottom of the status line and top of the scrollable region is considering as top region, LT starts from 1, "LT:1" is first line. LT:-1 is last line in that region.
  • Between bottom of the scrollable region and top of the navigation bar is considering as bottom region. Similarly, LB starts from 1, LB:1 is first line. LB:-1 is last line in that region (also the last line of the application).
  • Using out of bound line number will get null. (E.g. 2 lines at the bottom, LB:3 will get the null).
  • If one node height is similar to 2 vertical nodes height on the same line, the lower node will be placed in next line.

This can lead to complex query, e.g.

{query:"LB:-1||IX:2||OX:1||OY:-1"}

For line mode, the "intersectY" of the first node is applied of every line, usually it is fine, but in the below cases, it will return all nodes:

So "LT:1" will return all 6 nodes.

For non-scrollable applications, "LB" will return null but it is not hard to get the nodes:

You can use text as anchor: (use "ON:last" to make sure query works if other places have the same text):

>> device.sendAai({query:"T:Speed||ON:last", postActions:["intersectX", "getText"]})
{count: 2, list: [{count: 4},{retval: ['Speed','Video','VPN','Map']}]}

To retrieve all icons:

>> device.sendAai({query:"T:Speed||ON:last||OY:-1", postActions:["intersectX", "getIds"]})
{count: 2, list: [{count: 4},{count: 4, ids: ['181ce','19c15','1b65c','1d0a3']}]}

To click the icon on VPN:

>> device.sendAai({query:"T:VPN||ON:last||OY:-1", postAction:"click"})
{retval: true}

Template (TP)

For lack of better name, we developed several predefined templates that is built-in FindNode, the default template is "more".

  • all: no constrain, return all nodes.
  • more: "all" option with layout nodes removed. This is the default if no template is defined or empty query.
  • basic: "more" option with child count of zero.
  • anyText: search for nodes with content in text or description.
  • textInput: set with input type > 0 or class with ".EditText", attempt to find text input line.
  • Example:

Example:

{query:"TP:textInput ", postAction:"setText('Hello')"}

Single Node Selection (ON)

There are query you need multiple nodes (e.g. retrieve stock quotes), there are times single node is needed, for example, click a node or enter a text, FindNode offers "IX" to pick a node from a specific position, ON provides multiple ways to pick a node.

These are the options:

  • first: Take the first node from the matching list (Similar to IX:0).
  • last: Take the last node from the matching list (Similar to IX:-1).
  • min: Pick the smallest node (based on area) from the matching list.
  • top: For overlap nodes, pick the top most node. Useful for "BI" query with x and y coordinate.

ON and IX have to be mutually exclusive.

Query Precedence

Below are the query precedences:

  1. All nodes or line mode (LT/LB).
  2. Template (TP).
  3. Index (IX) or one node selection (ON).
  4. Offset OX and OY.
  5. If multiple nodes, sort.

"elements" property

“elements” accept an array of node IDs, the node IDs must be a valid node ID on the screen or it will generate error. When “elements” is specified, the search will not be performed, however, IX, OX, OY, ON will still work. It is useful for debugging purpose and access the nodes that cannot be retrieved easily (e.g. child/parent nodes). Each query with "elements" will traverse all nodes to locate corresponding node, try to put multiple IDs in the array instead of one at a time.

device.sendAai({elements:["1234a", "5678b"], postAction:"getNodes"})

var ids = device.sendAai({query:"C:.Button"}).ids;
var retval = device.sendAai({elements:ids, postAction:"getNodes"});

Synchronous actions

Great efforts have been put in to ensure the actions are synchronous, when a "click" is called, not only ensure the click is pressed synchronously, it will wait for certain "events" to ensure the rendering is "started" to happen, cannot ensure the rendering is completed. FindNode offers several ways to performed a click:

  • "click": it should work most of the time.
  • "waitSelector": if the new window takes very long time to complete the rendering, use this command to check if the node is available, add postAction:"click", will click when the node is found and ready.
  • "waitQuery" can be included in postActions, wait for node to appear before actions are carried out.

There are 2 types of popup windows, they appear the same but they are implemented differently:

  • The popup window changes the root node of node tree, this will be handled correctly (e.g. Whatsapp).
  • The popup window created a new branch in the node tree, the resulted nodes will be accumulated nodes of main window and popup window, may likely find the wrong nodes. FindNode will monitor the events when this happens, will correctly predict the top branch of the node tree when popup window is opened and closed. When you start FindNode, make sure the application is not in the popup mode (Google Mail, Google search).

For second type of popup window, FindNode can identify some of the popup windows (Many Google applications use this type of popup), If the popup window breaks the FindNode, it will exhibit with one the following behaviors:

  • Nodes not found (even "TP:all") in query.
  • Very little nodes are displayed in UI Explorer.
  • Accumulate the nodes from popup window and the main window. UI Explorer will show a lot of bounds that do not make sense (nodes from main window).

When this happens, close (remove from memory) and reopen application, if popup windows does not work, please contact support.

This version, the horizontal scrolling (display new page by swipe left and right) with accumulate nodes will not work (e.g. Slack). Horizontal scrolling and more reliable popup window detection will be supported in the next version.

Testing

To find the nodes via Query is not very difficult, you need is "UI Explorer" and Terminal:

  • Node ID is important.
  • Terminal: Type: device.sendAai({query:"…"}) to find if it matches the node. There are many search options, there is more than one way to find nodes.
  • UI Explorer:
    • Find bounds from different nodes.
    • Find class name, resource ID, description and text. Can help you to develop query.
    • The "Code" in UI Explorer is a best effort to locate a node, not the most optimize way. May break if the node has no assigned resource ID or text (use index "IX" is problematic), refer to postAction "getUniqQuery" for more information.
  • Terminal: When in doubt, use device.sendAai({elements:["<ID>","<ID>"…], postAction:"getNodes"}), this will help you to obtain more information.
  • Start with one device using "device.sendAai" change to multiple devices via "devices.sendAai", you can search the devices by using Device.searchObject() to locate the devices you need (e.g. all devices, devices in a group, specific devices).

Query in Action I

The most basic query is "{}" or in TC:

var output = device.sendAai({})

This returns most of the nodes on the screen and the default return is a list of nodes ID:

{count: 89, ids: ['80006cbe','7440','7bc2','7f83','8344','8705','8ac6','8e87','1f6e7',…]}

You can do something like:

{query: "CC:!=0"} or {query:"CC:>0}: Non-zero child count.

{query:"IT:>10000"}, accept custom input type value that is greater than 10 thousand.

{query:"CC:!=0||IT:>10000"}, find nodes with non-zero child count AND input type is greater than 10,000.

{query:"TP:textInput||IX:2", postAction:"setText", input:"Hello"}, find third text field and enter "Hello" in the text field.

{query:”T:Input text here”, postAction:”setText('Hello')"}, some empty text field has initial hint, search the hint and enter “Hello”.

{query:"C:.TextView||T: Contacts"}

{elements:["abcd", "123e", "234f"], query:"IX:1||OY:1", postAction:"click"}

>> device.sendAai({query:"T:*REAL, \nSANTA CLARA*", postAction:"getText"})
{retval: '3367 EL CAMINO REAL, 
SANTA CLARA, CA 95051'}

Query in Action II

Example 1: Finding nodes on the bottom of the screen:

>> device.sendAai({query:"LB:-1", postAction:"getText"})
{retval: ['Chats','Calls','Contacts','Notifications']}

By using offset (OX), index (IX) or query such as "T:", you can locate almost every node, below 3 examples on click "Contacts" icon:

>>  device.sendAai({query:"LB:1||IX:2", postAction:"click"})
{retval: true}
>> device.sendAai({query:"LB:-1||T:Contacts||OY:-1", postAction:"click"})
{retval: true}
>> device.sendAai({query:"LB:-1||IX:0||OX:2||OY:-1", postAction:"click"})
{retval: true}

The purpose of the line mode is to control the icons on the top/bottom. The red counter intersects with Chats icon will be removed. So for the line of icons, it is 4 nodes instead of 5 nodes:

>>  device.sendAai({query:"LB:1"})
{count: 4, ids: ['14735','17080','18ac7','1a50e']}

If you want the counter, use other query (IX:-1 or ON:last can be used to prevent finding other "Chats" on the screen, pick the last"Chats"), the second line below will work since offset will not ignore intersected nodes:

>>  device.sendAai({query:"T:Chats||IX:-1||OY:-1||OX:1", postAction:"getText"})
{retval: '1'}
>> device.sendAai({query:"LB:-1||T:Chats||OY:-1||OX:1", postAction:"getText"})
{retval: '1'}

Example 2: Retrieve information from About phone:

To obtain the information:

>> device.sendAai({query:"T:Model name||OX:1", postAction:"getText"})
{retval: 'Galaxy S10+'}

Alternatively, you can use UiElement class:

>> var obj = UiElement.findObject(device, "T:Model name||OX:1")
UiElement: 3d012
>> obj.getText()
Galaxy S10+

Example 3: Assuming you want to first page stock tickers from Yahoo Finance:

device.sendAai({query:"R:.ticker", fields:"T", postAction:"getNodes"}).list.forEach(
    function(p) {
    print(p.text + ":" + 
        device.sendAai({query:"OX:1||T:" + p.text, fields:"D", 
            postAction:"getNodes"}).list[0].description);
})
INTC:53.14
WMT:142.00
XOM:65.93
BIDU:146.53
PEP:173.23

Example 4: Enter text in Skype text box:

Before entering text:

After text is input:

This can be done by using in one query command:

var input = "Hello";
device.sendAai({query:"LB:1||TP:textInput", postActions:[aaix("setText", input), "addQuery('OX:2')", "click"]});

aaix("setText", input) will return 'setText("Hello")', much better than 'setText("'+input+'")', useful for multiple arguments.

The following function to find the person, click, send a message and return back. The 2 "back" keys are required, the first back key is to dismiss the keyboard, second back key is to go back to main page.

function sendAai(device, obj) {
    var retval = device.sendAai(obj);
    if (retval == null) {
        throw "Error on: " + JSON.stringify(obj) + ":" + lastError();
    }
    print("$ " + JSON.stringify(obj) + "\n" + JSON.stringify(retval));
    return retval;
}

function sendSkype(device, name, text) {
    sendAai(device, {query:"T:" + name, preAction:"scrollToView", postAction:"click"});
    sendAai(device, {query:"TP:textInput", postActions:[aaix("setText", text), "addQuery('OX:2')", "click", "sendKey('back')", "sendKey('back')"]})
}

sendSkype(device, "John", "I will get back to you");

Example 5: On the first example, you can use "T:<text>" and offset to click the icons, some applications do not provide the text:

Since this is display on the last line, we can use "LB:-1" to return the nodes, then use "IX" to locate a single node from the list of nodes, to click the search icon:

device.sendAai({query:"LB:-1||IX:2", postAction:"click"})

Example 6: In certain chat window, it shows the name (or phone number), to obtain it, use the following queries:

>> device.sendAai({query:"LT:1"})
{ids: ['1637ca','163f4c','165d54','166115','1664d6']}

To go back:

>> device.sendAai({query:"LT:1||IX:0", postAction:"click"})
{retval: true}

To obtain the name:

>> device.sendAai({query:"LT:1||IX:1", postAction:"getText"})
{retval: '(XXX)XXX-XXXX'}

Note: The new version of "Messages", "LT" and "LB" no longer works, you can still retrieve first line or last line of information, since the message conversation (resource ID .message_text) and contact icon (resource ID .contact_icon) are mixed in the top nodes, we need to filter them out. If the "query" is not there, will be defaulted to "TP:more". FindNode will not expand "." inside regular expression, safe to match partial resource ID of "\/message_text" and "\/contact_icon" instead of long "<package name>:id/<ID> ".

First line:

>> device.sendAai({postActions:["reduceNodes", "addQuery('IX:0')", "intersectX", "addQuery('R:!/\/message_text|\/contact_icon/')", "getIds"]})
{count: 5, list: [{count: 23, retval: true},{count: 1},{count: 9},{count: 6},{count: 6, ids: ['2ee8c0','2ef042','2f0e4a','2f120b','2f15cc','2f198d']}]}

Last line:

>> device.sendAai({query:"TP:textInput", postActions:["intersectX", "addQuery('R:!/\/message_text|\/contact_icon/')", "getIds"]})
{count: 3, list: [{count: 5},{count: 5},{count: 5, ids: ['2e5659','2e5a1a','2e9dac','2ea8ef','2ed23a']}]}

After receiving the nodes, can select the nodes using addQuery() or perform actions.

Example 7: Locate and click with ease.

To click the arrow, use:

>> device.sendAai({query:"T:Channels||OX:1", postAction:"click"})
{retval: true}

Query in Action III

Example 1: Consider the following calculator in UI Explorer:

To use this calculator can be very simple:

function calc(device, app, resultQuery, formula) {
    device.runAppSync(app);
    formula.split("").forEach((n)=>device.clickSync(n));
    return device.aaiGetText(resultQuery)[0].text;
}
var result = calc(
    Device.getMain(),
    "com.dalviksoft.calculator",
    "R:*result",
    "1230÷567="
);
print(result);

This will give you the result of 1230÷567.

To retrieve all the digit buttons:

>> device.sendAai({query:"T:/[0-9]/||C:.Button", preAction:"doSort", postAction:"getText"})
{retval: ['7','8','9','4','5','6','1','2','3','0']}

Retrieve digit 5-9, sort is based on the location of the nodes:

>> device.sendAai({query:"T:/[5-9]/||C:.Button", postActions:["sort","getText"]})
{count: 2, list: [{retval: true},{retval: ['7','8','9','5','6']}]}

To retrieve 5 buttons of arithmetic operations + DEL buttons, there are several ways to do it, one way is to use bounds on x:

function getNodes() {
    var ret = device.sendAai({query:"T:DEL||C:.Button", postAction:"getNodes", fields:"B"});
    if (ret == null) {
        print("Error:" + lastError());
        return null;
    }
    var rect = new Rect(ret.list[0].bounds);
    var ret = device.sendAai({query:"BI:[" + rect.centerX() + ",-1]||C:.Button"})
    if (ret == null) {
        print("Error:" + lastError());
        return null;
    }    
    return ret.ids;
}

Alternatively, you can also use the following to get the same thing:

device.sendAai({query:"T:0||OX:2", postActions:["intersectY", "addQuery('C:.Button')", "getIds"]})

You can do the operation 2+3 here manually, not to search every time to save CPU, use "elements", "=" does not have string representation, so use OX:1 from "T:0".

var opIds = getNodes();
var numIds = device.sendAai({query:"T:/[0-9]/||C:.Button", preAction: "doSort"}).ids;
device.sendAai({elements:numIds, query:"IX:-2", postAction:"click"});
device.sendAai({elements:opIds, query:"IX:2", postAction:"click"});
device.sendAai({elements:numIds, query:"IX:8", postAction:"click"});
device.sendAai({query:"T:0||C:.Button||OX:1", postAction:"click"});

Alternatively, you can enter text on the text field. Can use UiElement to do that:

var obj = UiElement.findObject(device, "TP:textInput");
var equal = UiElement.findObject(device, "T:0||C:.Button||OX:1");
obj.setText("2+3");
equal.clickSync();
var result = obj.getText();
print(result);

Example 2: Use "checked" to manipulate the checkboxes.

Change Sunday from on to off, see the value before and after:

>> device.sendAai({query:"D:Sunday||C:.CheckBox", postActions:["getChecked", "setChecked(false)", "getChecked"]})
{count: 3, list: [{retval: true},{checked: false, retval: true},{retval: false}]}

"intersectX" will return all nodes with Y coordinate intersect with matching node, in this case all checkboxes. Use "setChecked()" to set all nodes to true/false (on/off), it only applies to checkable nodes, none checkable nodes will be ignored. Only click checkbox if the value need to be changed.

>> device.sendAai({query:"D:Sunday", postActions:["intersectX","getIds","setChecked(true)"]})
{count: 3, list: [{count: 7},{count: 7, ids: ['9c52d','9c8ee','9ccaf','9d070','9d431','9d7f2','9dbb3']},{changedCount: 7, retval: true}]}

Example 3: The order of OY and OX is important, below is the developer option:

"Disable adb authorization timeout" does not have node on the right side so apply offset on X first will get null:

>> device.sendAai({query:"T:Disable adb authorization*||OX:1||OY:1"})
null
>> lastError()
Offset out of range

If apply offset on Y first will get proper node:

>> device.sendAai({query:"T:Disable adb authorization*||OY:1||OX:1"})
{count: 1, ids: ['40e9c4']}
>> device.sendAai({query:"T:Disable adb authorization*||OY:1||OX:1", postAction:"setChecked(true)"})
{changedCount: 1, retval: true}
Or
>> device.sendAai({query:"T:Disable adb authorization*||OY:1||OX:1", postAction:”click”})

Example 4: You can use one AAI commands to enter smiley emoji symbol, here is an example of Messenger, 4 clicks to get a smiley emoji:

>>  device.sendAai({query:"LB:-1||TP:textInput||OX:1", postActions:["click", "addQuery('OY:1')","intersectX", "addQuery('IX:3')", "save", "nsClick", "intersectY", "addQuery('IX:-1')", "intersectX", "addQuery('IX:1')", "nsClick", "load", "addQuery('OY:1')","intersectX", "addQuery('IX:0')", "addQuery('OY:1||OX:5')", "click", "sendKey('back')"]})
{count: 18, list: [{retval: true},{count: 1},{count: 5},{count: 1},{retval: true},{retval: false},{count: 21},{count: 1},{count: 10},{count: 1},{retval: false},{retval: true},{count: 1},{count: 7},{count: 1},{count: 1},{retval: true},{retval: true}]}

Can simplify the above command (reduce from 18 postActions to 12):

>> device.sendAai({query:"LB:-1||TP:textInput||OX:1", postActions:["click", "intersectX(OY:1,IX:3)", "save", "nsClick", "intersectY(IX:-1)", "intersectX(IX:1)", "nsClick", "load", "intersectX(OY:1,IX:0)", "addQuery(OY:1||OX:5)", "click", "sendKey(back)"]})
{count: 12, list: [{retval: true},{count: 1},{retval: true},{retval: false},{count: 1},{count: 1},{retval: false},{retval: true},{count: 1},{count: 1},{retval: true},{retval: true}]}

In order to click an emoji (upside down face), 4 clicks are required.

query:"LB:-1||TP:textInput||OX:1" - This select the icon on the right of textfield in last row (blue face icon).

"click" – Click on the node to open the page.

"addQuery('OY:1')" - Use "OY" to skip to the next line. In this case, it will likely land into smiley emoji icon, just to be safe, we use the following way to ensure the smiley emoji icon is found.

"intersectX" – Will return the nodes on that line:

"addQuery('IX:3')" – Pick the 4th node in that line (emoji icon).

"click" – Click the node to change the page.

"addQuery('OY:1')" – Go to next line.

"intersectX" – Will return the line of "OY:1".

"save" – Save the node, need it later.

"nsClick" – Click the node to change the page, likely won't change the output, use nsClick.

Click last line, 2nd icon to change it to emoji icon:

"intersectY", "addQuery('IX:-1')" – Perform "intersectY" and pick the last node (in the last line)

"intersectX", "addQuery('IX:1')" – Perform "intersectX" and pick the second icon.

"nsClick" – Click the node to change the page, likely won't change the output, use nsClick.

Find the first line, first node in the Emoji:

"load" – Load the node that we saved earlier (2nd click).

"addQuery('OY:1')" – Go to next line.

"intersectX","addQuery('IX:0')" – Picked the first line, first icon of the Emoji table.

"addQuery('OY:2||OX:2')" – Interest on the 3th line and 3rd node.

"click" – Click the selected emoji.

"sendKey('back')" – Send the Back key to go back.

To find out which node or nodes are selected or generated, open UI Explorer and add "getIds" to find out the node ID in UI Explorer.

You can also achieve the same result using setText and String.fromCodePoint:

>> device.sendAai({query:"TP:textInput", postAction:aaix("setText", String.fromCodePoint(0x1f643))})
{retval: true}

Or mix with normal message:

>> device.sendAai({query:"TP:textInput", postAction:aaix("setText", "hello" + String.fromCodePoint(0x1f643, 0x1f644) + "world")})
{retval: true}

Example 5: Use in-house developed app to open developer options, change the window animation scale to 2x and close the app, the automation is written entirely in postActions:

>> device.sendAai({postActions:["openApp(SigmaTestApp)", "newQuery(T:SETTINGS)", "click", "scrollToView(T:*DEVELOPMENT_SETTINGS)", "click", "newQuery(T:GO)", "click", "scrollToView(T:Window animation scale)", "click", "newQuery(T:*2x)", "click", "sendKey(back)", "sendKey(back)", "closeApp(SigmaTestApp)"]})
{count: 14, list: [{retval: true},{count: 1},{retval: true},{count: 1},{retval: true},{count: 1},{retval: true},{count: 1},{retval: true},{count: 1},{retval: true},{retval: true},{retval: true},{retval: true}]}

preAction, postAction and postActions

The main purpose of the FindNode develop query to find nodes, the “preAction” happens before search “postAction” and "postActions" happen after the search or nodes made available by preAction, the actions performed will be on the resulting nodes. For action that needs one node (e.g. “click”), the first node in the resulting node will be used (ON, IX, OX and OY can change that). There are many commands in pre and post actions, starting version level 11, can use "preAction:version" to obtain the version to ensure the version supports the commands required. Some commands require additional arguments (the arguments will be enclosed in parenthesis). Currently, the following types of arguments are supported:

  • "String" or ‘String': String enclosed by single quote or double quote.
  • <no quote>: If string does not contain comma or trailing spaces, no quote is required.
  • true/false: Boolean value.
  • Number: Number with or without decimal (depending on the types of actions).

A few examples:

postAction:"getNodes('C,R')"

If you want to use variable in the action commands, use one of the following 4 options:

var input = "Hello"
>> "setText('" + input + "')"
setText('Hello')
>> 'setText("' + input + '")'
setText("Hello")
>> 'setText("' + input + '")'
setText("Hello")
>> aaix("setText", input)
setText("Hello")
postActions:[aaix("setText", input), "addQuery('OX:2')", "click"]		
		

If the argument is string, below 3 examples are identical:

postActions:["intersectX(OY:1, IX:2)", "getText"]
postActions:["intersectX('OY:1', 'IX:2')", "getText"]
postActions:[aaix("intersectX", "OY:1", "IX:2"), "getText"]

preAction

"preAction" is used one of the following conditions, if the resulting output is one or mode nodes, it will proceed to the postAction otherwise the output will return to the caller.

  • Add more conditions to the existing query (e.g. getCount or doSort).
  • Obtain nodes that cannot be done by query (e.g. getFocus).
  • Action or information that is not related to nodes or query (e.g. waitForWindowUpdate, sendKey).


bgQuery:*

These commands are for background query, this is used by TC to support "addQueryListener()", will invoke callback when the matched is found.



getCount

Return the number of matched nodes instead of the list of the node IDs. It should be in the postAction, however, getting is count is faster than the list of node IDs, this command is used to determine the count or node search.

Return: Return the number of matching nodes, no postAction(s) will be performed.

Example:

>> device.sendAai({preAction:"getCount", query:"R:.ticker"})
{count: 6}


getFocus

Return the node receiving focus (usually text field) or null with error if no node gains focus. If node is found, it will proceed to postAction.

Return: null if no focus is found otherwise procced to postAction(s).

Example: To enter the next line from the existing line

{preAction:"getFocus", query:"OY:1", postAction:"setText('Hello')"}


scrollToView

Accept query string (in "query"), will try to match the query by scrolling up/down, when one or multiple nodes are found, it will proceed to postAction, if the match cannot be found, it will return an error. The scroll will use page up/down with the maximum of 10 pages, the scroll may take some time, this command will automatically lengthen the timeout in "sendAai" until it is done. Several options to determine how to the scroll is peformed, "showOnScreen" in postAction can be used to ensure the node is fully visible. On failure, to go back to the original location, run scrollToView in reverse direction.

preAction:"scrollToView(<from-direction>)"

Arguments:

<from-direction> can be one of the following, if not specify, the default is "fromCurrentDown".

"currentDown": Starting from current location and search by scrolling down.

"currentUp": Starting from current location and search by scrolling up.

"topDown": Fast scroll to the top of page and search by scrolling down.

"bottomUp": Fast scroll to the bottom of page and search by scrolling up.

Return: Will return null only if the node cannot be found otherwise it will proceed to postAction or postActions.

Example:

{query:"T:Bob", preAction:"scrollToView('topDown')", postActions:["showOnScreen", "getNodes('all')", "click"]}


version version 11

This command return a single digit to show the version of FindNode, version 10 does not support the command will return null with "lastError()" returns "Unknown preAction".

>> device.sendAai({preAction:"version"})
{retval: 11}


waitForWindowUpdate

Waits for a window content update event to occur. This is just a call to UiDevice. waitForWindowUpdate(), the event monitor is very sensitive, most of the time without any movement, it will return true (maybe the status line movement), use waitSelector and waitQuery is better.

waitForWindowUpdate(<timeout>)

Arguments:

"timeout:<integer>" (Optional): Wait until timeout is expired in ms. If no timeout is specified, 3000ms is used. Maximum timeout limits to 20 seconds.

Return:

true if a window update occurred, false if timeout has elapsed or if the current window does not have the specified package name.



waitSelector

With this mode, if search on the specified "query" is failed, it will retry every 500ms until the nodes are found and proceed, if the timeout expired, it will return error.

waitSelector(<timeout>)

Arguments:

timeout: timeout in ms for the search. "timeout" has a limit of no more than 20 seconds.

Example:

device.sendAai({query:"T:Done", preAction:"waitSelector(10000)"})

postAction and postActions

After the search (or nodes from preAction), if the search cannot be found, an error will be generated. If preAction:"doCount" is specified, the number of matches will be return. The matching nodes will be available for "postAction" or "postActions", these commands will take the resulting nodes and perform actions on them, certain actions only accept one node, the first node will be applied. "postAction" will take one command and return the output as a JSON object. "postActions" will take an array of commands and return an array of output as a JSON object. Some commands can work with multiple nodes (M), some commands generate more nodes (intersect) and if commands only work in one node (O), the first node will be used. For single node actions, if first node is not desire, use addQuery with ON, IX to change that. The postActions order is important.

device.sendAai({query:"T:OK||IX:-1", postAction:"click"});
device.sendAai({query:"R:.tickers", postActions:["refresh", "getNodes('T,D')"]});
device.sendAai({query:"R:.tickers", postActions:["getNodes", "sortX", "getNodes"]});

See example above to see postAction/postActions in action.



addQuery (O|M)

addQuery perform subquery on the resulting nodes, this way you can derive nodes from addQuery and act upon them. For instance, you can enter text and click send (e.g. addQuery('OX:2')) in one query. Only LT and LB are not supported. You can add as many queries as possible.

addQuery(<query string>)

Arguments:

Query string: Exact same query as main query line.

Returns: Number of matching nodes.

Example: Most chat program needs to type in text and click send button, with addQuery, you can do it in one command:

device.sendAai({query:"TP:textInput", postActions:["setText('Hello')","addQuery('OX:2')","getIds","click"]})

In Skype, if you want to click the "Calls" icon without using "LB":

>> device.sendAai({query:"TP:anyText||IX:-1", postActions:["intersectX", "addQuery('T:Calls||OY:-1')", "click"]})
{count: 3, list: [{count: 4},{count: 1},{retval: true}]}


click & longClick (O)

Various kinds of clicks on the node, this is synchronous click, will make sure screen is starting to refreshed when the control is returned.

  • click: click in the center of the node.
  • longClick: hold and click longer (500ms) to simulate a long click.
  • click2: click at the random location in the node.

All the above clicks will monitor the events for the change of the screen after the click is performed, if the screen is not changed, these clicks will fail and eventually timed out. If the screen is not changed after the click is performed, use "nsClick":

  • nsClick: Performed the click but does not monitor the events.

Return: true or false to indicate if the operation is successful.

Example:

>> device.sendAai({query:"T:5", postAction:"click"})
{count: 1, retval: true}


getBounds (M)

Return the bounds (format: [left, top, right, bottom]) of the nodes, it returns 2 arrays one with all the IDs the other one with all the bounds.

Return: The array of bounds, each bound is an array of 4-tuples: [x1, y1, x2. y2].

>> device.sendAai({query:"T:/[0-9]/", postAction:"getBounds"})
{bounds: [[18,1158,370,1521],[370,1158,722,1521],[722,1158,1074,1521],[18,1521,370,1884],[370,1521,722,1884],[722,1521,1074,1884],[18,1884,370,2247],[370,1884,722,2247],[722,1884,1074,2247],[18,2247,370,2610]], count: 10, ids: ['98d1','9c92','a053','a7d5','ab96','af57','b6d9','ba9a','be5b','c5dd']}


getChecked (O) & setChecked (M)

These commands apply to nodes with "checkable" nodes. Typically, toggle controls are checkable, checkboxes or radio buttons, example of class can be ".Switch" or ".Checkbox".

setChecked(true|false)

getChecked

Arguments:

"getChecked" will return true or false of the node checked status. Generate error if node is not checkable.

"setChecked(true|false)", since there is no permission to set the checked, it will click the node to toggle the value if value needs to be changed. It accepts multiple nodes, will ignore non-checkable nodes.

Returns: getChecked return true or false on the state of the checkable node. setChecked returns "changedCount", how many checkable nodes have been changed.



getIds (M)

This is the default postAction if no postAction is defined, this command will return a list of matching nodes and counter of matching nodes:

Return: count and array of IDs.

Example:

>> device.sendAai({})
{count: 26, ids: ['7708','d8a2','e7a6','ef28','824b','860c','89cd','8d8e','914f','9510','98d1','9c92','a053','a414','a7d5','ab96','af57','b318','b6d9','ba9a','be5b','c21c','c5dd','c99e','cd5f','d120']}


getNodes (M)

This is used to retrieve the information of nodes. The fields can determine what to display. "getNodes" will refresh the node before retrieving information.

getNodes

getNodes(<fields>)

Arguments:

fields: The optional "fields" is used to define the fields to return, multiple fields in "fields" are separated by comma. The valid field identifiers are:

  • P: Package name (S)
  • C: Class name (S)
  • R: Resource ID (S)
  • D: Description (S)
  • T: Text (S)
  • IT: Input type (I)
  • CC: Child count (I)
  • RI: RangeInfo, provide more information about the type, min, max, current of widget such as SeekBar (slider), use "setProgress" command to change the value.
  • BP: All boolean properties in the node.
  • B: [left top][right bottom] define node area. (S)

Return: Count and array of node information.

>> device.sendAai({query:"T:/[0-9]/", postAction:"getNodes('R,T')"})
{count: 10, list: [{id: '98d1', resourceId: '.button_seven', text: '7'},{id: '9c92', resourceId: '.button_eight', text: '8'},{id: 'a053', resourceId: '.button_nine', text: '9'},{id: 'a7d5', resourceId: '.button_four', text: '4'},{id: 'ab96', resourceId: '.button_five', text: '5'},{id: 'af57', resourceId: '.button_six', text: '6'},{id: 'b6d9', resourceId: '.button_one', text: '1'},{id: 'ba9a', resourceId: '.button_two', text: '2'},{id: 'be5b', resourceId: '.button_three', text: '3'},{id: 'c5dd', resourceId: '.button_zero', text: '0'}]}


getText (O|M)

Simpler version of "getNodes('T')", much easier to parse if you do not intent to know the node ID. It will detect one or multiple nodes.

Return: return an array if multiple nodes are found and return text if one node is found.

Example:

>> device.sendAai({query:"LB:2", postAction:"getText"})
{retval: ['Chats','Calls','Contacts','Notifications']} 
>> device.sendAai({query:"LB:2||IX:1", postAction:"getText"})
{retval: 'Calls'} 


intersectX/intersectY (O)

These commands make the query much more interesting, these two are only commands that generate more nodes. They take a node as a reference and return all the nodes that intersect horizontally (intersectX) and vertically (intersectY). The node can be any node on the intersect path, it will make use of the reference node bounds (top/bottom for intersectX, left/right for intersectY) to search for intersect nodes. This command will change the internal matching list, so you can use "addQuery" to set constraints or action to act upon them. This command will take one node and generate more nodes, if internal matching list contains multiple nodes, the first one will be used. "intersect" and "addQuery" are usually tied together, we have added addQuery before and after the intersect.

intersect?

intersect?(<post query>) = "intersect?", "addQuery(<post Query>)"

intersect?(<pre Query>, <post query>) = "addQuery(<pre query>)", "intersect?", "addQuery(<post query>"

Return:

true /false: if the operation is successful. If query is specified, the output of post query (see "addQuery") will be the output value of this command.

Example:

>> device.sendAai({query:"TP:textInput", postActions:["getIds", "intersectX", "getIds"]})
{count: 3, list: [{count: 1, ids: ['379ea6']},{count: 5},{count: 5, ids: ['37845f','379ea6','37a9e9','37c06f','37d6f5']}]}

Obtain 3rd node (IX:2) of the next line (OY:1):

>> device,sendAai({query:"T:7", postActions:["intersectX(OY:1, IX:2)", "getText"]})
{count: 2, list: [{count: 1},{retval: '6'}]}

Without preQuery without postQuery, use empty string:

>> device.sendAai({query:"T:Chats||IX:-1", postActions:["intersectX(OY:-1,'')", "getIds"]})
{count: 2, list: [{count: 4},{count: 4, ids: ['140ab','15eb3','178fa','19341']}]}


newQuery version 11

newQuery is like another "sendAai" command, there are 2 possible usages in newQuery:

  • For performance reason, an action such as "click" opens a new window, newQuery will obtain the nodes of the new window.
  • To query addition nodes outside of the main query, use "save", "newQuery", actions, "load".

newQuery(<query>)

Return:

If nodes are found, the node count will be return as "count". If not found, a null will return with lastError().

Example: "click" will open a new chat window, without newQuery, two sendAai() commands are required

>> device.sendAai({query:"T:John", postActions:["click", "newQuery(TP:textInput)", "setText(Hello)"]})
{count: 3, list: [{retval: true},{count: 1},{retval: true}]}


openApp/restartApp/closeApp version 11

3 commands to start or close the application on the device. All command accepts one argument, either a package name or the application name (name show on the launcher), if multiple applications with the same name is matched, the first application will be used. No partial match is allowed, it is case insensitive match for application name.

"openApp" will run the application, if the application is resided on the memory, it will resume the execution. "restartApp" will first close the application before start the application, if automation need to start application in a known state. "closeApp" will close the application (force-close). To open an application to reach a "stable known state", use "restartApp" and "waitQuery".

openApp(<name>)

restartApp(<name>)

closeApp(<name>)

Return:

retval: true for success or null with lastError() on failure.

Example:



reduceNodes (M)

This is the core functionality of AAI, reduceNodes accepts a list of node IDs, the intent to reduce the number of nodes to visible nodes. "TL", "BL", "OX/OY" and "intersect" all use the reduceNodes. Note that the reduceNodes will pick smaller node if bigger nodes are fully contain the smaller node, if ".Button" node is enclosed smaller ".TextView" node, the ".TextView" node will be selected over ".Button" node, the "click" on the .TextView node will still work. See UI Explorer "Optimize".

Return:

retval: true/false to indicate if the reduceNodes has reduced any nodes.

count: number of nodes (if false, maintain the same nodes as original nodes). Use "getIds" to retrieve the resulting node IDs.

Example:

>> device.sendAai({}).count
217
>> device.sendAai({postAction:"reduceNodes"}).count
63

For the above screen, try to get the nodes that intersect with , the reduceNodes reduce the nodes from 29 to 3.

>> var rect = new Rect(device.sendAai({query:"T:Schedule", postAction:"getNodes", fields:"B"}).list[0].bounds)
>> device.sendAai({query:'BI:[-1,'+rect.centerY()+']', postActions:["getIds", "reduceNodes", "sortX", "getIds"]})
{count: 4, list: [{count: 29, ids: ['7fbc','873e','8aff','9281','9642','a185','b089','b80b','bf8d','c34e','c70f','45b03','ce91','d613','d9d4','dd95','146b1','e156','e517','e8d8','ec99','28205','285c6','28987','28d48','2a78f','2ab50','2af11','2b2d2']},{count: 3, retval: true},{retval: true},{count: 3, ids: ['28987','2ab50','2af11']}]}


refresh (M)

The accessibility nodes information is cached, if the screen has been updated, the cache information may not be accurate, use "refresh" to force to reread the nodes information.

Return: Return count and true (always return true)

Example:

>> device.sendAai({query:"T:/[0-9]/", postAction:"refresh"})
{count: 10, retval: true}


save and load (O|M)

"save" save the current list of nodes for later used. "load" will load the saved list of nodes, the existing list of nodes will be discarded.



scrollToView

Similar to scrollToView in preAction:"scrollToView", it requires one mandatory query

scrollToView(<query>)

scrollToView(<from-direction>, <query>)

Arguments:

<from-direction> refer to preAction:"scrollToView" for options for "from-direction".

<query>: The query for scrollToView to match.

Return: Will return null if the query cannot be found otherwise the resulting nodes will replace the internal matched list.

Example:

Refer to "Query in Action III", example 5.



setProgress

To set the value of the slider bar, the min, max, type, value can be obtained by "getNodes(‘RI')",

setProgress(<number>)

Arguments:

<number>: Integer or decimal to set the value.

Return:

true/false: if the operation is successful.

Example: Set the slider to 50%:

>> device.sendAai({query:"C:.SeekBar", postActions:["getNodes('RI')", "setProgress(50)", "getNodes('RI')"]})
{count: 3, list: [{count: 1, list: [{id: '4cf6e', rangeInfo: {current: 17, max: 100, min: 0, type: 0, typeString: 'int'}}]},{retval: true},{count: 1, list: [{id: '4cf6e', rangeInfo: {current: 50, max: 100, min: 0, type: 0, typeString: 'int'}}]}]}


setText (O)

If the node is text field, it will enter required "input" string value.

setText(<input string>)

Arguments:

Input string: By default the text field will be cleared before entering the string, if the first letter is "+", it will add on to the existing string. For multiple lines of text field, use "\n" to skip to next line.

Return: true or false to indicate if the operation is successful.

Example:

>> device.sendAai({query:"TP:textInput||IX:1", postAction:"setText('Hello')"})
{count: 1, retval: true}
>> device.sendAai({query:"TP:textInput||IX:1", postAction: "setText('+ World')"})
{count: 1, retval: true}


showOnScreen (O)

This will ensure the node will appear fully on the screen, if associated UI element is partially displayed, it will scroll until the UI element is fully appeared on the screen. Depending on how the UI is designed, it may move to the top of the container.

Return: true to indicate the screen has been scrolled, false to indicate full node is found on screen, nothing is done.



sleep

This command will wait on the milliseconds specified, it is useful for time related simple wait before actions.

sleep(<time>)

Arguments:

time: Specify time to wait in milliseconds.

Returns: return true



sort/sortX/sortY

Sort the nodes based on the bounds:

  • "sort": Sort y then x, from top left to bottom right.
  • "sortX": Ignore Y and sort X from left to right.
  • "sortY": Ignore X and sort Y from top to bottom.


waitQuery

This command is useful to wait for certain node to appear on the screen before proceed to other post actions. This command waits the query and timeout specified. It does not change the underlying matching nodes.

waitQuery(<query string>, <timeout>)

waitQuery(<query string>)

Arguments:

query string: Exact same query as main query line.

timeout: Optional, if not specified,

Returns: return true if match is found, generate error if not found.

Example:

>> device.sendAai({query:"T:Start", postActions:["click", "waitQuery('T:Done', 8000)", "sendKey('back')"]})
{count: 3, list: [{retval: true},{retval: true},{retval: true}]}

Commands for both preAction and postAction/postactions

These commands applied to both preAction and postAction(s). For preAction, where node is required, use "elements" to supply the node.



getQuery/getUniqQuery (O)

Accept one node and attempt to generate a query to match the node (or multiple nodes). The query returned from "getQuery" can match multiple nodes, useful to obtain the tubular information from the applications. The query returned from "getUniqQuery" will match only one node, useful for actions such as click. This is a rudimentary at this point, we will develop better query string in the future:

  • Will use the first node of the matching list (query or element) as the reference node.
  • The output is not fully optimized, for instance, it does not use offset and template.
  • If multiple nodes are found, it will include "IX" to identify the node, this is error prone especially for applications that do not use resource ID, IX can be large number, any change in UI may impact the accuracy.
  • UI Explorer's "Code" button use this command to display the query string.
  • For text and description, if the length is longer than 30 characters, it will list the first 30 characters and add "*" at the end. E.g. If the text of the node is "The quick brown fox jumps over the lazy dog", the query generated will be: "T:The quick brown fox jumps over*".
  • If text is changing (e.g. clock), create a query to ignore text use "true" on the first argument, to ignore description, use "true" on the second argument.

getUniqQuery/getQuer - Default is to use text and description as part of the query.

getUniqQuery/getQuery(<true/false> to ignore text) – true to ignore text in search query.

getUniqQuery/getQuery(<true/false> to ignore text, <true/false> to ignore description) – true on second argument to ignore description in search query.

Arguments:

true or false: to ignore text in query.

true or false: to ignore description in query.

Return: return query string.

>> device.sendAai({elements:["123554"], preAction:"getUniqQuery"})
{query: 'T:INTC'}
>> device.sendAai({query:"T:INTC", postAction:"getUniqQuery"})
{query: 'T:INTC'}
>> device.sendAai({query:"T:INTC", postAction:"getUniqQuery(true)"})
{query: 'C:.TextView||R:.ticker||CC:0||IX:0'}
>> var ret = device.sendAai({query:"R:.ticker", postAction:"getQuery(true)"})
{query: 'C:.TextView||R:.ticker||CC:0'}
>> device.sendAai({query:ret.query, postAction:"getNodes('T,D')"})
{count: 5, list: [{id: '411a4', text: 'INTC'},{id: '3ec1a', text: 'WMT'},{id: '3c690', text: 'TGT'},{id: '3a106', text: 'AAPL'},{id: '37b7c', text: 'PEP'}]}
>> device.sendAai({query:ret.query, postAction:"getText"})
{retval: ['INTC','WMT','TGT','AAPL','PEP']} 


sendKey

sendKey accepts key code or meta state and send the key to the screen (not node), the UI element with the focus will receive the keycode and meta, examples are text field or virtual keyboards. sendKey also offers shortcut, a text string some of the common key codes. Not all key codes will generate characters, several special key codes will bring in new window such as go back, appswitch or go to home screen. The key codes and meta states are displayed in Android KeyEvent class.

The shortcuts is nothing but a mapping to different key codes, following are all the shortcuts (case insensitive): "home", "back", "backspace", "enter", "appswitch" (switch to last app), for old applications, "search" and "menu".

sendKey(<key code>)

sendKey(<key code>, <meta state>)

sendKey(<shortcut>)

Arguments:

key code and meta state: are in integer, "shortcut" is in string. Regarding the back key, some time you need to send 2 back keys to go back to previous page, first back key is to dismiss the keyboard (Sigma is invisible on screen but the input frame is visible on top of the window), the second back key is to go back to previous screen.

Return: return the status of the operation in true/false.

Example:

tcConst contains key code and meta state information. For example, the following 3 examples will get the same result:

sendKey('back')
sendKey(4)
sendKey(tcConst.keyCodes.BACK)

This will type "A":

>> device.sendAai({preAction:aaix("sendKey", tcConst.keyCodes.A, tcConst.keyCodes.META_SHIFT_ON)})
{retval: true}

Support

Please send email to support@sigma-rt.com for support and feedbacks.