FindNode
FindNode is an innovative shell program developed by Sigma Resources & Technologies Inc., enhancing the core Selector package. It's designed to intelligently identify UI elements (or Accessibility nodes), facilitating information extraction and interaction without the need for coordinate-based commands. This marks a significant advancement in the AAI (Accessibility and Automation Integration) project, aiming to revolutionize user interaction by replacing coordinates with intuitive queries.
Key Components of FindNode:
FindNode leverages Selector, positioned atop Accessibility and UI Automator frameworks, to uniquely identify UI elements. This approach negates the need for coordinate-based interactions, enabling scripts to adapt to varying device resolutions and sizes. Each UI element or container of UI elements is identified by node or sets of nodes, each node has its ID that is unique on the current screen. Selector provides different ways to search for nodes. once the nodes are acquired, many automation tasks can be done such as obtain the texts/images or perform actions such as click on button or entering the text on the text field, all without coordinates.
For example:
JS API: devices.click("OK") instead of devices.click(100, 200). Find node with text "OK", obtain coordinate and click on it. All done in runtime.
MDCC: Users click a button on the main device, the unique query of the button (e.g. "T:OK") will be sent to other devices, to search for the node and click. The intention is to provide various ways to locate nodes without using screen coordinates, there is no page up/down but "scrollToView" to locate a node, this way, the same script can run on different phones with different resolutions and screen size.
FindNode uses the JSON construct of Appium to carry out the JSON commands, the JSON format is:
{"cmd": "action", "action": "findnode", "params": {…}}
Every FindNode commands are carried out in "params" object.
Return value can have one of two types, successful run:
{"status":0, "value":"{…}"}
Or failure:
{"status":13,"value":"{<error message>}"}
TC provides "device.sendAai()" or "devices.sendAai()" to interact with FindNode:
Presumably you can do most of the task by using "device.sendAai()" and "devices.sendAai()", FindNode utilized Appium's "handler".
With the introduction of ATS Beta in Version 9.0 (Update 50) and later, ATS eliminates the need of setting up device driver and developer options. It allows each device to run FindNode scripts independently with or without Total Control. ATS also reduces the complicated settings as before, just grant ATS Accessibility and file manager permissions.
Limitations
FindNode uses Android Accessibility services to obtain information:
Query + Actions
Use our query language to find nodes on the screen and apply actions on the nodes found. We will cover Query and Actions in this manual.
For instance, to obtain the Android version in About Phone screen:
>> device.sendAai({query:"TP:basic&&T:Android version&&OY:1", action:"getText"})
{retval: '14'}
To click a button with "OK" label:
{query:"T:OK", action:"click"}
To obtain all the stock symbols:
{query:"R:.ticker", action:"getText"}
Query
Majority of the FindNode usages are about query, this is center piece of the FindNode (and part of AAI). It allows users to locate intended UI elements via different kind of useful queries on the Accessibility nodes. The entire screen is composed by many nodes, a node can be a smallest UI element or a container of smaller nodes, many are invisible. The entire screen is a tree structure starting from the single root node (some older devices have multiple root nodes). Query will be performed to obtain the nodes that meet the criteria, the intent is to reduce the large number of nodes to one or few intended nodes, users can obtain information or apply actions to the nodes.
We developed a small query language that is portable among devices and scripts and powerful enough to locate most of the nodes. The format is JSON:
{query:"<key>:<value>&&<key>:<value>&&<key>&&<key>…"}
We accept "||" and "&&" as identical separator, in the future it can be changed. Since the query is AND relationship, the query needs to meet all criterion to become "matching nodes", use of "&&" is more logical. FindNode does not offer OR relationship but the underlying Selector package provides that capability. "templates" use that construct. For certain key, value is not required.
The query string is divided into 3 sections: template, basic query and extended query. "template" (TP) will generate the initial nodes, "basic query" (BQ) will perform query from node property one at a time, "extended query" (EQ) will take output from TP and BQ and perform query across multiple nodes.
To get Main storage free size
Can use "query" + "action":
>> device.sendAai({query:"T:Main storage&&OX:1", action:"getText"})
{retval: '402 GB free'}
Can use "optional query" (OQ):
>> device.sendAai({action:"getText(T:Main storage&&OX:1)"})
{retval: '402 GB free'}
Similarly, to go back to previous screen (left arrow), can use OQ:
>> device.sendAai({action:"click(T:Storage Analysis&&OX:-1)"})
{retval: true}
To obtain the Audio size, different queries can be mixed and still obtain the identical result:
>> device.sendAai({query:"TP:all&&T:Main storage&&VG:2&&XT:Audio&&OX:1", action:"getText"})
{retval: '77 MB'}
The parser will separate the query into TP, BQ and EQ and execute them in the following order:
Template (TP)
Template query (for lack of better name) is required key for query, it generate the nodes for intended purpose. Missing template will be defaulted to "TP:more", multiple templates are not allowed, here are different templates:
Name | Description |
---|---|
all | Return all nodes. |
more | "all" option with with layout nodes removed. This is the default if template is not defined. |
basic | "more" option with child count of zero (leaf nodes). |
reduced | Return "interesting" nodes (see UI Explorer "Optimized" mode). |
findText,<text> | Return nodes with certain text or description. findText needs an extra parameter to specify text to search. |
anyText[,min][,max] | Return for nodes with content in text, optionally with certain length. |
anyDescription[,min][,max] | Return for nodes with content in description, optionally with certain length. |
textInput[,<hint text>] | Return the nodes that are "editable" that is sorted from top-left to bottom-right. Use along side with "IX" for multiple text inputs. The optional text is hint text if it is available. |
scrollable,[pos] | Return scrollable nodes, for screen more than one scrollable area, the "position" can be used to identify which scrollable area to match. |
line,<top|bottom>,<line number> | Return nodes that is top or bottom of scrollable nodes. |
Basic Query (BQ)
Basic query is queries that is purely based on the individual node information, BQ takes nodes generated from template (or default template if not specified) and apply queries. BQ is not required, if BQ is not defined, it will pass all nodes from templates to EQ.
The following are the basic queries that match information obtained from the nodes itself:
Name | Description |
---|---|
P:<package name> | Package name (S) |
C:<class name> | Class name (S) |
R:<resource ID> | Resource ID (S) |
D:<text> | Content description (S) |
T:<text> | Text (S) |
T0:<text> | Text node without Boolean property (T:<text>&&BP:none) with true |
T1:<text> | Text node without any editable (T:<text>&&BP:!editable) |
IT:<number> | Text input type (I) |
CC:<number> | Child count (I) |
ID:<hex string> | Node ID (S) |
BI:[x, y] | Return nodes contain (x,y) |
BI:[x1, y1, x2, y2] | Return nodes that enclosed by rectangle, x or y of -1 will be ignored |
BP:<prop name> | Return nodes that match boolean properties |
TD:<text> | Match text or description |
HT:<text> | Match hint text (will match getText() if not supported) |
Since the query is always performed on the existing app on the screen, "P" should not be used. The "(S)" stands for String, "I" integer, "[x, y]" is a location and "[x1, y1, x2, y2]" is a rectangle (or bounds). I/S can accept both integer and String. String is more powerful.
The keys are case insensitive, we use uppercase to differentiate keys from values.
(API 18) A screen may contain multiple apps (e.g. systemui) at the same time, by default the running app is used, use “P:<package name>” to change the app to perform query on. Action “getAllPackageNames” can be used to list all packages running at the same time.
Expanded Query (EQ)
The following are the keys for expanded query, it usually works on multiple nodes return by BQ or template. The expanded query work from left to right, same key can be applied multiple time. EQ is optional. The following are the keys for EQ:
Name | Description |
---|---|
BQ:<basic query> | Perform basic query on multiple keys in BQ. |
IX:<number> | Return one node from list of matching nodes based on position. |
OX:<number> | Offset to neighbor nodes horizontally (positive – right, negative – left). |
OY:<number> | Offset to neighbor nodes vertically (positive – down, negative – up). |
ON:<type> | Different ways to pick one node out of list of matching nodes. |
RN | Return the optimized nodes from a list of matching nodes. |
ST:<sort type> | Return sorted nodes based on the position of the nodes on screen. |
TX | Return nodes that intersect with reference node horizontally. |
TY | Return nodes that Intersect with reference node vertically. |
V:<value> | Set the value of setText, setChecked and setProgress |
VG:[level number] | Return nodes in a same view group. |
"X"<BQ key>:<value> | Perform basic query on specific key in BQ. |
All these queries are optional, you can pick and choose any field, if you are doing the query on the screen. The "(S)" stands for String, "I" integer, "[x, y]" array. I/S can accept both integer and String. String is more powerful.
The keys are case insensitive, we use uppercase to differentiate keys from values.
Matching List (ML)
After the query is performed, the matched nodes will be stored in Matching List (ML), nodes in ML are identified by node ID. Many actions use the nodes in the ML, some actions can change the ML. "save" and "load" and newer "push" and "pop" can used to save and retrieve saved ML. Use action:"getIds" to obtain the ML IDs. TX/TY/VG take one node and generate multiple nodes.
Another way to obtain the ML is to use "elements" array.
Previously search is always performed before action is started, without setting a "query", the default template can be used to construct ML, this is not effective if the action has nothing to do with nodes (e.g. openApp, function).
Starting version 16, delayed search is implemented, the search will only be performed when nodes are required.
Examples:
Previously, even query is not specified, implicitly it will apply search based on default template which is "TP:more", "getIds" will print out the node ID in ML.
>> device.sendAai({action:"getIds"})
{count: 11, retval: ['14801','1714c',...]}
>> device.sendAai({query:"TP:more", action:"getIds"})
{count: 11, retval: ['14801','1714c',...]}
Version 16's "delayed search", the search will be performed when "getIds" is called:
>> device.sendAai({action:"getIds"})
{count: 11, retval: ['14801','1714c',...]}
No search will be performed on openApp since the action does not involve ML:
>> device.sendAai({action:"openApp(Skype)"})
{retval: true}
Below example 2 searches will be performed, one is default template "TP:more", the other one is "TP:reduced". Version 16, one search of "TP:reduced" will be performed:
>> device.sendAai({action:"newQuery(TP:reduced)"})
{count: 8}
Use below command to go back to previous bahavior:
device.sendAai({action:"setConfig(selector:allowDelayedSearch,false)"})
"elements" property
"elements" accept an array of node IDs, the node IDs must be a valid node ID on the screen or it will generate error. When "elements" is specified, all the elements will be stored in ML, the search will not be performed. It is useful to apply actions on nodes in specific order. Each query with "elements" will traverse all nodes to locate corresponding node, try to put multiple IDs in the array instead of one at a time. Same node ID can appear on the elements. "elements" is the only that can compose "ML" directly, it is useful to combine with "forEach".
Examples:
device.sendAai({elements:["1234a", "5678b"], action:"getNodes"})
This query the digit buttons and build elements array and use "forEach" to click all buttons and print the result.
>> ids = device.sendAai({query:"T:/^[1-9]$/"}).retval
a71e,aadf,aea0,b622,b9e3,bda4,c526,c8e7,cca8
>> device.sendAai({elements:ids, action:"forEach(nsClick);getText(+R:.editText_result)"})
{count: 2, list: [{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]},{retval: '789456123'}]}
To retrieve one node, can use "ID:<node ID>".
device.sendAai({query:"ID:1234a", action:"getNodes(all)"})
Testing
To find the nodes via Query is not very difficult, you need is "UI Explorer" and Terminal:
Template in Detail
Template (TP) is a required field, TP returns list nodes based on the values, it is the initial list of nodes for BQ, EQ or actions. TP is a subclass of Selector, maybe able to allow users to define new template in the future. Several rules in TP:
TP:all, more, basic, reduced
These templates go thru the nodes in entire screen:
UI Explorer: 3 selections:
E.g. The first getCount without argument, will get the count of "ML", without "query", by default is "TP:more", so it has the same count as "TP:more". "+TP:all" will modify the ML, that is why the 3rd "getCount" will return the number of nodes in the ML built by "+TP:all".
>> device.sendAai({actions:[
"getCount",
"getCount(+TP:all)",
"getCount",
"getCount(+TP:more)",
"getCount(+TP:basic)",
"getCount(+TP:reduced)"
]})
{count: 6, list: [{count: 237},{count: 242},{count: 242},{count: 237},{count: 76},{count: 58}]}
TP:anyText[,min[,max]], anyDescription[,min[,max]]
This template will return the text in anyText or description in anyDescription with non-null content. Optionally can specify the minimum and maximum length (inclusive) of the text found. If no max is specified, the maximum length will not be enforced. The min/max of zero is not allowed.
Examples:
>> device.sendAai({query:"TP:anyText", action:"getText"})
{retval: ['Calculator','AC','÷','×','DEL','7','8','9','-' ,'4','5','6','+','1','2','3','%','0','.','±','=']}
>> device.sendAai({query:"TP:anyText,1,1", action:"getText"})
{retval: ['÷','×','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']}
TP:findText,<text>
Return the nodes with content in text or description that matches specified text. Text argument is required or generate an error. It will accept similar argument as "T:<…>" in "BQ".
Examples:
>> device.sendAai({query:"TP:findText,/[0-9]/", action:"getText"})
{retval: ['7','8','9','4','5','6','1','2','3','0']}
>> device.sendAai({query:"TP:findText,/^[A-Z]{1,4}$/", action:"getText"})
{retval: ['TGT','INTC','WMT','T','AAPL','SBUX','TSLA']}
>> device.sendAai({query:"TP:findText,*T", action:"getText"})
{retval: ['TGT','WMT','T']}
TP:textInput[,<hint text>]
This template returns the nodes which are "editable", the optional hint text if available (need Android 8.0 or later). If no nodes are found, it will generate an error. Alternatively, you can use "BP:editable" to obtain the same results.
Examples:
Obtain all text field:
>> device.sendAai({query:"TP:textInput"})
{count: 6, retval: ['e135','e8b7','f3fa','fb7c','106bf','10e41']}
>> device.sendAai({query:"BP:editable"})
{count: 6, retval: ['e135','e8b7','f3fa','fb7c','106bf','10e41']}
// Show hint text as text value, use "getHintText" to obtain hint text
>> device.sendAai({query:"TP:textInput", action:"getHintText"})
{retval: ['text','text','Number','Email','text password','Person Name']}
Locate the text field either thru index or hint text:
>> device.sendAai({query:"BP:editable&&IX:-1", action:"getHintText"})
{retval: 'Person Name'}
>> device.sendAai({query:"TP:textInput,text", action:"getText"})
{retval: ['text','text']}
Based on the hint of the text field: query:"TP:textInput&&T:name"
Use "setText" to change the content:
>> name = "john@noname"
>> device.sendAai({query:"TP:textInput,Person Name", action:`setText(${name})`})
{retval: true}
>> device.sendAai({query:"BP:editable&&T:Email", action:"setText(john@noname.com)"})
{retval: true}
>> device.sendAai({query:"TP:textInput&&IX:-1", action:"setText(John)"})
{retval: true}
>> device.sendAai({query:"TP:textInput&&T:text ", action:"setText(Hello)"})
{retval: true}
TP:line,<top|bottom>,<line number>
Currently, this only applies to app with scrollable container or TP:line will return null. FindNode will locate the UI elements outside of scrollable nodes, line mode will group UI elements into a series of lines, each line can have one or more nodes. Many applications have fixed UI elements on the top and bottom of the screen, specific node cannot located with conjunction of BQ or EQ. For instance "TP:line,top,1&&IX:2" will return the first line, second item or "TP:line,bottom,-1&&T:Chats" will return bottom of the line with Chats as text.
Examples:
>> device.sendAai({query:"TP:line,top,1"})
{count: 6, retval: ['1902b0','191936','192fbc','1cf0bc','19373e','194a03']}
To go back click the first item "IX:0":
>> device.sendAai({query:"TP:line,top,1", action:"click(IX:0)"})
{retval: true}
>> device.sendAai({query:"TP:line,bottom,-1"})
{count: 5, retval: ['2be5b8','2c0b42','2c1a46','2c30cc','2c3fd0']}
>> device.sendAai({query:"TP:line,bottom,-1", action:"setText(BP:editable, Hello)"})
{retval: true}
TP:scrollable[,<position>]
This template returns the on-screen items (leaf nodes) of scrollable identified by "position". "position" is the number starts from 0, top to bottom. Can obtain better result with resource ID in scrollable nodes. Default position is zero. Will raise error if cannot find scrollable node on screen. "TP:scrollable" return the leaf node on one scrollable node, "BP:scrollable" will match all the scrollable nodes on the screen.
For scrollable each item (or row/column) may contain multiple values, work well with resource ID to return specific value of the items. For instance, query:"TP:scrollable&&R:.tv_name".
Examples:
To obtains all scrollable nodes in the current screen:
>> device.sendAai({query:"TP:scrollable"})
{count: 5, retval: ['1e434','5bbba','5bf7b','5d9c2','6f70e']}
"BP:scrollable" returns all the scrollable nodes on the screen:
>> device.sendAai({query:"BP:scrollable", action:"getNodes(B)"})
{count: 5, list: [{bounds: '[0,268][1440,2738]', id: '1e434'},{bounds: '[42,457][1440,1303]', id: '5bbba'},{bounds: '[42,457][1440,1303]', id: '5bf7b'},{bounds: '[0,1303][1440,1742]', id: '5d9c2'},{bounds: '[0,1958][1440,2738]', id: '6f70e'}]}
Return all title of the Settings (in Samsung phones):
>> device.sendAai({actions:["openAndroidSetting(Settings)", "newQuery(TP:scrollable&&R:android:id/title)", "getText"]})
{count: 3, list: [{retval: true},{count: 10},{retval: ['My Name','Connections','Connected devices','Modes and Routines','Sounds and vibration','Notifications','Display','Wallpaper and style','Themes','Home screen']}]}
Basic Query in Detail
Basic query takes ML from TP and perform node-level matching, one node at a time and create a shorter ML, discard nodes that do not match. BQ extracts data from Accessibility, it contains single-node information and actions. Extend more functionalities in the query for more powerful matching (special characters) and added shortcuts to make query more concise.
Special Characters in Query
For number such as inputType or childCount, it can either accept number or string:
String can be one of the following, "\n" to match newline character:
"!" is use at the beginning of string as NOT. "!0" = not zero. "C:!/Layout/", non-layout classes.
For simplicity in examples, the standard prefix of {"cmd": "action","action": "findnode", "params":{…}} will be omitted, we will list the examples in the "param" object and use JavaScript object notations to reduce number of double of quotes.
Shortcuts
Two shortcuts are available for class name (C) and resource ID (R), prefix with "." for substitutions of commonly use prefixes:
The "." notation can only be used for normal matching and "!" matching, other types of matching (e.g. regular expression and wildcard) needs to use full form.
P:<package name>
On screen, it may have more than one application running, by default, omitting “P” in query, the active app (app takes the biggest screen area) will be used. You may want to obtain information from another app such as systemui. UI Explorer can choose the application or use ”getAllPackageNames” command to find out all running package names on the screen.
Examples:
To obtain the current battery % from system bar, one of the following methods, since OQ does not support “P”, use “newQuery” command:
>> device.sendAai({query:"P:com.android.systemui&&R:.battery_percentage_view", action:"getText"})
{retval: '61%'}
> device.sendAai({query:"P:com.android.systemui", action:"getText(R:.battery_percentage_view)"})
{retval: '61%'}
>> device.sendAai({action:"newQuery(P:com.android.systemui); getText(R:.battery_percentage_view)"})
{count: 2, list: [{retval: 28},{retval: '61%'}]}
To click home button in navigation bar:
device.sendAai({query:"P:com.android.systemui", action:"click(R:.home)"})
{retval: true}
C:<class name>, R:<resource ID>
All nodes contain package name and class name and optional resource ID. Since the matching is performed on the screen that running the application, "P" will default to the app running, other "P" does not make sense. So "P" is not required.
T:<text>, D:<text>, TD:<text>, T0:<text>, T1:<text> (T0 & T1 in API 18)
Two most important keys, the node with information usually carry text or description, this is much more accurate and faster than OCR. "until" command can monitor if the text or description has been changed. T:<text> with special characters are often used. "TD" match text or description, match is successful if any one of them match. “T0” is “T&&BP:none”, “T1” is “T&&BP:!editable”. T1 is used to prevent finding text in the text field. T0 is used to find the label that has no actions (ancestor nodes may have actions).
Example:
device.sendAai({query:"T:OK", action:"click"})
Click OK button.
device.sendAai({query:"T1:Name&&OX:1", action:"getText"})
This will not find text field with the “Name” on it.
Bound In (BI):
Bound in takes screen coordinate or bounds and return the nodes based on the criteria below. This mainly for screen manipulation such as UI Explorer or debugging purposes. FindNode's object mode attempt to find nodes in devices independent of resolutions and sizes, "BI" coordinate-based will not work on all devices.
Example:
{query:"BI:[-1, 1000]&&IX:-1", action:"click"}
{query:"BI:[0, 0, 1000, 1000]", action:"getBounds"}
CC:<child count>, IT:<input type>
These are property found in the node, seldom use. CC is number of children, input type usually a number text field.
BP:<property>
Boolean property is important in the node, these properties carry true or false value. Multiple values are allowed, "!" in front of the property name will match false. Here are the properties:
Multiple boolean properties are allowed, separated by comma, it is optional to include within quotes or not. “none” and “!none” cannot be combined with other properties or it will generate error.
Usage:
{query:"BP:<prop name>, !<prop name>, ..."}
Examples:
>> device.sendAai({query:"TP:textInput&&BP:clickable"})
{count: 1, retval: ['27ebc0']}
>> device.sendAai({query:"TP:textInput&&BP:'clickable,enabled'"})
{count: 1, retval: ['27ebc0']}
>> device.sendAai({query:"TP:textInput&&BP:clickable,enabled"})
{count: 1, retval: ['27ebc0']}
Use "getNodes(BP)" to output all Boolean properties:
>> device.sendAai({query:"TP:textInput&&BP:clickable,enabled", action:"getNodes(BP)"})
{count: 1, list: [{booleanProperties: {checkable: false, checked: false, clickable: true, editable: true, enabled: true, focusable: true, focused: false, longClickable: true, multiLine: true, scrollable: false, selected: false, visibleToUser: true}, id: '27ebc0'}]}
Starting API 14, "not" (!) is supported, you can not only find the true value but also false value by using "!".
Assuming loop thru this to obtain values until no further (arrow shows as gray):
>> device.sendAai({query:"R:.btn_next_period&&BP:!enabled", action:"getCount"})
{count: 1}
Reverse is not true:
>> device.sendAai({query:"R:.btn_next_period&&BP:enabled", action:"getCount"})
null
>> lastError()
No match found
"checkable" is usually "clickable", to remove checkable from the list:
>> device.sendAai({query:"BP:clickable", action:"getCount"})
{count: 26}
>> device.sendAai({query:"BP:checkable", action:"getCount"})
{count: 10}
>> device.sendAai({query:"BP:clickable,!checkable", action:"getCount"})
{count: 16}
Basic Query Example
device.sendAai({}) → device.sendAai({query:"TP:more", action:"getIds"})
Version 16, TP:more will be performed when "getIds" is called.
>> device.sendAai({})
{count: 9, retval: ['95b0','b779','bb3a','9d32','a0f3','a4b4','a875','ac36','aff7']}
>> device.sendAai({query:"TP:more", action:"getIds"})
{count: 9, retval: ['95b0','b779','bb3a','9d32','a0f3','a4b4','a875','ac36','aff7']}
>> device.sendAai({query:"T:/./", action:"getCount"})
{count: 25}
>> device.sendAai({query:"D:/./", action:"getCount"})
{count: 44}
>> device.sendAai({query:"TD:/./", action:"getCount"})
{count: 69}
Can include more complex query like:
>> device.sendAai({query:"T:/^.$/&&C:.Button&&R:.button_*", action:"getText"})
{retval: ['÷','×','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']}
If the match cannot be found, it will return null, use "lastError()" to display error message:
>> device.sendAai({query:"T:X", action:"getText"})
null
>> lastError()
No match found
>> device.sendAai({query:"X:X", action:"getText"})
null
>> lastError()
Error:Cannot find the key for "X"
Can check for null programmatically.
Expanded Query in detail
There are the queries left that is not BQ or TP. Keys in EQ are often calculations and heuristics that involve multiple nodes. EQ performs the keyed functions from left to right, manipulate along the way, hopefully at the end, the ML contains nodes that users want. The ML is uninteresting without actions, resulting ML will be passed to actions to apply actions or extract information. Keys can be applied multiple times "OX:1&&OX:1" is allowed.
Index (IX) M → O
IX:<number>. Return a node from ML nodes based on the position starting from zero (IX:0). IX can be negative value in which case the position is in reversed order (e.g. -1 is the last node).
Examples:
>> device.sendAai({})
{count: 9, retval: ['95b0','b779','bb3a','9d32','a0f3','a4b4','a875','ac36','aff7']}
>> device.sendAai({query:"IX:1"})
{count: 1, retval: ['b779']}
>> device.sendAai({query:"IX:-2"})
{count: 1, retval: ['ac36']}
Single Node Selection (ON) M → O
ON:<option> Return one node from multiple nodes in ML, currently 3 options are available. Only limited options at this time, will expand more options in the future.
These are the options:
Offset (OX & OY) O → O
OX:<integer> and OY:<integer>. Use the offset to find neighboring node base on one reference node, OX find horizontal position, positive integer moves right, negative integer moves left. For OY, positive integer moves down, negative integer moves up.
Examples:
In About phones:
>> device.sendAai({query:"T:Model name&&OX:1", action:"getText"})
{retval: 'Galaxy S22 Ultra'}
>> device.sendAai({query:"T:Model number&&OX:1", action:"getText"})
{retval: 'SM-S908U1'}
>> device.sendAai({query:"T:Android version&&OY:1", action:"getText"})
{retval: '12'}
Another example:
>> device.sendAai({query:"T:5&&OX:1", action:"getText"})
{retval: '6'}
>> device.sendAai({query:"T:5&&OY:-1", action:"getText"})
{retval: '8'}
// OX & OY orders may get different results
>> device.sendAai({query:"T:0&&OY:-1&&OX:2", action:"getText"})
{retval: '×'}
>> device.sendAai({query:"T:0&&OX:2&&OY:-1", action:"getText"})
{retval: '−'}
ViewGroup (VG) O → M
ViewGroup: VG or VG:<level>. This command accepts optional level, if level is not specified, 1 will be used as level. It accepts a node (from query), will do parent traversal until a "group" is found, level is used to specify when to stop. A "group" is layout that occupy 85% of the screen width or "ViewGroup" in the class. ML will contain the nodes within the group, with the first node is ViewGroup/Layout node itself that contains all the nodes. It is very useful to logically group interesting nodes and use "addQuery" or BQ to search the nodes within the group. "setConfig(selector:viewGroupWidthRatio,
Related action: "viewGroup".
Examples:
Select the right current before charging, first level of VG does not reach the intended nodes:
>> device.sendAai({query:"T:Start Charging&&VG"})
{count: 2, retval: ['53b16','53ed7']}
To obtain the charging group, use level 2. The first node will be the node that encloses all the nodes in the group (green rectangle). First node is useful to "showOnScreen" to ensure the entire group is visible on the screen.
>> device.sendAai({query:"T:Start Charging&&VG:2"})
{count: 24, retval: ['4a8af','4ac70','4b031','4b3f2','4bb74','4bf35','4c2f6','4c6b7','4ca78','4ce39','4d1fa','4d5bb','4d97c','4dd3d','4e0fe','4ec41','4f002','4f3c3','53755','53b16','53ed7','54298','54659','54a1a']}
To obtain the current Ampere:
>> device.sendAai({query:"T:Start Charging&&VG:2&&XT:/^[0-9]+ A$/", action:"getText"})
{retval: '32 A'}
To obtain and change, reduce by 1 Amp:
>> device.sendAai({query:"T:Start Charging&&VG:2&&XT:/^[0-9]+ A$/", actions:"getText;click(OX:-1);refresh;getText"})
{count: 4, list: [{retval: '32 A'},{retval: true},{retval: true},{retval: '31 A'}]}
Use "repeat" to reduce by 5 Amp:
>> device.sendAai({query:"T:Start Charging&&VG:2&&XT:/^[0-9]+ A$/", actions:"getText;repeat(5,nsClick(OX:-1));refresh;getText"})
{count: 4, list: [{retval: '32 A'},{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]},{retval: true},{retval: '27 A'}]}
Intersect (TX & TY) O → M
Intersect: TX and TY. These commands take a node as a reference node and change the ML to contain all the nodes that intersect horizontally (intersectX) or vertically (intersectY). The matcher will make use of the bounds (top/bottom for intersectX, left/right for intersectY) of the reference node to search for intersected nodes.
The size (width or height) of the reference node is very important, if a wide reference node is picked, TY will match most of the nodes on screen. Pick reference node wisely.
This command will take first node and generate multiple nodes in ML, you can use "addQuery" to set more constraints on them.
See also: "intersectX" and "intersectY".
Examples:
On the offset calculator example above:
>> device.sendAai({query:"T:2&&TX"})
{count: 6, retval: ['bb4b','bf0c','c2cd','e0d5','e496','111a2']}
>> device.sendAai({query:"T:2&&TY"})
{count: 6, retval: ['1d115','1d4d6','a886','b3c9','bf0c','ca4f']}
But if you pick result as reference node with wide scroll entire screen, OX will get one node (itself) and OY will get all nodes.
>> device.sendAai({query:"R:.result&&TX"})
{count: 1, retval: ['1d4d6']}
>> device.sendAai({query:"R:.result&&TY"})
{count: 23, retval: ['1d897','1d4d6','d592','f39a','a4c5','a886','ac47','dd14','1029e','b008','b3c9','b78a','e0d5','bb4b','bf0c','c2cd','111a2','e496','c68e','ca4f','ce10','120a6','e857']}
Reduce nodes (RN) M → M
Reduce Nodes: RN. This is the one of the important features of FindNode, this feature accepts a list of node IDs, the intent to reduce the number of visible nodes. "TL", "BL", "OX/OY" and "intersect" all use the reduceNodes. Note that the reduceNodes will pick smaller nodes if bigger nodes are fully contain the smaller nodes. For instance, if bigger ".Button" node encloses smaller ".TextView" node, the ".TextView" node will be selected over ".Button" node, the "click" on the .TextView node will still work. See UI Explorer "Optimize" mode.
Examples:
>> device.sendAai({}).count
242
>> device.sendAai({query:"RN"}).count
55
Related action:"reduceNodes".
Sort (ST) M → M
Sort: ST:X|Y|YX. Sort based on the bounds of the nodes, usually the way three is structure and the search, quite likely it has been ordered from top to bottom. Unless the order is important, there is little reason to sort.
X – compare the left bounds of the nodes
Y – compare the top bounds of the nodes
YX – compare top bounds and then left bounds of the nodes
Examples:
>> device.sendAai({query:"C:.Button", action:"getText"})
{retval: ['AC','÷','×','DEL','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']}
>> device.sendAai({query:"C:.Button&&ST:Y", action:"getText"})
{retval: ['AC','÷','×','DEL','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']}
>> device.sendAai({query:"C:.Button&&ST:YX", action:"getText"})
{retval: ['AC','÷','×','DEL','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']}
>> device.sendAai({query:"C:.Button&&ST:X", action:"getText"})
{retval: ['AC','7','4','1','0','÷','8','5','2','.','×','9','6','3','±','DEL','-','+','%','=']}
Related action: sort
Basic query(BQ) M → M
Basic query: BQ:'<basic query>' or "<basic query>". BQ is the command in EQ. Consider the following query (Obtained from UI Explorer Custom Query):
Examples:
Query: T:Overall Cpu Usage&&VG
"VG" is used to obtain the group, if we want to obtain the number of "Cpu Cores", the following query will not work:
T:Overall Cpu Usage&&VG&&T:Cpu Cores&&OX:1 → T:Cpu Cores&&OX:1&&VG
Two "T" commands are basic commands, second T will overwrite first T, since BQ commands applied before EQ. If we need the functionalities of basic query in EQ so the text can be matched. There 3 ways to do it:
Both will work:
>> device.sendAai({query:"T:Overall Cpu Usage&&VG&&XT:Cpu Cores&&OX:1", action:"getText"})
{retval: '8'}
>> device.sendAai({query:"T:Overall Cpu Usage&&VG&&BQ:'T:Cpu Cores'&&OX:1", action:"getText"})
{retval: '8'}
Starting version 14, there is a flag to allow any BQ after EQ command, in this example, "T:Cpu Cores" is considered as part of EQ
>> device.sendAai({action:"setConfig(selector:allowBqAfterEq, true)"})
{retval: true}
>> device.sendAai({query:"T:Overall Cpu Usage&&VG&&T:Cpu Cores&&OX:1", action:"getText"})
{retval: '8'}
Value (V) M → M
This should not be part of the query since it performs "actions" to set the values of the nodes, however it is convenient to construct a query with a value associated. In fact, this key will result the "set" actions to be performed to set the values. This only applies to the first node in the ML, it can be set either in "query" or "newQuery" action, for query, nothing will be returned if set correctly, wrong data type will result in error. 3 set types are supported:
Node Prop | Action | Type | Examples |
---|---|---|---|
Checkable | setChecked | boolean | V:true, V:false |
progressible | setProgress | float/integer | V:95.37, V:100 |
Editable | setText | string | V:I am John |
Examples:
To add new entry in Calendar:
By default, the action is "getIds":
>> device.sendAai({query:"T:Title&&V:Online Gaming Tournament&&OY:1&&OX:1&&V:true"})
{count: 1, retval: ['e72e2']}
>> device.sendAai({action:"newQuery(T:Title&&V:Online Gaming Tournament&&OY:1&&OX:1&&V:true);getIds"})
{count: 2, list: [{count: 1},{count: 1, retval: ['1a2e06']}]}
>> device.sendAai({query:"T:Title&&V:Online Gaming Tournament&&OY:1&&OX:1&&V:111"})
null
>> lastError()
Wrong format, expect boolean value
Actions
FindNode offers many action commands, use "function" can create more actions. "action" can accept a single string command or an array of strings with multiple commands. Query is optional, if query is not specified, it will include the default template. If query is specified, the query output will be saved to ML, the commands/actions require nodes will be obtained from underlying ML either via query or optional query, use action:"getIds" to obtain ML content. If no action is specified, action:"getIds" is used. Use of "action" and "actions" are identical.
Use "device.sendAai()" or "devices.sendAai()" to communicate with FindNode. For multiple devices, a thread will be created for each device to carry out the query and actions.
When sendAai encounter error from FindNode or timeout, it will return null with "lastError()" contain error message.
>> device.sendAai({query:"T:Not there", action:"getText"})
null
>> lastError()
No match found
For actions that require only one node (e.g. "click"), the first node in the ML will be chosen (ON, IX, OX and OY can change that). There are many commands in actions, improvements will be added on new versions that may introduce incompatibility, please use action:"version" to ensure the version supports the commands required. Some commands require additional arguments (the arguments will be enclosed in parenthesis).
"function" can create more actions, see "function" command for explanation and examples.
Data Type
FindNode support the following data type in arguments, version 17 added function, array and object data type:
String – single quote, double quote or unquote non-special characters in argument.
Integer – any number.
Double – any number with decimal point.
Boolean – true or false.
Function – function name with argument inside parenthesis (Version 17).
Array – elements separated by comma inside brackets (Version 17). Array in "retval" will also include a "count".
Object – elements with key value pairs inside braces (Version 17).
null – special character, useful for comparison.
Examples:
>> device.sendAai({action:"echo(this is a string)"})
{retval: 'this is a string'}
>> device.sendAai({action:"echo('this is a string')"})
{retval: 'this is a string'}
>> device.sendAai({action:'echo("this is a string")'})
{retval: 'this is a string'}
>> device.sendAai({action:'echo(1234567)'})
{retval: 1234567}
>> device.sendAai({action:'echo(1234567.89)'})
{retval: 1234567.89}
>> device.sendAai({action:"echo(true)"})
{retval: true}
>> device.sendAai({action:"echo(false)"})
{retval: false}
>> device.sendAai({action:"echo(getText(T:/^\\d$/))"})
{count: 10, retval: ['7','8','9','4','5','6','1','2','3','0']}
>> device.sendAai({action:"echo([101,102,103,104,105])"})
{count: 5, retval: [101,102,103,104,105]}
>> device.sendAai({action:"echo({a:1, b:2, c:[1,2,3,4,5]})"})
{retval: {a: 1, b: 2, c: [1,2,3,4,5]}}
>> device.sendAai({action:"echo(null)"})
{retval:null}
Single Action
In single action, only one command is present in action. In this case, the returning result is in "retval" (or null if error is found).
device/devices.sendAai({query:"…", action:<action>})
Example:
>> device.sendAai({query:"T:Product name&&OX:1", action:"getText"})
{retval: 'Galaxy S22 Ultra'}
// Absence of query, the default query of "TP:more" will be applied
>> device.sendAai({action:"getText(T:Product name&&OX:1)"})
{retval: 'Galaxy S22 Ultra'}
Error message:
>> device.sendAai({action:"getText(T:Product name&&OX:2)"})
null
>> lastError()
Query Error: Node not found on offset
Multiple Actions
Multiple actions are allowed in FindNode, many of the FindNode operations are done in multiple actions, the multiple actions will be carried out from left to right, one at a time, the result is an array which store the output of each action, if one of the actions fail, it will return null with lastError() showing the index of the action (zero-based) that fail and error message. There are 2 ways to define multiple actions:
Array:
actions:["action 1", "action 2", …]
Semicolon in a String (version 14+):
actions:"<action 1>;<action 2>"
device/devices.sendAai({query:"…", actions:[<action 1>, <action 2>, …]})
device/devices.sendAai({query:"…", actions:"<action 1>;<action 2>, …"})
Example:
Here are 2 ways to execute multiple actions, the count will show how many command and list will show a list of output of actions. List[0] is the output of "setText", list[1] is the output of refresh and list[2] is the output of gettext.
>> device.sendAai({query:"TP:textInput", actions:["setText(Hello)", "refresh", "getText"]})
{count: 3, list: [{retval: true},{retval: true},{retval: 'Hello'}]}
>> device.sendAai({query:"TP:textInput", actions:"setText(Hello);refresh;getText"})
{count: 3, list: [{retval: true},{retval: true},{retval: 'Hello'}]}
Error message will show the position (zero-based) of the error:
>> device.sendAai({query:"TP:textInput", actions:["setText(Hello)", "refresh", "noAction", "getText"]})
null
>> lastError()
[2 - noAction] Invalid action: noAction
Variables in Actions and Query
If arguments of an action are determined at runtime, there are 3 ways to do variable substitutions:
If you want to use variable in the action commands, can use "+" to concatenate multiple items, with quotes, it can become cumbersome:
JavaScript string functions:
var input = "Hello"
>> "setText('" + input + "')"
setText('Hello')
>> 'setText("' + input + '")'
setText("Hello")
>> 'setText(' + input + ')'
setText(Hello)
>> device.sendAai({query:"TP:textInput", action:["setText('" + input + "')", "click(OX:2)"]});
{count: 2, list: [{retval: true},{retval: true}]}
JavaScript template literals
Starting version 14, variable substitution (or expression) can be applied to string with backtick with ease, since it is basic construct of JavaScript, it can apply to both query and actions. With the introduction of multiple actions in string (separated by ";"), the substitution is much simpler, it also allow multiple actions across multiple lines.
Below is an example, it is good idea to surround ${input} with quotes in case input contains space, with the
>> device.sendAai({query:"TP:textInput", action:`setText("${input}");click(OX:2)`})
{count: 2, list: [{retval: true},{retval: true}]}
To retrieve all numbers in calculator:
>> var text = "/^[0-9]$/"
>> device.sendAai({query:`T:${text}`, action:"getText"})
{retval: ['7','8','9','4','5','6','1','2','3','0']}
Calling function to obtain class and description:
>> var c=()=>".CheckBox"
>> var x="T,D"
T,D
>> var d=()=>"Sunday"
>> device.sendAai({query:`C:${c()}&&D:${d()}`,action:`getNodes("${x}")`})
{count: 1, list: [{description: 'Sunday', id: '509a7', text: 'S'}]}
>> var disp=()=>"getNodes('${x}')"
>> disp()
getNodes('T,D')
>> device.sendAai({query:`C:${c()}&&D:${d()}`,action:`${disp()}`})
{count: 1, list: [{description: 'Sunday', id: '509a7', text: 'S'}]}
Multiple actions (create a function called “enterText”):
>> device.sendAai({action:"function(enterText(Hello))",
enterText:`setText(BP:editable, %1);
if(getText(BP:editable) != %1, error(Not match));
click(BP:editable&&OX:2);
sendKey(Back)
`})
aaix:
A simple helper function on pre-version 14 to make actions easier to write, with the introduction of JavaScript template literals, there is no reason to use "aaix".
>> aaix("setText", input)
setText("Hello")
>> aaix("sendKey", tcConst.keyCodes.KEYCODE_HOME, 0)
sendKey(3, 0)
>> device.sendAai({query:"TP:textInput", action:[aaix("setText", input), "click(OX:2)"]})
{count: 2, list: [{retval: true},{retval: true}]}
>> device.sendAai({action:aaix("sendKey", tcConst.keyCodes.KEYCODE_HOME, 0)})
{retval: true}
If the argument is string, below 3 examples are identical:
action:["intersectX(OY:1, IX:2)", "getText"]
action:["intersectX('OY:1', 'IX:2')", "getText"]
action:[aaix("intersectX", "OY:1", "IX:2"), "getText"]
Most of the actions have optional query string in first argument, for instance:
{query:"T:John", action:"click"}
{actions:["newQuery(T:John)", "click"]} → {action:"click(+T:John)"}
{actions:["addQuery(T:Mary)", "click"]} → {action:"click(T:Mary)"}
The added query string will make command more concise. "newQuery" prefixed by "+" will clear ML, make the search and reconstruct the ML. "addQuery" will search based on the existing ML.
Special Character in Action Parameters
There are several characters in action parameters that are meaningful for FindNode parser, starting version 16, these 4 characters can be escaped by prefixed with "\\", "\\" will be translated to "\" in Java.
setText(I\\'m John)
setText(John\\'s Book)
Since FindNode parser consider unknown text type in parentheses is considered as text, you do not need to protect text inside parentheses, just use escaped character:
>> device.sendAai({query:"T:Field 7&&OX:1", action:"setText(I\\'m John\\, she is Mary\\;);getText"})
{count: 2, list: [{retval: true},{retval: 'I'm John, she is Mary;'}]}
Domino pizza's app is "Domino's", prior version 16, it cannot be run, now it can use escape character:
>> device.sendAai({action:"openApp(Domino\\'s)"})
{retval: true}
Synchronous Actions
Great efforts have been put in to ensure the actions are synchronous, when a "click" is called, not only ensure the click is pressed synchronously, it will wait for certain "events" to ensure the rendering "starts" to happen, cannot ensure the rendering is completed. FindNode offers several ways to perform such check:
Optional Query (OQ)
Actions require nodes will obtain from the ML, actions require one node, will obtain the first node from ML. However, users can change the behavior by adding option query (OQ) in actions.
E.g.{query:"T:OK", action:"click"} → {action:"click(T:OK)"}
All OQ is optional, if OQ is not specified, the actions will take the nodes from the ML. If actions require one node (e.g. click), the first node in the ML will be used, if the actions support multiple nodes (e.g. getNodes or getText), the entire ML is used. Limited capability in "TP", any TP requires parameter will generate error.
OQ is regular query language with different types identified by different prefixes, OQ likes other queries can generate one or multiple nodes, couple of query types will modify ML. Here are the supported 3 prefixes:
Assuming calculator, the numbers 1 to 9 in 3 rows, "7,8,9", "4,5,6" and "1,2,3" (top to bottom):
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(T:5)", "getText"]})
{count: 2, retval: [{retval: '5'},{retval: ['7','8','9','4','5','6','3']}]}
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(+T:5)", "getText"]})
{count: 2, list: [{retval: '5'},{retval: '5'}]}
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(*T:5)", "getText"]})
{count: 2, list: [{retval: '5'},{retval: '5'}]}
Here is the difference between regular and "*":
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(OX:1)", "getText(OX:1)", "getText"]})
{count: 3, list: [{retval: '8'},{retval: '8'},{retval: ['7','8','9','4','5','6','3']}]}
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(*OX:1)", "getText(*OX:1)", "getText"]})
{count: 3, list: [{retval: '8'},{retval: '9'},{retval: '9'}]}
Calculator, return printable and non-digit:
>> device.sendAai({action:"getCount(+TP:anyText&&T:/[^0-9]/);getText(T:/^.$/)"})
{count: 2, list: [{retval: 11},{retval: ['÷','×','-','+','%','.','±','=']}]}
It looks like "*" and "+" is similar, consider the following, the "*" matches the list from "ML" (T:2 is not in ML), "+" search the entire screen content.
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(+T:1)", "getText"]})
{count: 2, list: [{retval: '1'},{retval: '1'}]}
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(*T:1)", "getText"]})
null
>> lastError()
[0 - getText] Query Error:No match found
Since regular query (no prefix) and query with updating ML ("*" prefix) will use ML, use of "TP" is only allowed in "newQuery" or "+" prefix:
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(TP:all)", "getCount"]})
null
>> lastError()
[0 - getText] Query Error:Key "TP" is not permitted in this query
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(+TP:all)", "getCount"]})
{count: 2, list: [{retval: [{},{},{},{},{},{},{},'Calculator',{},{},{},{},'AC','÷','×','DEL','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']},{retval: 32}]}
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(*TP:all)", "getCount"]})
null
>> lastError()
[0 - getText] Query Error:Key "TP" is not permitted in this query
Expressions Inside Functions (VFC) Version 17
For "if", "assert", "echo", "log" and "return", the function can include function inside parameter, the return value of inner function will be passed to the outer function. If the "comparison expression" is used, it will return Boolean result based on the comparison. The parameter in the usage will show "
Format 1: function() // Function call with return value of "retval".
Format 2: function().<key> // Function call with property access, will return the value of the key.
Format 3: function()[.<key>] <comp operator> <value> // Comparison expression, will return true or false
Several notes:
Examples:
The function is required in left parameter or error will be generated:
>> device.sendAai({action:"if(1==1, echo(true), echo(false))"})
null
>> lastError()
Wrong format, expect comparison expression
>> device.sendAai({action:"if(echo(1)==1, echo(true), echo(false))"})
{retval: true}
Use "()" in parameter to identify string from function:
>> device.sendAai({action:"echo(getCount)"})
{retval: 'getCount'}
>> device.sendAai({action:"echo(getCount())"})
{retval: 59}
For the 3rd form: comparison expression is very useful for "if" and "assert". Without comparison operator, the value of the function call (and optional key) can be used to determine the Boolean value (Only in "if" and "assert", other function will accept as value).
The true/false is similar to JavaScript (except Array):
The following is false:
The following is true:
Example:
>> device.sendAai({action:"function(clickExists)", clickExists:"if(exists(T:%1), click(T:%1), return(false))"})
{retval: true}
>> device.sendAai({action:"clickExists(Skype)"})
{retval: true}
>> device.sendAai({action:"clickExists(NotThere)"})
{retval: false}
The comparison operator can be one of the following:
== or = // Test if test if the left parameter is equal to the right parameter
> // Test if the left parameter is greater than the right parameter
< // Test if the left parameter is less than the right parameter
>= // Test if the left parameter is greater than or equal to the right parameter
<= // Test if the left parameter is less than or equal to the right parameter
!= // Test if the left parameter is not equal to the right parameter
The left parameter returned from a function will be translated to the same format as right parameter. Similar to JavaScript if the returned text contains digits in the front, it will parse only the digits until non-digit is found. So "85%" will be translated to integer of 85.
There is a "null" can be used for function return value or right parameter, it is valid to perform a comparison with null, for instance "if(getText() != null, click)".
Examples:
The function is required in left parameter or error will be generated:
>> device.sendAai({action:"if(1==1, echo(true), echo(false))"})
null
>> lastError()
Wrong format, expect comparison expression
>> device.sendAai({action:"if(echo(1)==1, echo(true), echo(false))"})
{retval: true}
Click if the T:OK is available
>> if(exists(T:OK), click(T:OK))
{retval: true}
Or write a function:
>> device.sendAai({action:"function(clickText)",clickText:"if(exists(T:%1),
click(T:%1))"})
{retval: true}
>> device.sendAai({action:"clickText(OK)"})
{retval: true}
Check Android version:
>> device.sendAai({action:"function(getAndroidVersion)", getAndroidVersion:"return(getText(T:Android version&&OY:1))"})
{retval: true}
>> device.sendAai({action:"getAndroidVersion"})
{retval: '14'}
>> device.sendAai({action:"function(isAndroid14OrLater)", isAndroid14OrLater:"if(getAndroidVersion() >= 14, return(true), return(false))"})
{retval: true}
>> device.sendAai({action:"isAndroid14OrLater"})
{retval: true}
Many apps have several screens deep, to go to top of the app, you need to check if the top screen is found otherwise use Back button until it is found, you can run until true is found:
>> device.sendAai({action:"function(goBack)", goBack:"if(exists(T:%1), return(true), sendKey(Back));return(exists(T:%1))"})
{retval: true}
>> device.sendAai({action:"goBack(WhatsApp)"})
{retval: false}
>> device.sendAai({action:"goBack(WhatsApp)"})
{retval: true}
Show the output of 3 different format:
>> device.sendAai({action:"echo(listAndroidSettings())"})
{retval: ['SETTINGS','ACCESSIBILITY_SETTINGS', ..., 'ZEN_MODE_PRIORITY_SETTINGS']}
>> device.sendAai({action:"echo(listAndroidSettings().count)"})
{retval: 70}
>> device.sendAai({action:"echo(listAndroidSettings().count > 50)"})
{retval: true}
>> device.sendAai({action:"echo(listAndroidSettings().count > 100)"})
{retval: false}
Array and object examples:
>> device.sendAai({action:"echo({})"})
{retval: {}}
>> device.sendAai({action:"echo({a:1, b:2})"})
{retval: {a: 1, b: 2}}
>> device.sendAai({action:"echo({a:1, b:2, c:[100,200,300]})"})
{retval: {a: 1, b: 2, c: [100,200,300]}}
>> device.sendAai({action:"echo([])"})
{retval: []}
// Array will put count automatically
>> device.sendAai({action:"echo([1,2,3])"})
{count: 3, retval: [1,2,3]}
Array comparison:
>> device.sendAai({action:"if(echo([1,2,3]) == [1,2,3], echo(true), echo(false))"})
{retval: true}
>> device.sendAai({action:"if(echo([1,2,3]) == [1,2,5], echo(true), echo(false))"})
{retval: false}
Object comparison, order not important:
>> device.sendAai({action:"if(echo({a:1, b:2}) == {b:2, a:1}, echo(true), echo(false))"})
{retval: true}
>> device.sendAai({action:"if(echo({a:1, b:2}) > {b:2, a:1}, echo(true), echo(false))"})
null
>> lastError()
This data type only allow == or !=
Function return object:
>> device.sendAai({action:"function(getObj)", getObj:"return({a:37,b:[100,200,300],c:{x:50, y:60}})"})
{retval: true}
>> device.sendAai({action:"if(getObj() == 37, echo(true), echo(false))"})
null
>> lastError()
getObj() == 37: Unmatched or unsupported data type: JSONObject vs Integer
>> device.sendAai({action:"if(getObj().a == 37, echo(true), echo(false))"})
{retval: true}
>> device.sendAai({action:"if(getObj().b == [100,200,300], echo(true), echo(false))"})
{retval: true}
>> device.sendAai({action:"if(getObj().b == [100,200,400], echo(true), echo(false))"})
{retval: false}
>> device.sendAai({action:"if(getObj().c == {y:60, x:50}, echo(true), echo(false))"})
{retval: true}
// Same function
>> device.sendAai({action:"function(dupObj)", dupObj:"getObj"})
{retval: true}
>> device.sendAai({action:"dupObj"})
{retval: {a: 37, b: [100,200,300], c: {x: 50, y: 60}}}
>> device.sendAai({action:"if(getObj() == dupObj(), echo(true), echo(false))"})
{retval: true}
>> device.sendAai({action:"if(getObj().c == dupObj().c, echo(true), echo(false))"})
{retval: true}
>> device.sendAai({action:"echo(getObj().b)"})
{count: 3, retval: [100,200,300]}
Get Commands
getBounds (M)
Return the bounds (format: [left, top, right, bottom]) of the nodes in ML, it returns 2 arrays one with all the IDs the other one with all the bounds.
Usage:
getBounds[(<query>)]
Return:
The array of bounds, each bound is an array of 4-tuples: [x1, y1, x2. y2].
>> device.sendAai({query:"T:/[0-9]/", action:"getBounds"})
{bounds: [[18,1158,370,1521],[370,1158,722,1521],[722,1158,1074,1521],[18,1521,370,1884],[370,1521,722,1884],[722,1521,1074,1884],[18,1884,370,2247],[370,1884,722,2247],[722,1884,1074,2247],[18,2247,370,2610]], count: 10, ids: ['98d1','9c92','a053','a7d5','ab96','af57','b6d9','ba9a','be5b','c5dd']}
getBoolProp (O)
Return the Boolean property of a node, it will return "editable", "progressible", "clickable", "checkable", "scrollable" or "longClickable". If no property is found, it will return null.
Usage:
getBoolProp [(<query>)]
Return:
The array of string properties in "retval"
>> device.sendAai({query:"T:/12:30\\sPM/&&VG", action:"getBoolProp(T:Every day&&OX:1)"})
{retval: ['checkable','clickable']}
>> device.sendAai({query:"T:/12:30\\sPM/&&VG&&PQ:'T:Every day&&OX:1'", action:"getBoolProp"})
{retval: ['checkable','clickable']}
>> device.sendAai({query:"T:ci:calculator", action:"getBoolProp"})
null
>> lastError()
No boolean property is found
getBoolPropInAncestors (O)
Similar to getBoolProp except it traverse parent to its parent until it reaches the top node tree.
Usage:
getBoolPropInAncestors [(<query>)]
Return:
The array of array of property and node ID or null if not found.
>> device.sendAai({query:"T:John", action:"getBoolPropInAncestors"})
{count: 2, retval: [['clickable','1b86f'],['scrollable','16d5b']]}
getChecked (O/M)
This command returns the boolean value of "checkable" nodes. Typically, toggle controls, checkboxes or radio buttons are checkable. For instance, class name of ".Switch" or ".Checkbox" are checkable.
Usage:
getChecked [(<query>)]
Return:
If input is one node, it will return true/false in "retval", if multiple nodes, retval will contain the array of true/false, for non-checkable nodes, it will show as "N/A". Return null if none of the checkable node is found. "BP:checkable" to find nodes that is checkable, "BP:checked" to find nodes that have been checked.
See also: setChecked
>> device.sendAai({action:"getChecked(D:Sunday&&TX)"})
{retval: [false,true,true,true,true,true,false]}
If do not wish to use description, can use class to search, no other UI element is having class of ".CheckBox":
>> device.sendAai({action:"getChecked(C:.CheckBox&&IX:0&&TX)"})
{retval: [false,true,true,true,true,true,false]}
To find all the checked description:
>> device.sendAai({action:"getDescription(D:Sunday&&R:.day_button_0&&TX&&XBP:checked)"})
{retval: ['Monday','Tuesday','Wednesday','Thursday','Friday']}
getCount (O/M)
This command returns the length of ML or OQ. If OQ is used, it will return zero if not matching is found in OQ.
Usage:
getCount [(<query>)]
Return:
Return the length of ML or OQ in "retval" (prior of version 17, it returns in "count").
Examples:
>> device.sendAai({action:"getCount"})
{retval: 26}
>> device.sendAai({action:"getCount(T:/\\d/)"})
{retval: 10}
>> device.sendAai({query:"T:Not there", action:"getCount"})
null
>> lastError()
No match found
>> device.sendAai({action:"getCount(T:Not there)"})
{retval: 0}
getDescription/getText/getHintText (O|M)
Simpler version of "getNodes", much easier to return obtain the content of text, description and hint text.
The actions will detect one or multiple nodes, it will return the value if one node is detected otherwise it will return the array of text/description or hint text.
Usage:
getDescription [(<query>)]
getHintText [(<query>)]
getText [(<query>)]
Return:
Return an array of text/description/hint text in retval if multiple nodes are found, return text/description in retval if one node is found.
>> device.sendAai({query:"TP:line,bottom,-1", postAction:"getText"})
{retval: ['Chats','Calls','Contacts','Notifications']}
// One value will not return an array
>> device.sendAai({query:"TP:line,bottom,-1&&IX:1", postAction:"getText"})
{retval: 'Calls'}
>> device.sendAai({query:"BP:editable", action:"getText"})
{retval: ['text','text','Number','Email','text password','Person Name']}
>> device.sendAai({query:"BP:editable", action:"setText(T:Number, '100');setText(T:text password, secretWord)"})
{count: 2, list: [{retval: true},{retval: true}]}
>> device.sendAai({query:"BP:editable", action:"getText"})
{retval: ['text','text','100','Email','••••••••••','Person Name']}
// getHintText will maintain the value of the original hints, getText does not
>> device.sendAai({query:"BP:editable", action:"getHintText"})
{retval: ['text','text','Number','Email','text password','Person Name']}
getFocus
Return the node receiving focus ("focusable" in getNodes(BP)) (usually text field or slider) or null if no node gains focus. If node is found, it will set the ML to the found node.
Usage:
getFocus
Return:
Return "retval" with node ID is found otherwise return null with error.
Example:
To enter the next line from the existing line
>> device.sendAai({action:["getFocus", aaix("setText", username), "addQuery(OY:1)", aaix("setText", password)]})
{count: 4, list: [{retval: '67693'},{retval: true},{count: 1},{retval: true}]}
getFuncRetval Version 17
During the execution of a function, the outputs of each command are stored. To display the output from the current function, getFuncRetval is utilized. It is recommended to use getFuncRetval as the final command within the function for optimal results. When getFuncRetval is invoked without any arguments, it retrieves the outputs from all commands executed during the current function. However, if an argument is provided, getFuncRetval will return the output corresponding to the specified command. This feature is particularly beneficial in functions that contain numerous commands, especially when the output from specific commands, such as getText, is of interest. It's important to note that getFuncRetval must be executed within the context of the function and does not affect operations like forEach and repeat. When using the search command, include only the command itself without any arguments.
Usage:
getFuncRetval[(<search command>)]
Return:
Return "retval" with array of output in JSON object format. Will generate error if the search command cannot be found.
Example:
>> device.sendAai({action:"function(testEcho)", testEcho:`
echo(This is a test to echo in different types);
echo(true);
echo(1000);
echo(3.14159);
echo({a:1, b:{x:1, y:'string'}, c:[100, 200, 300]});
getFuncRetval()`})
{retval: true}
>> device.sendAai({action:"testEcho"})
{retval: [{retval: 'This is a test to echo in different types'},{retval: true},{retval: 1000},{retval: 3.14159},{retval: {a: 1, b: {x: 1, y: 'string'}, c: [100,200,300]}}]}
To obtain the current location of Tesla:
>> device.sendAai({action:"function(getAddress)", getAddress:`
click(T:Location);
newQuery(R:.map_header_text, 1000);
getText;
sendKey(Back);
getFuncRetval(getText)`});
{retval: true}
>> device.sendAai({action:"getAddress"})
{count: 1, retval: [{retval: '9999 Ocean Drive'}]
getIds (M)
This is the default action if action is not defined, this command will return count and array of the ML IDs.
Previously it returns as “ids”, for consistency, it has been changed to “retval”.
Usage:
getids [(<query>)]
Return:
count and array of IDs.
>> device.sendAai({})
{count: 26, retval: ['7708','d8a2','e7a6','ef28','824b','860c','89cd','8d8e','914f','9510','98d1','9c92','a053','a414','a7d5','ab96','af57','b318','b6d9','ba9a','be5b','c21c','c5dd','c99e','cd5f','d120']}
>> device.sendAai({action:"getIds(C:.Button)"})
{count: 4, retval: ['8dc5','9547','9908','9cc9']}
getNodes (M)
This is used to retrieve the information of nodes. The fields can determine what to display. "getNodes" will refresh the node before retrieving information.
Usage:
getNodes
getNodes([<query>] [,<fields>])
Arguments:
fields: The optional "fields" is used to define the fields to return, multiple fields in "fields" are separated by comma. Since it has comma inside the "field", you need to use the quote. String The valid field identifiers are:
Return:
Count and array of node information.
Examples:
>> device.sendAai({action:"getNodes(T:/[0-9]/,'R,T')"})
{count: 12, retval: [{id: '9b0a', resourceId: '.editText_history', text: '1360+100'},{id: '9ecb', resourceId: '.editText_result', text: '=1460'},{id: 'b190', resourceId: '.button_seven', text: '7'},{id: 'b551', resourceId: '.button_eight', text: '8'},{id: 'b912', resourceId: '.button_nine', text: '9'},{id: 'c094', resourceId: '.button_four', text: '4'},{id: 'c455', resourceId: '.button_five', text: '5'},{id: 'c816', resourceId: '.button_six', text: '6'},{id: 'cf98', resourceId: '.button_one', text: '1'},{id: 'd359', resourceId: '.button_two', text: '2'},{id: 'd71a', resourceId: '.button_three', text: '3'},{id: 'de9c', resourceId: '.button_zero', text: '0'}]}
>> device.sendAai({query:"C:.SeekBar", action:"getNodes(RI)"})
{count: 1, retval: [{id: 'b6f0', rangeInfo: {current: 0, max: 100, min: 0, type: 0, typeString: 'int'}}]}
>> device.sendAai({query:"C:.SeekBar", action:"getNodes(BP)"})
{count: 1, retval: [{booleanProperties: {checkable: false, checked: false, clickable: false, editable: false, enabled: true, focusable: true, focused: false, longClickable: false, multiLine: false, scrollable: false, selected: false, visibleToUser: true}, id: 'b6f0'}]}
getPackageName, getCurrentPackageName (API 18), getAllPackageNames (API 18)
“getPackageName” returns the current running package name, it is the same as “getCurrentPackageName”, “getAllPackageNames” will return all current running package names on the screen.
Usage:
getPackageName
Return:
Package name in "retval".
>> device.sendAai({action:"getPackageName"})
{retval: 'com.google.android.gm'}
>> device.sendAai({action:"getAllPackageNames"})
{count: 3, retval: ['com.android.systemui','com.sec.android.app.launcher','com.sigma_rt.sigmatestapp']}
getProgress (O)
Return the values of the progress type of UI element such as slider (shorthand of "getNodes(RI)"):
Usage:
getProgress [query]
Return:
the RangeInfo in "retval" of a progressible node, return null if the node is not progressible node:
RangeInfo has one of 3 types: "float" – floating point between min and max, "int" integer between min and max and "percent" between 0 and 1. "current" contains the current value.
>> device.sendAai({action:"getProgress(C:.SeekBar)"})
{retval: {current: 41, max: 100, min: 0, type: 0, typeString: 'int'}}
See also: setProgress
getQuery/getUniqQuery (O)
Accept one node and attempt to generate a query to match the node (or multiple nodes). The query returned from "getQuery" can match multiple nodes, useful to get the text or description from multiple nodes. The query returned from "getUniqQuery" will match only one node, the query obtained will be sent to other devices to locate the nodes. So far, it is rudimentary, we will develop better query string in the future:
Usage:
getUniqQuery/getQuery - Default is to use text and description as part of the query.
getUniqQuery/getQuery(<true/false>) – true to ignore text in search query.
getUniqQuery/getQuery(<true/false>, <true/false>) – true on second argument to ignore description in search query.
Return:
return query string.
To obtain query for each node:
>> device.sendAai({query:"T:Controls&&TX"}).ids.forEach(
x => print(device.sendAai({elements:[x],action:"getUniqQuery"}).retval)
)
T:Controls&&OX:-1
T:Controls
T:Controls&&OX:1
>> var query = device.sendAai({query:"T:AAPL", action:"getQuery(true)"}).retval
>> query
C:.TextView&&R:.ticker&&CC:0
>> device.sendAai({query:query, action:"getText"})
{retval: ['MSFT','INTC','TGT','T','AAPL','SBUX','TSLA','CSCO']}
Query Commands
The following commands with one exception (waitQuery) will involve ML.
newQuery/addQuery/waitQuery/exists
All accept query string, "newQuery" will start the new query and recreate ML (similar to "+" in OQ). "addQuery" perform search within ML, place the resulting nodes in ML (similar to "*" in OQ). "waitQuery" will not change the ML, it has a timeout, will make sure the query is fulfilled before timeout expire or it will return null. Since waitQuery does not change ML, if you want to change ML like "newQuery", use "newQuery" with timeout. "exists" will return true or false to indicate if query finds matching nodes, exists with timeout only applies to OQ with "+" (new query) or error will be generated. Commands with "timeout" cannot switch package before timeout expires or it will generate error, the commands will search within the same package name (or app).
For instance: "VG" will usually follow by "addQuery" instead of "newQuery". On the other hand, click to open a new window, will need "newQuery".
newQuery(<query>[, <timeout>])
addQuery(<query>)
exists(<optional query>[, <timeout>])
waitQuey(<query>[, <timeout>]) // Default timeout = 2000ms
Return:
newQuery and addQuery return the size of resulting ML in "retval". When timeout expire waitQuery will return null otherwise "retval" will contain true.
Example:
The following commands will obtain the same result but "newQuery" will change the ML and impact subsequent actions that use ML:
>> device.sendAai({action:`waitQuery(T:${text},10000);click(T:${text})`})
{count: 2, list: [{retval: true},{retval: true}]}
>> device.sendAai({action:`newQuery(T:${text},10000);click`})
{count: 2, list: [{retval: 1},{retval: true}]}
>>device.sendAai({action:`exists(+T:${text},10000);click`})
{count: 2, list: [{retval: 1},{retval: true}]}
intersectX/intersectY (O → M)
If query is specified, the first node of the query will be used for intersect otherwise ML first node will be used. This command will take one node and generate more nodes and store in ML.
Usage:
intersect?[(<query>)]
Return:
true /false: if the operation is successful. If the operation is successful, it will return "count"
If "post query" is specified, the output of post query (see "addQuery") will be stored as ML. If "post query" is not specified, the output of intersect will replace the ML.
See also: "TX" and "TY" queries for more explanation.
Example:
Assuming calculator app
>> device.sendAai({actions:["intersectX(T:3)", "getText"]})
{count: 2, list: [{retval: 4},{retval: ['1','2','3','%']}]}
"intersect Y" may include other none visible nodes such as result or viewgroup.
>> device.sendAai({actions:["intersectY(T:3)", "getText"]})
{count: 2, list: [{retval: 8},{retval: [{},{},{},'×','9','6','3','±']}]}
Add a regular expression to ensure at least one character is visible:
>> device.sendAai({actions:["intersectY(T:3)", "addQuery(T:/.+/)", "getText"]})
{count: 3, list: [{retval: 8},{count: 5},{retval: ['×','9','6','3','±']}]}
This will work too:
>> device.sendAai({actions:["intersectY(T:3)", "getText(T:/.+/)"]})
{count: 2, list: [{retval: 8},{retval: ['×','9','6','3','±']}]}
push/pop & load/save (M)
These commands store and retrieve the ML, this is only useful for multiple commands, push/save the ML, newQuery + actions, pop/load to retrieve last save ML and continue. save/load is there for compatibility reason since they do not have limit number of ML to store. In fact, in implementation, "save" and "push", "load" and "pop" are the same. You can use multiple "save" and "load".
Usage:
push
pop
save
load
Return: Always return "retval" of true, will generate error if stack has nothing to pop.
reduceNodes (M → M)
It is identical to "RN" query, no argument. Will return number of nodes after results as "count".
Usage:
reduceNodes
Return:
Return the number of resulting nodes in "retval".
See also: Query "RN".
>> device.sendAai({query:"TP:reduced"}).count
54
>> device.sendAai({action:"reduceNodes"})
{retval: 54}
Sort (M → M)
Sort based on the node bounds. It is identical to "ST" query, same argument as ST.
Usage:
sort(X|Y|YX)
Return:
Return true/false in "retval".
See also: Query "ST".
viewGroup (O → M)
This command is similar to "VG" query, will accept a level and query. Query if defined is used to identify the first node to seach for the group. Similar to "VG", it will change ML. Previously this command is called "getViewGroup".
Usage:
viewGroup
viewGroup([<query>,] <level>)
Return:
If ViewGroup is found, it will return the count in the ML, if ViewGroup is not found, it will return null.
See also: Query "VG".
>> device.sendAai({query:"T:Battery&&VG", action:"getText(T:/\\d%/)"})
{retval: '92% available'}
>> device.sendAai({actions:["viewGroup(T:Battery)","getText(T:/\\d%/)"]})
{count: 2, list: [{retval: 6},{retval: '92% available'}]}
Action Commands
Commands in this section perform actions to the nodes.
click, nsClick, click2, longClick
Various kinds of clicks on the node, this is synchronous click, will make sure screen is starting to refreshed when the control is returned.
All the above clicks will monitor the events for the changes of the screen after the click is performed, if the screen is not changed, these clicks will fail and eventually timed out. If the screen is not changed after the click is performed, use "nsClick":
"nsClick" with also accepts 2 integers, it will perform a click on x and y coordinate if hardcoded x and y values are used the scripts are not portable will fail on new device, resolution or updated apps. Certain apps that use native interface, some of the nodes can be found (e.g. WebView), can use found nodes to calculate the missing node location and perform a click. Use "getBounds" or "getNodes(B)" to obtain the dimension (or bounds) of the node.
Usage:
click/nsClick/click2/longClick [(<query>)]
nsClick(<x>, <y>) Version 17
Return:
true or false to indicate if the operation is successful.
click, nsclick, click2, longClick
click(<query>), nsclick(<query>), click2(<query>), longClick(<query>)
Example:
>> device.sendAai({query:"T:5", action:"click"})
{retval: true}
>> device.sendAai({action:"nsClick(T:5)"})
{retval: true}
Calculator app, use button 4 dimension to calculate the location of button 5 and perform a click:
>> device.sendAai({action:"getBounds(T:4)"})
{bounds: [[18,1585,370,1981]], count: 1, retval: ['a641']}
>> var dim = device.sendAai({action:"getBounds(T:4)"}).bounds[0]
>> var x = (dim[2]-dim[0])/2 + dim[2]
>> var y = (dim[3]-dim[1])/2 + dim[1]
>> print(x + "," + y)
546,1783
>> device.sendAai({action:`nsClick(${x}, ${y})`})
{retval: true}
echo,logVersion 17
echo and log has exact same usage, echo display the output in "retval", log display the output in logcat. They accept the following types:
Usage:
echo (<VFC>)
log (<VFC>)
Return:
Examples:
>> device.sendAai({action:"echo(string); echo(100); echo(3.14159); echo(true); echo(false); echo(null)"})
{count: 6, list: [{retval: 'string'},{retval: 100},{retval: 3.14159},{retval: true},{retval: false},{retval: 'null'}]}
Perform simple arithmetic, not too useful at this time:
>> var s = "(123 + 456) * (789-456)"
>> device.sendAai({action:`setText(BP:editable,${s}); echo(getText(BP:editable))`})
{count: 2, list: [{retval: true},{retval: 192807}]}
>> device.sendAai({action:`setText(BP:editable,${s});log(getText(BP:editable))`})
{count: 2, list: [{retval: true},{retval: true}]}
Logcat will show "Log: 192807".
errorVersion 18
Will stop function execution and return null on sendAai() with lastError() contain the “User Exception: ” with error message.
Usage:
error (<error message>)
Return:
Will stop execution, no return is necessary.
Examples:
>> device.sendAai({query:"P:com.android.systemui&&R:.battery_percentage_view",
action:`if(getText() < 10,
error(battery too low),
echo(battery level is adequate))
`})
null
>> lastError()
User Exception: battery too low
forEach (M)
The action loop through each node in ML and execute the provided actions, one at a time. Several rules apply:
For multiple actions, "forEach" takes the name of JSON object, the value defines the actions either in array or string will semicolon delimiter. For single action, can put the action name in the argument.
Usage:
forEach(<action name>)
forEach(<JSON name>)
Return:
Examples:
The first line runs "getText" once, the second line run "getText" 4 times.
>> device.sendAai({query:"T:1&&TX", action:"getText"})
{retval: ['1','2','3','%']}
>> device.sendAai({query:"T:1&&TX", action:"forEach(getText)"})
{retval: [{retval: '1'},{retval: '2'},{retval: '3'},{retval: '%'}]}
>> device.sendAai({query:"R:.ticker&&T:/TSLA|AAPL|INTC|CSCO/", actions:"forEach(getTicker)", getTicker:"refresh(OX:1);getDescription(OX:1)"})
{retval: [[{retval: true},{retval: '160.25'}],[{retval: true},{retval: '190.41'}],[{retval: true},{retval: '50.51'}],[{retval: true},{retval: '29.36'}]]}
>> device.sendAai({query:"R:/.day_button_[0-6]/", action:"getText"})
{retval: ['S','M','T','W','T','F','S']}
>> device.sendAai({query:"R:/.day_button_[0-6]/", action:"forEach(setChecked(true))"})
{retval: [{changedCount: 0},{changedCount: 0},{changedCount: 0},{changedCount: 1},{changedCount: 1},{changedCount: 0},{changedCount: 1}]}
In the example document, for calculator, it fills in all operations in "elements" and use "forEach(nsClick)" to key in calculation.
function/funcExists/funcDelete/funcReset/funcList
This action command will create a new action composed:
Usage:
Create new function:
function(<action name in JSON>(arguments)) <JSON name>:<actions> in [] or ";" or single action
Delete created function:
funcDelete(<function name>)
Return true/false whether the function exists:
funcExists(<function name>)
Return list of function names as array and count (version 17):
funcList
Reset – delete all functions
funcReset
Return:
Return null if error occur or it will return true in retval.
Examples:
Assuming we want to use calculator to add 2 single-digit values:
>> device.sendAai({action:"function(add(1,2))", add:"click(T:%1);click(T:+);click(T:%2);click(T:=);getText(R:.editText_result)"})
{retval: true}
This accepts 2 substitutions %1 and %2, the default value of %1 is "1", the default value of %2 is "2".
The following will substitute first argument value with %1 ("3"), second argument with %2 ("4"), thus click "3", "+", "4", "=".
Since function return the last command return value, which is getText:
>> device.sendAai({action:"add(3,4)"})
{retval: '=7'}
To bring back old behavior, use "getFuncRetval()":
>> device.sendAai({action:"function(add(1,2))", add:"click(T:%1);click(T:+);click(T:%2);click(T:=);getText(R:.editText_result);getFuncRetval()"})
{retval: true}
>> device.sendAai({action:"add(3,4)"})
{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: '=7'}]}
The following does not specify argument, both default values will be used, thus click "1", "+", "2", "=".
>> device.sendAai({action:"add"})
{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: '=3'}]}
One argument is specified, %1 will be "9" and %2 will be the default value of "2", click "9", "+", "2", "=".
>> device.sendAai({action:"add(9)"})
{retvak: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: '=11'}]}
Check for function existence and delete a function.
>> device.sendAai({action:"funcExists(add)"})
{retval: true}
>> device.sendAai({action:"funcDelete(add)"})
{retval: true}
>> device.sendAai({action:"funcExists(add)"})
{retval: false}
>> device.sendAai({action:"add(9)"})
null
>> lastError()
Invalid action: add
Another example: This function "sendText" is used to send text in Skype, enter text in text field and click send button:
>> device.sendAai({action:"function(sendText(Hello))", sendText:[
"newQuery(TP:textInput)",
"setText(%1)",
"click(OX:2)",
"getFuncRetval"
]})
{retval: true}
>> device.sendAai({action:"sendText(How are you?)"})
{sendText: [{count: 1},{retval: true},{retval: true}]}
>> var input = "Good morning"
>> device.sendAai({action:`sendText(${input})`})
{sendText: [{count: 1},{retval: true},{retval: true}]}
The function below will find a name, click to chat screen, send a text using "sendText" function just created, click back key to go back to main screen:
>> device.sendAai({action:"function(findAndSend)", findAndSend:"scrollIntoView(T:%1);click;sendText(%2);sendKey(Back);getFuncRetval"})
{retval: true}
>> device.sendAai({action:"findAndSend(John Doe, Nice to meet you)"})
{findAndSend: [{retval: true},{retval: true},{sendText: [{count: 1},{retval: true},{retval: true}]},{retval: true}]}
Another example: To obtain the SolarEdge production from mySolarEdge application:
>> device.sendAai({actions:"function(getProduction)", getProduction:[
"openApp(mySolarEdge)",
"waitQuery(T:Lifetime,20000)",
"newQuery(R:.pv_power_now)",
"getText",
"sendKey(Back)",
"sendKey(Back)",
"getFuncRetval(getText)"]});
{retval: true}
>> var info = device.sendAai({action:"getProduction"})
>> info
{retval: [{retval: '5.89 kW
Solar Power Now'}]}
To obtain the production, use info.getProduction[3].retval, SolarEdge will return W or kW depending on the amount of output, can write a simple function to get solar production:
function getProduction() {
var p = device.sendAai({action:"getProduction"});
if (p != null) {
var prodText = p.retval[3].retval;
var matches = /^(\d+(\.?\d+)?) (W|kW)/.exec(prodText);
if (matches != null) {
return matches[3] == 'kW' ? Math.round(matches[1]*1000) : matches[1];
}
}
return -1;
}
>> getProduction()
5890
If, assert Version 17
"if" and "assert" commands can use condition to determine true/false (refer to "Expressions Inside Functions" in Action section), for "if" command, if true, call "then" otherwise call "else", for "assert" true will return true, fal Several rules:
Usage:
if(<VFC>, <then>, <else>)
assert(<VFC>[, <message>])
Return:
The return value of "then" or "else". Assert will either return true or will raise AssertionError in RingoJS assert.fail().
Examples:
In Tesla app, try to find out the % of the battery remain, it can show in range (suffix in "mi") or percentage (suffix "%"), click on it to change mode. Here is the function to obtain battery remains in %:
>> device.sendAai({action:"function(getPctg)", getPctg:"if(exists(+T:/^\\d+ mi/), nsClick);getText(+T:/^\\d+%/)"})
{retval: true}
>> device.sendAai({action:"getPctg"})
{retval: '79%'}
Create a function that return the low % remaining that needs to be charge:
>> device.sendAai({action:"function(getLowPctg)", getLowPctg:"echo(20)"})
{retval: true}
Create a function to return true or false that needs to be charge or not:
>> device.sendAai({action:"function(needCharge)", needCharge:"if(getPctg() <= getLowPctg(), return(true), return(false))"})
{retval: true}
>> device.sendAai({action:"needCharge"})
{retval: false}
Alternatively, use return:
>> device.sendAai({action:"function(needCharge)", needCharge:"return(getPctg() <= getLowPctg())"})
{retval: true}
>> device.sendAai({action:"needCharge"})
{retval: false}
>> device.sendAai({action:"function(getLowPctg)", getLowPctg:"echo(100)"})
{retval: true}
>> device.sendAai({action:"needCharge"})
{retval: true}
Also, replace "return" with "echo" will achieve the same result. You can also use object symbol:
You can also use the object to store the values. For example:
device.sendAai({action:"function(getNum)", getNum:"return({min:10, max:90})"})
device.sendAai({action:"if(getPctg() < getNum().min, chargeNow())"})
device.sendAai({action:"if(getPctg() > getNum().max, stopNow())"})
Assuming calculator app, resource of .display will display the result, we can create a function for single-digit addition as follow:
>> device.sendAai({action:"function(add)", add:"click(T:%1);click(T:+);click(T:%2);click(T:=);getText(+R:.display)"})
{retval: true}
>> device.sendAai({action:"add(8,9)"})
{retval: '17'}
function calculator app against arithmetic expression in echo:
>> device.sendAai({action:"function(equal)", equal:"if(add(%1,%2) == echo(%1+%2), return(true), return(false))"})
{retval: true}
>> device.sendAai({action:"equal(8,9)"})
{retval: true}
Use "assert":
>> device.sendAai({action:"assert(calc(8,9))"})
{retval: true}
>> device.sendAai({action:"assert(add(8,9)==10)"})
[AssertionError 'Assertion error on "add(8,9)==10"']
>> device.sendAai({action:"assert(echo(true))"})
{retval: true}
>> device.sendAai({action:"assert(echo(false))"})
[AssertionError 'Assertion error on "echo(false)"']
>> device.sendAai({action:"assert(echo(false), 'This should not happen')"})
[AssertionError 'This should not happen']
refresh (M)
The accessibility nodes information is cached, if the screen has been updated, the cache information may not be accurate, use "refresh" to force to reread the nodes information.
Usage:
refresh [(<query>)]
Return:
Return count and true (always return true)
Example:
>> device.sendAai({query:"T:Overall Cpu Usage&&OX:1", actions:["getText", "sleep(2000)", "refresh", "getText"]})
{count: 4, list: [{retval: '29'},{retval: true},{retval: true},{retval: '78'}]}
repeat (M)
This command is useful for repeated actions, it shares the action input (JSON or single action) and similar return values as "forEach", it will repeat the action/actions execution in given number of runs. The maximum repeat count is 20. Use "setConfig" on "action:repeat:maxCount" to increase the maximum repeat count.
Usage:
repeat (<count>, <action name>)
repeat (<count>, <JSON name>)
Return:
Examples:
This command clicks 5 "1" in calculator:
>> device.sendAai({action:"repeat(5,click(T:1))"})
{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]}
To set the charging current of EV car:
Reduce by 1 Amp:
>> device.sendAai({action:"click(T:/[0-9]+ A/&&OX:-1)"})
{retval: true}
Reduce by 5 Amp:
>> device.sendAai({action:"repeat(5, click(T:/[0-9]+ A/&&OX:-1))"})
{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]}
Reduce and verify:
>> device.sendAai({query:"T:/[0-9]+ A/", action:"getText;repeat(5,click(OX:-1));refresh;getText"})
{count: 4, list: [{retval: '27 A'},{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]},{retval: true},{retval: '22 A'}]}
return Version 17
"return" is an important function that serves 2 purposes:
Return can only be effective inside function, outside of function, it behaves like "echo". By default, the function will return the return value of the last command in the function, "return" along with "getFuncRetval" can change that. return accepts one argument, if no argument is specified, it will return true. "return(null)" is allowed. Prior version 17, the function execution will return the return values of each commands running in the function. To bring back the same functionality, use "return(getFuncRetval())".
Usage:
return [<VFC>] // return true if no argument is specified.
Examples:
>> device.sendAai({action:"if(exists(+BP:editable), return(getText), return(null))"})
{retval: 'Hello'}
Return has no effect to main action, only inside function:
>> device.sendAai({action:"echo(1);echo(2);return;echo(3)"})
{count: 4, list: [{retval: 1},{retval: 2},{retval: true},{retval: 3}]}
Demonstrate it can stop in the middle of function execution:
>> device.sendAai({action:"function(testecho)", testecho:"echo(1);echo(2);return(getFuncRetval());echo(3);return(getFuncRetval())"})
{retval: true}
>> device.sendAai({action:"testecho"})
{retval: [{retval: 1},{retval: 2}]}
Examples:
getFuncRetval
scrollIntoView → (M)
Accept query string and scrolling direction, will try to match the query by scrolling up/down based on provided direction, when one or multiple nodes are found, it will return, if the match cannot be found, it will return null with an error. The scroll will use page up/down with the maximum of 30 pages (controlled by setConfig(navi:maxPageScroll,
Usage:
scrollIntoView(<query>) // Default down
scrollIntoView(<query>, <direction>)
Arguments:
<direction> can be one of the following, if not specify, the default is "down".
"down": Starting from current location and search by scrolling down.
"up": Starting from current location and search by scrolling up.
"top": Fast scroll to the top of page and search by scrolling down.
"bottom": Fast scroll to the bottom of page and search by scrolling up.
Return:
Will return null if the query cannot be found otherwise will return true, ML will be set to the first matched node ID.
Example:
Scroll the current application until "John" is found and click on the found node.
>> device.sendAai({actions:["scrollIntoView(T:John)", "click"]})
{count: 2, list: [{retval: true},{retval: true}]}
sendKey
sendKey accepts key code or meta state and send the key to the screen (not node), the UI element with the focus will receive the keycode and meta, examples are text field or on-screen keyboards. sendKey also offers shortcut, a text string some of the common key codes. Not all key codes will generate characters, several special key codes will bring in new window such as go back, appswitch or go to home screen. The key codes and meta states are displayed in Android KeyEvent class. "sendKey" sends to the screen, not a specific node so ML is not involved. The shortcuts mapping or keycode/meta will not work in ATS setting.
There are several shortcuts that are used as "global action keys", these keys offer different set of features, it will work on all environment.
Usage:
sendKey(<key code>)
sendKey(<key code>, <meta state>)
sendKey(<shortcut>)
Arguments:
key code and meta state: are in integer, "shortcut" is in single string. Regarding the back key, if in Sigma input method input mode, single "back" key will send 2 "back" keys to the window, first back key to dismiss the input method, second back key to dismiss the window. Use "setConfig(sigmaInput:backKeyAutoDismiss, false)" to change the behavior. The two back keys will not work on global action key.
Here are the shortcuts (some of them may not work depending on the Android version) that maps to keycode (with meta of zero), these shortcuts are mappings to the keycode. It will not work on ATS:
appswitch switch apps
back send back key
backspace send backspace key
delete send delete key
enter send enter key
home send home key
menu send menu key
search send search key
Here are the global action keys, "home" and "back" exist in both tables, if ATS is enabled, global actions will be used:
allApps Show app drawer of all the apps
back send back key
home send home key
lockScreen Lock the device screen
notifications show notifications screen
powerDialog Show the power dialog screen (power off, restart, emergency call)
quickSettings show quick setting screen
recentApps show recent Apps screen
takeScreenshot Take screenshot and store in photo gallery
Return:
return the status of the operation in true/false.
Example:
tcConst contains key code and meta state information. For example, the following 3 examples will get the same result:
sendKey('back')
sendKey(4)
sendKey(tcConst.keyCodes.BACK)
This will type "A":
>> device.sendAai({preAction:aaix("sendKey", tcConst.keyCodes.A, tcConst.keyCodes.META_SHIFT_ON)})
{retval: true}
setChecked (M)
This command applies to nodes with "checkable" nodes. Typically, toggle controls are checkable, checkboxes or radio buttons, example of class can be ".Switch" or ".Checkbox". For radio button, setChecked(false) will not work since FindNode does not know which radio button to click to disable the existing radio button.
Usage:
setChecked(true|false)
setChecked([query], true|false)
Arguments:
"setChecked(true|false)", since there is no permission to set the checked value, it will click the node to toggle the value if value needs to be changed. It accepts multiple nodes, will ignore non-checkable nodes.
Returns:
setChecked returns "changedCount", how many checkable nodes have been changed. Starting version 16, to be consistent with other commands, it will return true/false to indicate at least 1 node has been changed (not found).
setProgress (O)
To set the value of the slider bar, the min, max, type, value can be obtained by "getNodes('RI’)",
Usage:
setProgress(<number>)
setProgress(<query>,<number>)
Arguments:
<number>: Integer or decimal to set the value determined by "type", "min" and "max"
Return:
true/false: return true if value has been successful set or false if set to the same value or out of range. Return null if the node is not progress node.
See also: getProgress
Example:
Set the display brightness to 50%:
>> device.sendAai({query:"T:Brightness&&VG&&PQ:'C:.SeekBar'", action:"getProgress"})
{retval: {current: 85983232, max: 267386880, min: 0, type: 0, typeString: 'int'}}
>> var max = device.sendAai({query:"T:Brightness&&VG&&PQ:'C:.SeekBar'", action:"getProgress"}).retval.max
267386880
>> device.sendAai({query:"T:Brightness&&VG&&PQ:'C:.SeekBar'", action:aaix("setProgress", max/2)})
{retval: true}
In sound setting, set all 4 sound volumes to 50%:
>> device.sendAai({query:"C:.SeekBar"}).retval.forEach(
x => {
var max = device.sendAai({elements:[x], action:"getProgress"}).retval.max;
device.sendAai({elements:[x], action:`setProgress(${max/2})`});
}
)
setText (O)
If the node is text field, it will enter "input string" value to the input text field programmatically, no Sigma input method is involved.
For Android 11 or later, if the message contains "\n" or "\r\n" as the suffix of the input string:
Usage:
setText([query], <input string>)
setText([query], +<input string>)
Arguments:
Input string: By default the text field will be cleared before entering the string, if the first letter is "+", it will add on to the existing string. For multiple lines of text field, use "\n" to skip to next line.
Return:
true or false to indicate if the operation is successful.
Examples:
>> device.sendAai({query:"BP:editable", action:"getText"})
{retval: ['text','text','Number','Email','text password','Person Name']}
>> device.sendAai({query:"BP:editable", action:"setText(T:Number, '100');setText(T:text password, secretWord)"})
{count: 2, list: [{retval: true},{retval: true}]}
>> device.sendAai({query:"BP:editable", action:"getText"})
{retval: ['text','text','100','Email','••••••••••','Person Name']}
>> device.sendAai({query:"BP:editable", action:"getHintText"})
{retval: ['text','text','Number','Email','text password','Person Name']}
>> device.sendAai({query:"TP:textInput&&IX:1", action:"setText('Hello')"})
{retval: true}
>> device.sendAai({query:"TP:textInput&&IX:0", action: "setText('+ World')"})
{retval: true}
>> device.sendAai({query:"TP:textInput", action: "setText('Hello\n')"})
{retval: true}
Enter Emoji via String.fromCodePoint:
>> device.sendAai({query:"TP:textInput",action:aaix("setText", String.fromCodePoint(0x1f60a))})
{retval: true}
Or mix with normal message:
>> device.sendAai({query:"TP:textInput", action:aaix("setText",
"hello" + String.fromCodePoint(0x1f643, 0x1f644) + "world")})
{retval: true}
showOnScreen (O)
This will ensure the first node will appear fully on the screen, if associated UI element is partially displayed, it will scroll until the UI element is fully visible on the screen. If the node is too large to fit into the scrollable area, it will generate an error.
Usage:
showOnScreen [(<query>)]
Return:
true to indicate the screen has been scrolled, false to indicate the node is already fully visible on screen, nothing is done. Return null if the node is too large to fit into scrollable area.
sleep
This command will pause execution on the milliseconds specified, it is useful for time related wait before actions.
Usage:
sleep(<time>)
Arguments:
time: Specify time to wait in milliseconds.
Return:
return true
until
until will wait for certain condition based on the option specified, if the condition is met, it will return true, if timeout expired it will return null with error. Currently 2 types are supported:
until([query], gone, <timeout>)
"gone" will wait until the specified node (first element in ML or OQ) disappear.
Example:
The following will wait until reset button is disappeared, both formats are supported:
>> device.sendAai({query:"R:.btn_img_reset_time", action:"until(gone, 20000)"})
{retval: true}
>> device.sendAai({action:"until(R:.btn_img_reset_time, gone, 20000)"})
{retval: true}
until([query], changed, T|D, <timeout>)
Will check until timeout or text (T) or description (D) content is changed.
Example:
The following example check if Nasdaq index is changed:
>> device.sendAai({query:"T:Nasdaq*&&OY:1", actions:["getDescription", "until(changed,D,30000)","getDescription"]})
{count: 3, list: [{retval: '10,989.25'},{retval: true},{retval: '10,988.50'}]}
>> device.sendAai({actions:["until(T:Nasdaq*&&OY:1, changed, D, 30000)","getDescription"]})
{count: 2, list: [{retval: true},{retval: '10,988.50'}]}
Other Commands
These commands do not involve nodes but useful for creating scripts.
openApp/restartApp /closeApp Version 11
The specified app name can be a package name or the application display name (name show on the launcher), if multiple applications with the same name is matched, the first matching application will be used. No partial match is allowed, the match is case insensitive. "closeApp" with no argument will close the existing application. "openApp" and "restartApp" will accept optional query and timeout, after the application is launched, it will enter into "waitQuery" to wait for the query to match before return the control back to the caller. The timeout is in milliseconds.
Work has been done to ensure the openApp or restartApp will return the control when app stop in stable state (rendering is done, waiting for user input), try "restartApp" and "openApp" in terminal, if open/restart take a long time, use the associated query and timeout.
"restartApp" = "closeApp" + "openApp"
"restartApp" is useful for automation script since it will always start the app in a known state.
"openApp" will run the application, if the application is resided on the memory, it will bring the app to the front, if the app is not in the memory, it will act as start the application anew. "closeApp" will close the application (force-close).
Usage:
openApp(<name>[,<query>[,<timeout>]])
restartApp(<name>[,<query>[,<timeout>]])
closeApp(<name>)
closeApp
Return:
retval: true for success or null with lastError() when application is not found, fail to start app or query does not match.
Example:
with time measurement on (in ms)
>> device.sendAai({action:"openApp(Skype)"})
{retval: true, timeFindNode: 3040}
>> device.sendAai({action:"openApp(Skype)"})
{retval: true, timeFindNode: 672}
>> device.sendAai({action:"restartApp(Skype)"})
{retval: true, timeFindNode: 2859}
"mySolarEdge" will always take a long time to open application, use query and timeout:
>> device.sendAai({action:"openApp(mySolarEdge,T:Lifetime,15000)"})
{retval: true, timeFindNode: 13098}
openAndroidSetting
Will open the android system setting window based the setting name specified, the setting name will be converted to uppercase letter, " " will be converted to "_" and name will add "_SETTINGS" to suffix if it is not already there:
Provide part of the setting in https://developer.android.com/reference/android/provider/Settings, use "listAndroidSetting" for list of settings available in this command. This command will open the setting window. If the setting is not found, it will return null.
Usage:
openAndroidSetting(<Setting name>)
For instance: The following command open wireless setting:
device.sendAai({action:"openAndroidSetting(wireless)"})
The following open Apps setting:
>> device.sendAai({action:"openAndroidSetting(manage applications)"})
{retval: true}
"MANAGE_APPLICATIONS_SETTINGS", "manage applications setting ", "Manage Application" are the same.
listAndroidSettings
This command returns the list of settings as an array.
Usage:
listAndroidSettings
Return:
List of valid settings in array.
Example:
>> device.sendAai({action:"listAndroidSetting"})
{list: ['SETTINGS','ACCESSIBILITY_SETTINGS','ACTION_CONDITION_PROVIDER_SETTINGS', ..., 'ZEN_MODE_PRIORITY_SETTINGS']}
>> device.sendAai({action:"listAndroidSetting"}).list.length
70
setClipData, getClipData, getClipText Version 18
These two commands get or set Android clipboard, the data is called ClipData. ClipData supports 4 major data types, they are plain text, HTML, URI and Intent. getClipData will display the data and type, getClipText will return the text representation of the ClipData, setClipData will accept data as String and type. If type is not specified, plain text will be used. Most app will return ClipData as text.
Data Type | Description | Common Use Cases |
---|---|---|
Text | Plain, unformatted text. Used for copying simple strings like messages or labels. | Copying simple strings like messages or labels. |
Html | Formatted text using HTML tags. Ideal for copying rich content with formatting, like web pages or emails. | Copying rich content from web pages or emails. |
Uri | A reference to a resource (file, image, web link). Useful for copying links to content or files. | Copying references to files, images, or content. |
Intent | A messaging object to trigger actions (e.g., open a webpage or start an app). | Copying actions like opening a webpage or starting an app. |
Usage:
getClipData()
getClipText()
setClipData(<text>[, <type>])
Return:
setClipData will return true on success or null on error. getClipData will return data in retval and type or null on error.
Examples:
If pasted in a regular text field, it will usually be converted to plain text. However, in an HTML-aware text entry, such as Gmail, if you long press, you will have the option to "Paste" or "Paste as plain text". Choosing the "Paste" option will display the content as 3 separate lines in the email.
>> device.sendAai({action:"setClipData(
Line 1
Line 2
Line3, html)"})
{retval: true}
When copying 3 lines from an email text entry (like Gmail), the clipboard stores the content as HTML with inline styles. This includes not just basic HTML tags like <br>, but also extra CSS for font, color, and formatting, resulting in larger, more complex ClipData to preserve the original appearance.
>> device.sendAai({action:"getClipData"})
{retval: '<span style="color: ...>Line 3</span>', type: 'html'}
>> device.sendAai({action:"getClipText"})
{retval: '<br>Line 1<br>Line 2<br>Line3'}
URI and Text examples (do not know how to copy-n-paste Intent):
>> device.sendAai({action:"setClipData(http://www.sigma-rt.com, uri)"})
{retval: true}
>> device.sendAai({action:"getClipData()"})
{retval: 'http://www.sigma-rt.com', type: 'uri'}
>> device.sendAai({action:"getClipText"})
{retval: 'http://www.sigma-rt.com'}
>> device.sendAai({action:"setClipData(http://www.sigma-rt.com)"})
{retval: true}
>> device.sendAai({action:"getClipData()"})
{retval: 'http://www.sigma-rt.com', type: 'text'}
>> device.sendAai({action:"getClipText"})
{retval: 'http://www.sigma-rt.com'}
setConfig, getConfig, unsetConfig
Change the value of the FindNode (or Selector), various values are identified by name, use "getConfig" to obtain the value, "setConfig" to change the value and "unsetConfig" to revert back to the original value.
getConfig(<name>) → retval:<value>
setConfig(<name>, <value>) → retval: true
unsetConfig(<name>) → retval: true
If the name is not found, it will return null.
Name | Type | Default | Description |
---|---|---|---|
text:maxQueryLength | Integer | 30 | Length of text and description limit to be replaced with "*" |
navi:maxPageScroll | Integer | 30 | Maximum number of page swipes to find a node |
navi:scrollVertSteps | Integer | 40 | Number of steps on page swipes, low value is fast and incurrate, higher value is slow and accurate |
navi:slowScrollSteps | Integer | 100 | Number of swipe steps to make node fully visible |
sigmaInput:backKeyAutoDismiss | Boolean | true | Sigma input method, send extra back key to dismiss invisble keyboard |
selector:defaultTemplate | String | more | The default template if template is not specified in query |
selector:allowBqAfterEq | Boolean | false | If set to true, will allow basic query that defined after expanded query will be carried out as part of expanded query in sequence (left to right) |
selector:allowDelayedSearch | Boolean | true | Enable dalayed search, search when it is needed |
action:repeat:maxCount | Integer | 20 | Increase the number of repeat count |
action:enhancedParser | Boolean | true | Enable parser to support escape characters |
action:oldReturnKey | Boolean | false | Since version 18, all values in get commands will return as "retval", setting this to true will revert back to previous return keys. |
Example:
>> device.sendAai({action:"getCount"})
{retval: 58}
>> device.sendAai({action:"setConfig(selector:defaultTemplate, reduced)"})
{retval: true}
>> device.sendAai({action:"getCount"})
{retval: 46}
>> device.sendAai({action:"unsetConfig(selector:defaultTemplate)"})
{retval: true}
>> device.sendAai({action:"getCount"})
{retval: 58}
Command Reference
FindNode Command Reference |
---|
Action Name | OQ | Arguments | Input | Output | ML Change | Prop |
---|---|---|---|---|---|---|
Get Commands | ||||||
getAllPackageNames | No | - | None | retval:[<pname>] | - | - |
getBounds | Yes | - | All Nodes | ids:[<id>], bounds:[<bounds>] | - | - |
getBoolProp | Yes | - | First Node | retval:[<prop>] | - | - |
getBoolPropInAncestors | Yes | - | First Node | retval:[<prop>] | - | - |
getChecked | Yes | - | One Node All Nodes |
retval:<boolean> retval:[boolean|N/A] |
- | - |
getCount | Yes | - | All Nodes | retval:<count> | - | - |
getDescription | Yes | - | One Node All Nodes |
retval:<text> retval:[<text>] |
- | - |
getHintText | Yes | - | One Node All Nodes |
retval:<text> retval:[<text>] |
- | - |
getFocus | No | - | None | retval:<ID> | - | - |
getFuncRetval | No | - | None | retval:[] or retval: {} | - | - |
getIds | Yes | - | All Nodes | retval:[<ID>] | - | - |
getNodes | Yes | [fieldList(S)] | All Nodes | retval:[<info>]; count:<count> |
- | - |
getPackageName | No | - | None | retval:<PackageName> | - | - |
getProgress | Yes | - | First Node | retval:<rangeInfo> | - | - |
getQuery | No | [ignoreText(B)] [IgnoreDesc(B)] |
First Node | retval:<query String> | - | - |
getText | Yes | - | One Node All Nodes |
retval:<text> retval:[<text>] |
- | - |
getUniqQuery | No | [ignoreText(B)] [IgnoreDesc(B)] |
First Node | retval:<query String> | - | - |
Query Commands | ||||||
addQuery | No | <query> | All Nodes | retval:<count> | Yes | - |
exists | Yes | <query>[, <timeout>] | All Nodes | retval:<boolean> | - | - |
intersectX | Yes | - | First Node | retval:<count> | Yes | TX |
intersectY | Yes | - | First Node | retval:<count> | Yes | TY |
load/pop | No | - | None | retval:<boolean> | Yes | - |
newQuery | No | <query>[,<timeout>] | None | retval:<count> | Yes | - |
reduceNodes | No | - | All Nodes | retval:<count> | Yes | RN |
save/push | No | - | All Nodes | retval:<boolean> | - | - |
sort | No | <type(S)> | All Nodes | retval:<boolean> | Yes | ST |
viewGroup | Yes | [level(I)] | First Node | retval:<count> | Yes | VG |
waitQuery | No | <query>[, <timeout>] | None | retval:<boolean> | - | - |
Action Commands | ||||||
assert | No | <VFC> | None | retval:<boolean> | - | - |
click | Yes | - | First Node | retval:<boolean> | - | - |
click2 | Yes | - | First Node | retval:<boolean> | - | - |
echo | No | <VFC> | None | retval:<value> | - | - |
error | No | <error message> | None | null | - | - |
forEach | No | <action|JSON name> | All Nodes | retval:<name>:[[...]] | - | - |
function | No | <name + arg> | None | retval:<boolean> | - | - |
funcExists | No | <name> | None | retval:<boolean> | - | - |
funcDelete | No | <name> | None | retval:<boolean> | - | - |
funcList | No | - | None | retval:[<names>];count:<count> | - | - |
funcReset | No | - | None | retval:<boolean> | - | - |
if | No | <VFC>,<then>,<else> | None | retval:[<value>] | - | - |
log | No | <VFC> | None | retval:[<boolean>] | - | - |
longClick | Yes | - | First Node | retval:<boolean> | - | - |
nsClick | Yes | [<x>,<y>] | First Node | retval:<boolean> | - | - |
repeat | No | <count, action|JSON> | None | retval:<name>:[[...]] | - | - |
refresh | Yes | - | All Nodes | retval:<boolean> | - | - |
return | No | <VFC> | None | retval:<value> | - | - |
scrollIntoView | Yes | <query>[direction(S)] | None | retval:<boolean> | Yes | - |
sendKey | No | <code(I)> [meta(I)] | None | retval:<boolean> | - | - |
setChecked | Yes | <boolean> | All Nodes | retval:<boolean> | - | - |
setProgress | Yes | <value(F/I)> | First Node | retval:<boolean> | - | - |
setText | Yes | <text(S)> | First Node | retval:<boolean> | - | - |
showOnScreen | Yes | - | First Node | retval:<boolean> | - | - |
sleep | No | <duration in ms(I)> | None | retval:<boolean> | - | - |
until | Yes | <option> <arg> | None | retval:<boolean> | - | - |
Other Commands | ||||||
version | No | - | None | retval:<version number> | - | - |
openApp | No | <name>(S)> | None | retval:<boolean> | - | - |
restartApp | No | <name>(S)> | None | retval:<boolean> | - | - |
closeApp | No | [<name>(S)>] | None | retval:<boolean> | - | - |
getClipData | No | - | None | retval:<data>, type:<type> | - | - |
setClipData | No | <data>[,<type>] | None | retval:<boolean> | - | - |
openAndroidSetting | No | <setting(S)> | None | retval:<boolean> | - | - |
listAndroidSettings | No | - | None | retval:[<settings>]; count:<count> | - | - |
getConfig | No | <name(S)> | None | retval:<value> | - | - |
setConfig | No | <name(S)> <value(*)> | None | retval:<boolean> | - | - |
unsetConfig | No | <name(S)> | None | retval:<boolean> | - | - |
(S) - String, (I) - Integer, (B) - Boolean, (F) - Float, (*) - All types <VFC> - Value/Function/Comparison (see Expressions Inside Functions) [Optional Arguments] or <name>:[<return array>] |
||||||
device/devices.sendAai({query:<query>, action:<command>})
device/devices.sendAai({query:<query>, actions:[<command>]}) |
Support
Please send email to support@sigma-rt.com for support and feedbacks.