Total Control FindNode | UI Node Detection, Element Search & Android Automation

FindNode

FindNode is an innovative shell program developed by Sigma Resources & Technologies Inc. It's designed to intelligently identify UI elements (or Accessibility nodes), facilitating information extraction and interaction without the need for coordinate-based commands. This marks a significant advancement in the AAI (Accessibility and Automation Integration) project, aiming to revolutionize user interaction by replacing coordinates with intuitive queries.

Key Components of FindNode:

Query Language: A simple, yet powerful syntax designed for precise node identification.
Core Technology: FindNode integrates seamlessly with each device, powered by the Selector core.
Object Mode: Enhances synchronization across multiple devices, allowing for unified queries and actions regardless of screen resolutions.
UI Explorer: An essential tool for learning the query language, offering insights into node properties and assisting in query formulation.
AAIS: A straightforward scripting language for multi-device automation within the AAI ecosystem.
Integration: Complements existing REST and JavaScript APIs, enabling script-based automation through the "device" and "devices" constructs.

The main component of FindNode is Selector, a search algorithm that locates nodes through our query language. It operates atop the Accessibility and UI Automator frameworks to uniquely identify UI elements. This approach removes the need for coordinate-based interactions and allows scripts to adapt automatically to different device resolutions and screen sizes. Each UI element or container of UI elements is represented as one or more nodes, and each node has an ID that is unique on the current screen. Selector offers multiple ways to search for nodes, and once nodes are obtained, many automation tasks can be performed—such as retrieving text or images, clicking buttons, or entering text into fields—all without using coordinates.

For example:

devices.click("OK") instead of devices.click(100, 200).

FindNode locates the node with text "OK", obtains its coordinates, and performs the click—all at runtime.

To synchronize actions across multiple devices, when a user clicks a button on the main device, the unique query for that button (e.g. "T:OK") is sent to the other devices so they can search for the same node and click it. The intention is to provide multiple ways to locate nodes without relying on screen coordinates. There is no concept of “page up/down”; instead, commands like scrollToView are used to bring a node into view. This ensures that the same script can run on different phones with different resolutions and screen sizes.

sendAai

Obtaining Device Objects

Before calling sendAai(), you must obtain one or more device objects. These objects represent the devices you want to control and are required for all FindNode interactions.

Device objects can be obtained from the JS API Device class (JS API Documentation Section 5) or from the sigma/device module. Using the sigma/device module is recommended.

Examples:

var sigmaDevice = require('sigma/device');
var device = sigmaDevice.getDevice();
var devices = sigmaDevice.getDevices();

sigma/device module API	Class	Description
getDevice()	Device	Returns the controlling device
getDevices()	DeviceArray	Returns the controlling + selected devices
getAllDevices()	DeviceArray	Returns all connected devices
searchName(<device name>)	Device	Returns the device matching the device name
searchGroup(<group name>	DeviceArray	Returns devices with matching group name
searchTag(<tag name>)	DeviceArray	Returns devices with specified tag
searchSerial(<serial number>)	Device	Returns the device matching the given serial number
selectOne()	Device	Pops up a device list for selecting one device
selectMultiple()	DeviceArray	Pops up a device list for selecting multiple devices
selectGroup()	DeviceArray	Pops up a group list and returns devices in selected group

FindNode uses the JSON construct of Appium to carry out the JSON commands, the JSON format is:

{"cmd": "action", "action": "findnode", "params": {…}}

Every FindNode commands are carried out in "params" object.

Return value can have one of two types, successful run:

{"status":0, "value":"{…}"}

Or failure:

{"status":13,"value":"{<error message>}"}

All interactions with FindNode are done through the sendAai() API:

var output = device.sendAai({query:"...", action:"..."});
var output = devices.sendAai({query:"...", action:"..."});

device and devices refer to Device or DeviceArray objects, obtained through Device.searchObject() or require('sigma/device'). Multiple commands are allowed, the commands are separated by ";" or put commands in an array.

sendAai() will return three types of output:

Single device: Returns results in JSON format, with the main value usually stored under the "retval" key.
Multiple devices: Returns an ArrayOutput object containing the results from each device. (Refer to the JS API for details.)
Error: Returns null, and lastError() will store the error message.

The "query" field is not mandatory. If it is specified and no match is found, an error will be raised and null will be returned. The same behavior can also be achieved using "newQuery" in action.

Multiple commands are allowed. Commands can be separated by ";" or placed inside an array. Examples:

>> device.sendAai({action:"echo(2*2);echo(3*3);echo(4*4)"})
{count: 3, list: [{retval: 4},{retval: 9},{retval: 16}]}
>> device.sendAai({actions:["echo(2*2)", "echo(3*3)", "echo(4*4)"]})
{count: 3, list: [{retval: 4},{retval: 9},{retval: 16}]}

Total Control provides device.sendAai() and devices.sendAai() to interact with FindNode:

Send the commands.
Generates a timeout error if FindNode does not return within 5 seconds. Certain commands, such as sleep, openApp and scrollIntoView, will not generate a timeout.
Automatically extends the timeout for commands that require longer execution.
Handles return values (returns null on error, non-null on success, and stores the error in lastError()).

Multiple commands are allowed. Commands can be separated by ";" or placed inside an array.

Limitations

FindNode uses Android Accessibility services to obtain information:

Portrait mode only: Landscape mode is not yet supported and will be added in future updates.
Horizontal scrolling and popup windows: These are not fully supported, which may result in fewer or additional nodes being detected.
No multi-threading: Each execution must run exclusively. MDCC (object mode), terminal/scripts, UI Explorer, and REST API components that use FindNode cannot run concurrently.
WebView and custom-rendered controls: Some UI elements inside WebViews or native frameworks with custom rendering cannot be obtained. Apps such as Facebook, YouTube, and many games may therefore show missing nodes. When nodes cannot be obtained, OCR or geometry-based methods can sometimes be used as alternatives. Use UI Explorer to confirm which nodes are recognized by FindNode.

Query + Actions

Use our query language to find nodes on the screen and apply actions on the nodes found. We will cover Query and Actions in this manual.

For instance, to obtain the Android version in About Phone screen:

>> device.sendAai({query:"TP:basic&&T:Android version&&OY:1", action:"getText"})
{retval: '14'}

TP:basic template
T:Android version basic query
OY:1 expanded query

To click a button with "OK" label:

{query:"T:OK", action:"click"}

To obtain all the stock symbols:

{query:"R:.ticker", action:"getText"}

Query

Majority of the FindNode usages are about query, this is center piece of the FindNode (and part of AAI). It allows users to locate intended UI elements via different kind of useful queries on the Accessibility nodes. The entire screen is composed by many nodes, a node can be a smallest UI element or a container of smaller nodes, many are invisible. The entire screen is a tree structure starting from the single root node (some older devices have multiple root nodes). Query will be performed to obtain the nodes that meet the criteria, the intent is to reduce the large number of nodes to one or few intended nodes, users can obtain information or apply actions to the nodes.

We developed a small query language that is portable among devices and scripts and powerful enough to locate most of the nodes. The format is JSON:

{query:"<key>:<value>&&<key>:<value>&&<key>&&<key>…"}

We accept "||" and "&&" as identical separator, in the future it can be changed. Since the query is AND relationship, the query needs to meet all criterion to become "matching nodes", use of "&&" is more logical. FindNode does not offer OR relationship but the underlying Selector package provides that capability. "templates" use that construct. For certain key, value is not required.

The query string is divided into 3 sections: template, basic query and extended query. "template" (TP) will generate the initial nodes, "basic query" (BQ) will perform query from node property one at a time, "extended query" (EQ) will take output from TP and BQ and perform query across multiple nodes.

To get Main storage free size

Can use "query" + "action":

>> device.sendAai({query:"T:Main storage&&OX:1", action:"getText"})
{retval: '402 GB free'}

Can use "optional query" (OQ):

>> device.sendAai({action:"getText(T:Main storage&&OX:1)"})
{retval: '402 GB free'}

Similarly, to go back to previous screen (left arrow), can use OQ:

>> device.sendAai({action:"click(T:Storage Analysis&&OX:-1)"})
{retval: true}

To obtain the Audio size, different queries can be mixed and still obtain the identical result:

>> device.sendAai({query:"TP:all&&T:Main storage&&VG:2&&XT:Audio&&OX:1", action:"getText"})
{retval: '77 MB'}

The parser will separate the query into TP, BQ and EQ and execute them in the following order:

TP: TP:all
BQ: T:Main storage
EQ: VG:2&&XT:Audio&&OX:1

Template (TP)

A template defines the scope of a query. It acts as a producer of nodes (ML) that the query operates on.

Every query must have a template. If no template is specified, a default template (currently More) is used.

A query cannot access nodes outside the selected template. All operations—such as filtering, matching, and applying offsets—are performed only on the nodes produced by that template.

For example, when using TP:scrollable in Gmail, the query is limited to the nodes within that scrollable area (e.g., the navigation drawer). Only elements inside that area will be visible to the query.

Multiple templates are available (see the table below), each designed for different UI structures and use cases.

Template Scope Warning

Avoid using templates that return only a single node when extended operations are required.

For example, in a calculator:

TP:findText,1&&OX:1

may fail because TP:findText,1 returns only one node (1). Since OX:1 searches for a neighboring node within the template scope, no matching node is available.

Instead, use a broader template that returns multiple nodes:

TP:findText,/^.$/&&T:1&&OX:1

This template returns all single-character nodes, allowing OX:1 to locate neighboring buttons correctly.

Alternatively, use the default template (more):

T:1&&OX:1

Name	Description
all	Returns all nodes
more	Return all nodes with layout/container nodes removed. This is the default if no template is specified.
basic	Returns nodes with a child count of zero (leaf nodes).
reduced	Returns "interesting" nodes (see UI Explorer "Optimized" mode).
findText,<text>	Returns nodes containing the specified text or description. findText requires an additional parameter to specify text to search.
anyText[,min][,max]	Returns nodes containing text, optionally filtered by text length.
anyDescription[,min][,max]	Returns nodes containing description, optionally filtered by description length.
textInput[,<hint text>]	Returns editable text input nodes, sorted from top-left to bottom-right. Use "IX" for multiple text inputs. The optional parameter specifies the hint text if available.
scrollable[,<pos>\|<query>]	Returns scrollable nodes. Only one optional parameter is allowed: - If a number is provided, it specifies the scrollable position (starting from 0, top to bottom). - If a query is provided, it selects the scrollable area that matches the query. If multiple matches are found, the one with the smaller number of nodes is selected.
line,<top\|bottom>,<line number>	Returns nodes located on the specified line of a scrollable area. Use top or bottom to indicate the reference point, and <line number> to specify the line index.
appNavBar[,icons\|labels]	Returns application navigation bar items at the bottom of the screen. Specify icons or labels to return only icons or labels; if omitted, both are returned.

Basic Query (BQ)

Basic query is queries that is purely based on the individual node information, BQ takes nodes generated from template (or default template if not specified) and apply queries. BQ is not required, if BQ is not defined, it will pass all nodes from templates to EQ.

The following are the basic queries that match information obtained from the nodes itself:

Name	Description
P:<package name>	Package name (S)
C:<class name>	Class name (S)
R:<resource ID>	Resource ID (S)
D:<text>	Content description (S)
T:<text>	Text (S)
T0:<text>	Text node without Boolean property (T:<text>&&BP:none) with true
T1:<text>	Text node without any editable (T:<text>&&BP:!editable)
IT:<number>	Text input type (I)
CC:<number>	Child count (I)
ID:<hex string>	Node ID (S)
BI:[x, y]	Return nodes contain (x,y)
BI:[x1, y1, x2, y2]	Return nodes that enclosed by rectangle, x or y of -1 will be ignored
BP:<prop name>	Return nodes that match boolean properties
TD:<text>	Match text or description
HT:<text>	Match hint text (will match getText() if not supported)

Since the query is always performed on the existing app on the screen, "P" should only be used on system application such as "com.android.systemui". The "(S)" stands for String, "I" integer, "[x, y]" is a location and "[x1, y1, x2, y2]" is a rectangle (or bounds). I/S can accept both integer and String. String is more powerful.

The use of coordinates is strongly discouraged unless they are calculated dynamically at runtime.

The keys are case insensitive, we use uppercase to differentiate keys from values. (API 18) Multiple apps—including system apps (e.g. com.android.systemui)—may be visible on screen at the same time. By default, queries target the current (foreground) app. Use P:<package name> to specify a different app. You can call getAllPackageNames to list all currently active packages.

Expanded Query (EQ)

The following are the keys for expanded query, it usually works on multiple nodes return by BQ or template. The expanded query work from left to right, same key can be applied multiple time. EQ is optional. The following are the keys for EQ:

Name	Description
BQ:<basic query>	Perform basic query on multiple keys in BQ.
IX:<number>	Return one node from list of matching nodes based on position.
OX:<number>	Offset to neighbor nodes horizontally (positive – right, negative – left).
OY:<number>	Offset to neighbor nodes vertically (positive – down, negative – up).
ON:<type>	Different ways to pick one node out of list of matching nodes.
RN	Return the optimized nodes from a list of matching nodes.
ST:<sort type>	Return sorted nodes based on the position of the nodes on screen.
TX	Return nodes that intersect with reference node horizontally.
TY	Return nodes that Intersect with reference node vertically.
V:<value>	Set the value of setText, setChecked and setProgress
VG:[level number]	Return nodes in a same view group.
"X"<BQ key>:<value>	Perform basic query on specific key in BQ.

All these queries are optional, you can pick and choose any field, if you are doing the query on the screen. The "(S)" stands for String, "I" integer, "[x, y]" array. I/S can accept both integer and String. String is more powerful.

The keys are case insensitive, we use uppercase to differentiate keys from values.

Matching List (ML)

After a query is executed, the matched nodes are stored in the Matching List (ML), where each node is identified by a node ID. Many actions operate on nodes in the ML, while some actions can modify it. The ML can be saved and restored using "save" and "load", while "push" and "pop" provide additional ways to store and retrieve saved ML states. The "filter" command allows restricting the ML based on specific conditions, removing nodes that do not meet the given criteria. To retrieve the node IDs in the ML, use action:"getIds". Additionally, ML entries can be generated dynamically—TP, TX, TY, and VG can produce multiple nodes, expanding the ML as needed. An alternative way to populate the ML is by using the "elements" array or text string.

Previously, a search was always performed before executing an action, and if no "query" was provided, the default template was used to construct the ML. However, this was inefficient for actions that do not involve nodes, such as "openApp" or "function". Starting from Version 16, delayed search was introduced, meaning a search is only performed when nodes are actually required, improving efficiency.

Examples:

Previously, even query is not specified, implicitly it will apply search based on default template which is "TP:more", "getIds" will print out the node ID in ML.

>> device.sendAai({action:"getIds"})
{count: 11, retval: ['14801','1714c',...]}
>> device.sendAai({query:"TP:more", action:"getIds"})
{count: 11, retval: ['14801','1714c',...]}

Version 16's "delayed search", the search will be performed when "getIds" is called:

>> device.sendAai({action:"getIds"})
{count: 11, retval: ['14801','1714c',...]}

No search will be performed on openApp since the action does not involve ML:

>> device.sendAai({action:"openApp(Skype)"})
{retval: true}

Below example two searches will be performed, one is default template "TP:more", the other one is "TP:reduced". Version 16, one search of "TP:reduced" will be performed:

>> device.sendAai({action:"newQuery(TP:reduced)"})
{count: 8}

"elements" property

The "elements" property accepts either an array of node IDs or a string of hexadecimal values separated by ";". Each node ID must be a valid identifier present on the screen; otherwise, an error will be generated.

When "elements" is specified, all matching nodes will be stored in ML, and the "query" operation will be skipped. This feature is useful for caching node IDs for later use. A node ID can appear multiple times in the "elements" list without issue.

Examples:

device.sendAai({elements:["1234a", "5678b", "3a4a5b"], action:"getNodes"})
device.sendAai({elements:"1234a;5678b;3a4a5b"], action:"getNodes"})

This query the digit buttons and build elements array and use "forEach" to click all buttons and print the result.

>> ids = device.sendAai({query:"T:/^[1-9]$/"}).retval
a71e,aadf,aea0,b622,b9e3,bda4,c526,c8e7,cca8
>> device.sendAai({elements:ids, action:"forEach(nsClick);getText(+R:.editText_result)"})
{count: 2, list: [{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]},{retval: '789456123'}]}

To retrieve one node, can use "ID:<node ID>".

device.sendAai({query:"ID:1234a", action:"getNodes(all)"})

Testing

To find the nodes via Query is not very difficult, you need is "UI Explorer" and Terminal:

Node ID is important, it is usually unique to the application.
Terminal: Type: device.sendAai({query:"…"}) to find if it matches the node. There are many search options, there are more than one way to find nodes.
UI Explorer:
- Can build query with query helper.
- Experience different query to view the matching nodes.
- Unlimited undo/redo to compare different queries.
- Find properties inside the node.
- The "Code" in UI Explorer is a best effort to locate a node, refer to action:"getUniqQuery" for more information.
Terminal: When in doubt, use UI Explorer to find node ID, use device.sendAai({elements:["<ID>","<ID>"…], action:"getNodes"}), this will help you to obtain more information.
Start with one device using "device.sendAai" change to multiple devices via "devices.sendAai", you can search the devices by using Device.searchObject() to locate the devices you need (e.g. all devices, devices in a group, specific devices).

Template in Detail

Template (TP) is a required component of every query. It defines the initial set of nodes (ML) that the query operates on.

A TP generates a list of nodes based on its definition. This list becomes the input (ML) for subsequent Basic Queries (BQ), Extended Queries (EQ), or actions.

Rules and Behavior

One TP per query
- Only one template is allowed in a query. Using more than one TP will result in an error.
Default template
- If no TP is specified, a default template is used. The default is TP:more.
- You can change the default template using:
  setConfig(selector:defaultTemplate, <template>)
- The default template cannot be one that requires parameters (e.g., TP:findText is not allowed as a default).
Null result handling
- If a TP returns no nodes (e.g., findText finds no match, or textInput finds no input fields), an error will be generated.
Template parameters
- If a TP requires arguments, they are specified using commas:
  TP:<template>,<argument>
ML generation
- The nodes produced by TP are stored in ML (Match List) and used by all subsequent query operations.

Example: Template Limits the Query Scope

Using findText Template

TP:findText,/^[0-9]$/

This template returns only nodes containing digits (0–9).

Therefore, the query operates only on these nodes.

If you try to access "%" using offset from "3":

>> device.sendAai({query:"TP:findText,/^[0-9]$/&&T:3&&OX:1", action:"getText"})
null
>> lastError()
Node not found on offset

This results in an error because "%" is not part of the template.

Using Default Template

If no template is specified, the default template (TP:more) is used, which includes more nodes (including "%"):

>> device.sendAai({query:"T:3&&OX:1", action:"getText"})
{retval: '%'}

A query cannot access nodes outside the template.

All operations (including offset) are limited to the nodes returned by the template.

TP:all, more, basic, reduced

These templates go through all nodes on the screen:

all return all nodes including root node.
more "all" option with layout nodes removed. This is the default if TP is not defined in query.
basic "more" option with child count of zero.
reduced applies heuristic node reduction and attempts to keep only the most important or meaningful UI nodes on the screen.

UI Explorer: 3 selections:

Default TP:more
All TP:all
Optimized TP:reduced

E.g. The first getCount without arguments returns the count of the current ML. Without a query, the default template is "TP:more", so it has the same count as "TP:more". "+TP:all" modifies the current ML, which is why the third getCount returns the number of nodes in the ML built by "+TP:all".

>> device.sendAai({actions:[ 
    "getCount", 
    "getCount(+TP:all)",
    "getCount",
    "getCount(+TP:more)",
    "getCount(+TP:basic)",
    "getCount(+TP:reduced)"
]})
{count: 6, list: [{count: 237},{count: 242},{count: 242},{count: 237},{count: 76},{count: 58}]}

Notes to TP:reduced

The following three methods perform similar node reduction operations in different contexts:

TP:reduced — reduced template (query/template)
RN — reduced query filter applied to the current ML (query)
reduceNodes — action that reduces the current ML (action)

>> device.sendAai({action:"getCount"})
{retval: 59}
>> device.sendAai({action:"getCount;reduceNodes"})
{count: 2, list: [{retval: 59},{retval: 44}]}
>> device.sendAai({query:"RN", action:"getCount"})
{retval: 44}
>> device.sendAai({query:"TP:reduced", action:"getCount"})
{retval: 44}

TP:anyText[,min[,max]], anyDescription[,min[,max]]

This template will return the text in anyText or description in anyDescription with non-null content. Optionally can specify the minimum and maximum length (inclusive) of the text found. If no max is specified, the maximum length will not be enforced. The min/max of zero is not allowed.

Examples:

>> device.sendAai({query:"TP:anyText", action:"getText"})
{retval: ['Calculator','AC','÷','×','DEL','7','8','9','-' ,'4','5','6','+','1','2','3','%','0','.','±','=']}

Limit the length to one character:

>> device.sendAai({query:"TP:anyText,1,1", action:"getText"})
{retval: ['÷','×','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']}

TP:findText,<text>

Return the nodes with content in text or description that matches specified text. Text argument is required or generate an error. It will accept similar argument as "T:<…>" in "BQ".

Examples:

Top to bottom of calculator, that is why 7, 8, 9 first:

>> device.sendAai({query:"TP:findText,/[0-9]/", action:"getText"})
{retval: ['7','8','9','4','5','6','1','2','3','0']}
>> device.sendAai({query:"TP:findText,/^[A-Z]{1,4}$/", action:"getText"})
{retval: ['TGT','INTC','WMT','T','AAPL','SBUX','TSLA']}
>> device.sendAai({query:"TP:findText,*T", action:"getText"})
{retval: ['TGT','WMT','T']}

TP:textInput[,<hint text>]

This template returns the nodes which are "editable", the optional hint text if available (need Android 8.0 or later). If no nodes are found, it will generate an error. Alternatively, you can use "BP:editable" to obtain the same results.

Examples:

Obtain all text field:

>> device.sendAai({query:"TP:textInput", action:"getText"})
{count: 5, retval: ['Text','Number','Email','Text password','Text']}
// Hint Text starts support on API 26
>> device.sendAai({query:"TP:textInput", action:"getHintText"})
{count: 5, retval: ['Text','Number','Email','Text password','Text']}
// serText based on hint text
>> device.sendAai({query:"TP:textInput,Email", action:"setText(support@sigma-rt.com)"})
{retval: true}

See Also: setText (many examples there)

TP:line,<top|bottom>,<line number>

Currently, this is a best-effort feature and may not work reliably across all applications. It mainly applies to apps with a scrollable container; otherwise, TP:line may return null. FindNode attempts to locate UI elements outside of scrollable nodes. In line mode, UI elements are grouped into a series of lines, where each line may contain one or more nodes. Many applications have fixed UI elements at the top and bottom of the screen that cannot be easily located using BQ or EQ alone. For instance, "TP:line,top,1&&IX:2" returns the first line, second item, or "TP:line,bottom,-1&&T:Chats" returns the bottom line containing text "Chats".

For non-scrollable apps such as a calculator, it will always return null.
The area between the bottom of the status bar and the top of the scrollable node is considered the top region. Line numbers start from 1. Negative line numbers return lines in reverse order; -1 is the last line in the region.
The area between the bottom of the scrollable node and the top of the navigation bar is considered the bottom region. Line numbers also start from 1 or can be negative.
Using an out-of-bound line number returns null (e.g., if there are 2 lines at the bottom, TP:line,bottom,3 returns null). If the number of lines is unknown, use negative numbers.
If one node’s height is similar to two vertically stacked nodes on the same line, the lower node will be placed in the next line.
Many applications place the top line within the scrollable area; in such cases, TP:line,top,<number> may return null.
Does not interfere with "IX"; small counters or red dots next to nodes are excluded from the result list.
Line mode will be enhanced in the future to better support non-scrollable areas.

Examples:

>> device.sendAai({query:"TP:line,top,1"})
{count: 6, retval: ['1902b0','191936','192fbc','1cf0bc','19373e','194a03']}

To go back click the left arrow which is the first item "IX:0":

>> device.sendAai({query:"TP:line,top,1", action:"click(IX:0)"})
{retval: true}

>>  device.sendAai({query:"TP:line,bottom,-1"})
{count: 5, retval: ['2be5b8','2c0b42','2c1a46','2c30cc','2c3fd0']}
>> device.sendAai({query:"TP:line,bottom,-1", action:"setText(BP:editable, Hello)"})
{retval: true}

See also: TP:appNavBar.

TP:appNavBar[,<icons|labels>]

Introduced in version 20, this template will generate the bottom icons and labels of application navigation bar. By default, even number of nodes will be saved into ML, first half are icons, second half are labels. If icons or labels is specified, it will store nodes of icons or labels. For large icon across both icons and labels, the icon node will appear twice, see example. If you want to find out which has been selected, use "BP:selected", most of the applications support it.

Examples:

>> device.sendAai({query:"TP:appNavBar", action:"getIds"})
{count: 8, retval: ['141e1','7ead','15fe9','17a30','14d24','150e5','1676b','181b2']}
>> device.sendAai({query:"TP:appNavBar,labels", action:"getText"})
{count: 4, retval: ['Chats','Updates','Communities','Calls']}
>> device.sendAai({query:"TP:appNavBar,labels", action:"click(T:Calls)"})
{retval: true}
// Same as above but click the icon
>> device.sendAai({query:"TP:appNavBar", action:"click(IX:3)"})
{retval: true}
>> device.sendAai({query:"TP:appNavBar&&BP:selected"})
{count: 2, retval: ['141e1','14d24']
>> device.sendAai({query:"TP:appNavBar,labels&&BP:selected", action:"getText"})
{retval: 'Chats'}

This example, the "+" icon height covers both icons and labels, the "+" will appear on both icons and labels.

>> device.sendAai({query:"TP:appNavBar", action:"getIds"})
{count: 10, retval: ['1d9d0','1fb99','21d62','233e8','255b1','1f056','2121f','21d62','24a6e','26ff8']}
>> device.sendAai({query:"TP:appNavBar&&T:/./", action:"getText"})
{count: 4, retval: ['Home','Shorts','Subscriptions','You']}
>> device.sendAai({query:"TP:appNavBar", action:"filter(node.text);getText"})
{count: 2, list: [{retval: true},{count: 4, retval: ['Home','Shorts','Subscriptions','You']}]}
>> device.sendAai({action:"getText('+TP:appNavBar,labels&&BP:selected')"})
{retval: 'Home'}

From UI Explorer, the icons and labels cannot be separated, it will return 3 nodes each node includes icon and label.

It still can be clicked by using IX: (not T:):

>> device.sendAai({query:"TP:appNavBar", action:"click(IX:2)"});
{retval: true}

TP:scrollable[,<position>|<query>]

This template returns the on-screen items (leaf nodes) of a scrollable area. The scrollable can be identified either by position or by a query.

If a position is provided, it is a zero-based index (starting from 0), ordered from top to bottom. The default position is 0.
If a query is provided, it selects the scrollable area that matches the query. If multiple matches are found, the one with fewer nodes is selected.

Using a resource ID in the scrollable node can help achieve more accurate results.

An error will be raised if no matching scrollable node is found on the screen.

TP:scrollable returns the leaf nodes within a single scrollable area, while BP:scrollable matches all scrollable areas on the screen.

Some apps (for example, Gmail Settings) display content as a popup-style window within a scrollable area. In such cases, TP:scrollable can be used to focus on that scrollable region, helping isolate the popup content and avoid interference from nodes in the main application window. See example below.

Examples:

To obtains all scrollable nodes in the current screen:

>> device.sendAai({query:"TP:scrollable"})
{count: 5, retval: ['1e434','5bbba','5bf7b','5d9c2','6f70e']}

"BP:scrollable" returns all the scrollable nodes on the screen:

>> device.sendAai({query:"BP:scrollable", action:"getNodes(B)"})
{count: 5, list: [{bounds: '[0,268][1440,2738]', id: '1e434'},{bounds: '[42,457][1440,1303]', id: '5bbba'},{bounds: '[42,457][1440,1303]', id: '5bf7b'},{bounds: '[0,1303][1440,1742]', id: '5d9c2'},{bounds: '[0,1958][1440,2738]', id: '6f70e'}]}

Return all title of the Settings (in Samsung phones):

>> device.sendAai({actions:["openAndroidSetting(Settings)", "newQuery(TP:scrollable&&R:android:id/title)", "getText"]})
{count: 3, list: [{retval: true},{retval: 6},{count: 6, retval: ['John James','Sign in quickly and safely','Connections','Connected devices','Galaxy AI','Modes and Routines']}]}

Click the menu icon (three horizontal lines) in Gmail to open the navigation drawer.

The navigation drawer is displayed as a popup window containing multiple scrollable areas.

Using TP:scrollableAsPopup,2, the navigation drawer popup can be identified. In this case, the screen contains multiple scrollable areas, and "2" specifies the navigation drawer popup.

Can search based on position or query:

>> device.sendAai({query:"TP:scrollable,2", action:"getIds"})
{count: 39, retval: ['639b7','64139','96fd2','97393','97ed6','9d8ee','9dcaf','6d761','72db8','73179','9fab7','75342','75703','75ac4','7750b','7804e','7840f','787d0','7a5d8','7a999','7c7a1','7d2e4','7d6a5','7da66','7f86e','7fc2f','81df8','821b9','8257a','84382','84743','84b04','8690c','86ccd','88e96','89257','8b420','8b7e1','8bba2']}
>> device.sendAai({query:"TP:scrollable,T:Sent", action:"getIds"})
{count: 39, retval: ['639b7','64139','96fd2','97393','97ed6','9d8ee','9dcaf','6d761','72db8','73179','9fab7','75342','75703','75ac4','7750b','7804e','7840f','787d0','7a5d8','7a999','7c7a1','7d2e4','7d6a5','7da66','7f86e','7fc2f','81df8','821b9','8257a','84382','84743','84b04','8690c','86ccd','88e96','89257','8b420','8b7e1','8bba2']}

Search is now possible by limiting the query to the navigation drawer popup:

>> device.sendAai({query:"TP:scrollable,T:Sent&&T:Sent&&OX:1", action:"getText"})
{retval: '505'}

Without this template, the query is not restricted to the popup window, so the search may become inaccurate or fail:

>> device.sendAai({query:"T:Sent&&OX:1", action:"getText"})
null
>> lastError()
Query Search Error: Node not found on offset

Basic Query in Detail

Basic query takes ML from TP and perform node-level matching, one node at a time and create a shorter ML, discard nodes that do not match. BQ extracts data from Accessibility, it contains single-node information and actions. Extend more functionalities in the query for more powerful matching (special characters) and added shortcuts to make query more concise.

Special Characters in Query

For number such as inputType or childCount, it can either accept number or string:

<number>, "<number>" or "=<number>": Match exacting the number.
"> <number>": Match number greater than <number>.
"< <number>": Match number less than <number>.

String can be one of the following, "\n" to match newline character:

"<string>": Exact match on full text, case sensitive.
"(?i)<string>": Match case insensitive match in regular expression.
"ci:<string>": Match case insensitive in string.
"*<string>": Match string at the end, e.g. "*Group".
"<string>*": Match string at the beginning. e.g. "android.widget.*".
"*<string>*": Match any substring "*widget*".
"/<regex>/: Match any regular expression. E.g. "/ImageView|TextView/"

"!" is use at the beginning of string as NOT. "!0" = not zero. "C:!/Layout/", non-layout classes.

For simplicity in examples, the standard prefix of {"cmd": "action","action": "findnode", "params":{…}} will be omitted, we will list the examples in the "param" object and use JavaScript object notations to reduce number of double of quotes.

Shortcuts

Two shortcuts are available for class name (C) and resource ID (R), prefix with "." for substitutions of commonly use prefixes:

Class name (C): Prefix "." as substitution of "android.widget.". E.g. ".Button" = "android.widget.Button".
Resource ID (R): Prefix "." as substitution of "<package name>:id/". E.g. ".text_connect" = "com.sigma_rt.totalcontrol:id/text_connect".

The "." notation can only be used for normal matching and "!" matching, other types of matching (e.g. regular expression and wildcard) needs to use full form.

P:<package name>

On screen, it may have more than one application running, by default, omitting “P” in query, the active app (app takes the biggest screen area) will be used. You may want to obtain information from another app such as systemui. UI Explorer can choose the application or use ”getAllPackageNames” command to find out all running package names on the screen.

Examples:

To obtain the current battery % from system bar, one of the following methods, since OQ does not support “P”, use “newQuery” command:

>> device.sendAai({query:"P:com.android.systemui&&R:.clock", action:"getText"})
{retval:'3:22'}
>> device.sendAai({query:"P:com.android.systemui", action:"getText(R:.clock)"})
{retval:'3:22'}
>> device.sendAai({action:"newQuery(P:com.android.systemui); getText(R:.clock)"})
{count:2,list:[{retval:29},{retval:'3:22'}]}

To click home button in navigation bar:

>> device.sendAai({query:"P:com.android.systemui", action:"click(R:.home)"})
{retval: true}

C:<class name>, R:<resource ID>

All nodes contain package name and class name and optional resource ID. Since the matching is performed on the screen that running the application, "P" will default to the app running, other "P" does not make sense. So "P" is not required.

T:<text>, D:<text>, TD:<text>, T0:<text>, T1:<text> (T0 & T1 in API 18)

Two most important keys, the node with information usually carry text or description, this is much more accurate and faster than OCR. "until" command can monitor if the text or description has been changed. T:<text> with special characters are often used. "TD" match text or description, match is successful if any one of them match. “T0” is “T&&BP:none”, “T1” is “T&&BP:!editable”. T1 is used to prevent finding text in the text field. T0 is used to find the label that has no actions (ancestor nodes may have actions).

Examples:

device.sendAai({query:"T:OK", action:"click"})

Click OK button.

device.sendAai({query:"T1:Name&&OX:1", action:"getText"})

This will not find text field with the “Name” on it.

Bound In (BI):

Bound in takes screen coordinate or bounds and return the nodes based on the criteria below. This mainly for screen manipulation such as UI Explorer or debugging purposes. FindNode's object mode attempt to find nodes in devices independent of resolutions and sizes, "BI" coordinate-based will not work on all devices.

[x, y]: Will return the nodes with bounds contain (x, y).
[x, -1] or [-1,y] where one of them is -1: if x = -1, will ignore x and return nodes with bounds that contains y, similarly for y = -1, it will only match x. As if drawing a horizontal/vertical line on the y and x location, the nodes that touch the line will be returned.
[x1, y1, x2, y2]: Will match the nodes bounds within the rectangle specified by (x1, y1) - (x2, y2).

Example:

>> device.sendAai({query:"BI:[-1,645]&&IX:-1", action:"click"})
{retval:true}
>>  device.sendAai({query:"BI:[0, 0, 1000, 1000]", action:"getBounds"})
{bounds:[[0,113,196,309],[252,164,529,257],[70,572,702,695],[702,572,737,576],[70,765,480,888],[480,765,515,769],[515,765,925,888],[925,765,960,769]],count:8,ids:["1b21e","1ae5d","17d90","18151","18c94","19055","19416","197d7"]}

CC:<child count>, IT:<input type>

These are property found in the node, seldom use. CC is number of children, input type usually a number text field.

BP:<property>

Boolean property is important in the node, these properties carry true or false value. Multiple values are allowed, "!" in front of the property name will match false. Here are the properties:

checkable: if node contains checkbox or on/off switch, usually this comes with "clickable".
checked: if node contains checkbox/switch, true if it is checked/on or false if unchecked/off.
clickable: if node is clickable (usually a button or UI element response to click such as checkbox).
editable: if node allows text to be entered or modified.
enabled: if node is enabled, disable node does not response to actions. Usually display in gray color.
focusable: if node can be focused.
focused: if the node is focused, usually one focus per screen, pick this will return null or 1 node.
longClickable: If node is long clickable.
progressible: If the node is "progress" node such as slider.
selected: if the node has been selected.
scrollable: if node is scrollable node (container), the node can be scrolled, horizontally or vertically. Classes such as RecyclerView, ScrollView and ListView are scrollable.
none: none of the Boolean property returns true. (API 18)
!none: any of the Boolean property returns true. (API 18)

Multiple boolean properties are allowed, separated by comma, it is optional to include within quotes or not. “none” and “!none” cannot be combined with other properties or it will generate error.

Usage:

{query:"BP:<prop name>, !<prop name>, ..."}

Examples:

>> device.sendAai({query:"TP:textInput&&BP:clickable"})
{count: 1, retval: ['27ebc0']}
>> device.sendAai({query:"TP:textInput&&BP:'clickable,enabled'"})
{count: 1, retval: ['27ebc0']}
>> device.sendAai({query:"TP:textInput&&BP:clickable,enabled"})
{count: 1, retval: ['27ebc0']}

Use "getNodes(BP)" to output all Boolean properties:

>> device.sendAai({query:"TP:textInput&&BP:clickable,enabled", action:"getNodes(BP)"})
{count: 1, list: [{booleanProperties: {checkable: false, checked: false, clickable: true, editable: true, enabled: true, focusable: true, focused: false, longClickable: true, multiLine: true, scrollable: false, selected: false, visibleToUser: true}, id: '27ebc0'}]}

Starting API 14, "not" (!) is supported, you can not only find the true value but also false value by using "!".

Assuming loop thru this to obtain values until no further (arrow shows as gray):

>> device.sendAai({query:"R:.btn_next_period&&BP:!enabled", action:"getCount"})
{count: 1}

Reverse is not true:

>> device.sendAai({query:"R:.btn_next_period&&BP:enabled", action:"getCount"})
null
>> lastError()

No match found

>> device.sendAai({query:"T:Day&&TX", action:"getText"})
{count: 5, retval: ['Day','Week','Month','Year','Billing']}

"checkable" is usually "clickable", to remove checkable from the list:

>> device.sendAai({query:"BP:clickable", action:"getCount"})
{count: 26}
>> device.sendAai({query:"BP:checkable", action:"getCount"})
{count: 10}
>> device.sendAai({query:"BP:clickable,!checkable", action:"getCount"})
{count: 16}

Basic Query Examples

device.sendAai({}) → device.sendAai({query:"TP:more", action:"getIds"})

"getIds" will print out the ID of the ML.

>> device.sendAai({})
{count: 9, retval: ['95b0','b779','bb3a','9d32','a0f3','a4b4','a875','ac36','aff7']}
>> device.sendAai({query:"TP:more", action:"getIds"})
{count: 9, retval: ['95b0','b779','bb3a','9d32','a0f3','a4b4','a875','ac36','aff7']}
>> device.sendAai({query:"T:/./", action:"getCount"})
{count: 25}
>> device.sendAai({query:"D:/./", action:"getCount"})
{count: 44}
>> device.sendAai({query:"TD:/./", action:"getCount"})
{count: 69}

Can include more complex query like following on calculator:

>> device.sendAai({query:"T:/^.$/&&C:.Button&&R:.button_*", action:"getText"})
{retval: ['÷','×','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']}

If the match cannot be found, it will return null, use "lastError()" to display error message:

>> device.sendAai({query:"T:X", action:"getText"})
null
>> lastError()
No match found
>> device.sendAai({query:"X:X", action:"getText"})
null
>> lastError()
Error:Cannot find the key for "X"

Expanded Query in detail

There are the queries left that is not BQ or TP. Keys in EQ are often calculations and heuristics that involve multiple nodes. EQ performs the keyed functions from left to right, manipulate along the way, hopefully at the end, the ML contains nodes that users want. The ML is uninteresting without actions, resulting ML will be passed to actions to apply actions or extract information. Keys can be applied multiple times "OX:1&&OX:1" is allowed.

Index (IX) M → O

IX:<number>. Return a node from ML nodes based on the position starting from zero (IX:0). IX can be negative value in which case the position is in reversed order (e.g. -1 is the last node).

Examples:

>> device.sendAai({})
{count: 9, retval: ['95b0','b779','bb3a','9d32','a0f3','a4b4','a875','ac36','aff7']}
>> device.sendAai({query:"IX:1"})
{count: 1, retval: ['b779']}
>> device.sendAai({query:"IX:-2"})
{count: 1, retval: ['ac36']}

Single Node Selection (ON) M → O

ON:<option> Return one node from multiple nodes in ML, currently 3 options are available. Only limited options at this time, will expand more options in the future.

These are the options:

first: Take the first node from the ML (Similar to IX:0).
last: Take the last node from the ML (Similar to IX:-1).
min: Pick the smallest node (in area) from the ML.

Offset (OX & OY) O → O

OX:<integer> and OY:<integer>. Use the offset to find neighboring node base on one reference node, OX find horizontal position, positive integer moves right, negative integer moves left. For OY, positive integer moves down, negative integer moves up.

Examples:

In About phones:

>> device.sendAai({query:"T:Model name&&OX:1", action:"getText"})
{retval: 'Galaxy S22 Ultra'}
>> device.sendAai({query:"T:Model number&&OX:1", action:"getText"})
{retval: 'SM-S908U1'}

>> device.sendAai({query:"T:Android version&&OY:1", action:"getText"})
{retval: '12'}

Another example:

>> device.sendAai({query:"T:5&&OX:1", action:"getText"})
{retval: '6'}
>> device.sendAai({query:"T:5&&OY:-1", action:"getText"})
{retval: '8'}

// OX & OY orders may get different results
>> device.sendAai({query:"T:0&&OY:-1&&OX:2", action:"getText"})
{retval: '×'}
>> device.sendAai({query:"T:0&&OX:2&&OY:-1", action:"getText"})
{retval: '−'}

ViewGroup (VG) O → M

ViewGroup: VG or VG:<level>. This command accepts optional level, if level is not specified, 1 will be used as level. It accepts a node (from query), will do parent traversal until a "group" is found, level is used to specify when to stop. A "group" is layout that occupy 85% of the screen width or "ViewGroup" in the class. ML will contain the nodes within the group, with the first node is ViewGroup/Layout node itself that contains all the nodes. It is very useful to logically group interesting nodes and use "addQuery" or BQ to search the nodes within the group. "setConfig(selector:viewGroupWidthRatio, <percentage>)" to modify 85%.

Related action: "viewGroup".

Examples:

Select the right current before charging, first level of VG does not reach the intended nodes:

>> device.sendAai({query:"T:Start Charging&&VG"})
{count: 2, retval: ['53b16','53ed7']}

To obtain the charging group, use level 2. The first node will be the node that encloses all the nodes in the group (green rectangle). First node is useful to "showOnScreen" to ensure the entire group is visible on the screen.

>> device.sendAai({query:"T:Start Charging&&VG:2"})
{count: 24, retval: ['4a8af','4ac70','4b031','4b3f2','4bb74','4bf35','4c2f6','4c6b7','4ca78','4ce39','4d1fa','4d5bb','4d97c','4dd3d','4e0fe','4ec41','4f002','4f3c3','53755','53b16','53ed7','54298','54659','54a1a']}

To obtain the current Ampere:

>> device.sendAai({query:"T:Start Charging&&VG:2&&XT:/^[0-9]+ A$/", action:"getText"})
{retval: '32 A'}

To obtain and change, reduce by 1 Amp:

>> device.sendAai({query:"T:Start Charging&&VG:2&&XT:/^[0-9]+ A$/", actions:"getText;click(OX:-1);refresh;getText"})
{count: 4, list: [{retval: '32 A'},{retval: true},{retval: true},{retval: '31 A'}]}

Use "repeat" to reduce by 5 Amp:

>> device.sendAai({query:"T:Start Charging&&VG:2&&XT:/^[0-9]+ A$/", actions:"getText;repeat(5,nsClick(OX:-1));refresh;getText"})
{count: 4, list: [{retval: '32 A'},{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]},{retval: true},{retval: '27 A'}]}

Intersect (TX & TY) O → M

Intersect: TX and TY. These commands take a node as a reference node and change the ML to contain all the nodes that intersect horizontally (intersectX) or vertically (intersectY). The matcher will make use of the bounds (top/bottom for intersectX, left/right for intersectY) of the reference node to search for intersected nodes.

The size (width or height) of the reference node is very important, if a wide reference node is picked, TY will match most of the nodes on screen. Pick reference node wisely.

This command will take first node and generate multiple nodes in ML, you can use "addQuery" to set more constraints on them.

See also: "intersectX" and "intersectY".

Examples:

On the offset calculator example above:

>> device.sendAai({query:"T:2&&TX"})
{count: 6, retval: ['bb4b','bf0c','c2cd','e0d5','e496','111a2']}
>> device.sendAai({query:"T:2&&TY"})
{count: 6, retval: ['1d115','1d4d6','a886','b3c9','bf0c','ca4f']}

But if you pick result as reference node with wide scroll entire screen, OX will get one node (itself) and OY will get all nodes.

>> device.sendAai({query:"R:.result&&TX"})
{count: 1, retval: ['1d4d6']}
>> device.sendAai({query:"R:.result&&TY"})
{count: 23, retval: ['1d897','1d4d6','d592','f39a','a4c5','a886','ac47','dd14','1029e','b008','b3c9','b78a','e0d5','bb4b','bf0c','c2cd','111a2','e496','c68e','ca4f','ce10','120a6','e857']}

Reduce nodes (RN) M → M

RN (Reduce Nodes) is one of the important features of FindNode. It applies heuristic node reduction to the current ML to reduce the number of visible nodes and keep only the most meaningful UI elements.

Many extended operations such as TL, BL, OX/OY, and intersect internally use node reduction.

The reduction algorithm generally prefers smaller actionable or meaningful nodes over larger container nodes. For example, if a larger .Button node fully contains a smaller .TextView node, the .TextView node will usually be selected instead of the .Button. Actions such as click will still work correctly on the selected node.

See also UI Explorer Optimize mode.Reduce Nodes: RN.

Examples:

>> device.sendAai({}).count
242
>> device.sendAai({query:"RN"}).count
55

See Also: TP:reduced, reduceNodes.

Sort (ST) M → M

Sort: ST:X|Y|YX. Sort based on the bounds of the nodes, usually the way three is structure and the search, quite likely it has been ordered from top to bottom. Unless the order is important, there is little reason to sort.

X – compare the left bounds of the nodes

Y – compare the top bounds of the nodes

YX – compare top bounds and then left bounds of the nodes

Examples:

>> device.sendAai({query:"C:.Button", action:"getText"})
{retval: ['AC','÷','×','DEL','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']}
>> device.sendAai({query:"C:.Button&&ST:Y", action:"getText"})
{retval: ['AC','÷','×','DEL','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']}
>> device.sendAai({query:"C:.Button&&ST:YX", action:"getText"})
{retval: ['AC','÷','×','DEL','7','8','9','-','4','5','6','+','1','2','3','%','0','.','±','=']}
>> device.sendAai({query:"C:.Button&&ST:X", action:"getText"})
{retval: ['AC','7','4','1','0','÷','8','5','2','.','×','9','6','3','±','DEL','-','+','%','=']}

Related action: sort

Basic query(BQ) M → M

Basic query: BQ:'<basic query>' or "<basic query>". BQ is the command in EQ. Consider the following query (Obtained from UI Explorer Custom Query):

Examples:

Query: T:Overall Cpu Usage&&VG

"VG" is used to obtain the group, if we want to obtain the number of "Cpu Cores", the following query will not work:

T:Overall Cpu Usage&&VG&&T:Cpu Cores&&OX:1 → T:Cpu Cores&&OX:1&&VG

Two "T" commands are basic commands, second T will overwrite first T, since BQ commands applied before EQ. If we need the functionalities of basic query in EQ so the text can be matched. There 3 ways to do it:

Use BQ:'…' to include BQ keys. BQ:'T:Cpu Cores'.
Added a "X" prefix in basic query. XT:Cpu Cores.
Configure this flag "selector:allowBqAfterEq" to true, any BQ command after EQ will be done as part of EQ in sequence.

Both will work:

T:Overall Cpu Usage&&VG&&XT:Cpu Cores&&OX:1
T:Overall Cpu Usage&&VG&&BQ:'T:Cpu Cores'&&OX:1

>> device.sendAai({query:"T:Overall Cpu Usage&&VG&&XT:Cpu Cores&&OX:1", action:"getText"})
{retval: '8'}
>> device.sendAai({query:"T:Overall Cpu Usage&&VG&&BQ:'T:Cpu Cores'&&OX:1", action:"getText"})
{retval: '8'}

Starting version 14, there is a flag to allow any BQ after EQ command, in this example, "T:Cpu Cores" is considered as part of EQ

>> device.sendAai({action:"setConfig(selector:allowBqAfterEq, true)"})
{retval: true}
>> device.sendAai({query:"T:Overall Cpu Usage&&VG&&T:Cpu Cores&&OX:1", action:"getText"})
{retval: '8'}

Value (V) M → M

This should not be part of the query since it performs "actions" to set the values of the nodes, however it is convenient to construct a query with a value associated. In fact, this key will result the "set" actions to be performed to set the values. This only applies to the first node in the ML, it can be set either in "query" or "newQuery" action, for query, nothing will be returned if set correctly, wrong data type will result in error. 3 set types are supported:

Node Prop	Action	Type	Examples
Checkable	setChecked	boolean	V:true, V:false
progressible	setProgress	float/integer	V:95.37, V:100
Editable	setText	string	V:I am John

Examples:

To add new entry in Calendar:

By default, the action is "getIds":

>> device.sendAai({query:"T:Title&&V:Online Gaming Tournament&&OY:1&&OX:1&&V:true"})
{count: 1, retval: ['e72e2']}
>> device.sendAai({action:"newQuery(T:Title&&V:Online Gaming Tournament&&OY:1&&OX:1&&V:true);getIds"})
{count: 2, list: [{count: 1},{count: 1, retval: ['1a2e06']}]}

>> device.sendAai({query:"T:Title&&V:Online Gaming Tournament&&OY:1&&OX:1&&V:111"})
null
>> lastError()
Wrong format, expect boolean value

Actions

FindNode offers many action commands, and additional actions can be created using the "function" command. The "action" parameter accepts either a single string command or an array of strings containing multiple commands.

The "query" parameter is optional. If not specified, the delayed query mechanism will be used, meaning nodes will only be queried when the actions require them. If a query is explicitly specified, its output will be stored in ML (Matching List). Commands or actions that require nodes will retrieve them from ML, either through a query or an optional query. To inspect ML content, use action:"getIds". If no action is specified, action:"getIds" is used by default. The "action" and "actions" parameters function identically.

Use device.sendAai() or devices.sendAai() to communicate with FindNode. When sending requests to multiple devices, a separate thread is created for each device to execute the query and actions.

If sendAai encounters an error from FindNode or times out, it will return null, and lastError() will contain the error message:

>> device.sendAai({query:"T:Not there", action:"getText"})
null
>> lastError()
No match found

There are two types of actions:

Actions that operate on nodes (ML).
Actions that are independent of nodes.

For actions that require a single node (e.g., "click"), the first node in ML will be selected. The ON, IX, OX, and OY parameters can modify this selection.

FindNode actions are continuously improved in new versions, which may introduce incompatibilities. To ensure the required commands are supported, use action:"version".

Custom actions can be created using the "function" command. For more details and examples, refer to the "function" documentation.

Single Action

A single action command means that only one command is specified in the "action" parameter. In this case, the result will be returned in "retval", or null if an error occurs.

device/devices.sendAai({query:"…", action:<action>})

Error Handling

If the query does not find the expected node, the command returns null, and lastError() will provide the reason, for example:

>> device.sendAai({action:"getCurrentPackageNameX"});
null
>> lastError()
Invalid action: getCurrentPackageNameX

Multiple Actions

FindNode allows multiple actions to be executed sequentially. Many FindNode operations involve multiple actions, which are carried out from left to right, one at a time. The result is an array that stores the output of each action.

If any action fails, sendAai will return null, and lastError() will indicate the zero-based index of the failed action along with the error message.

There are two ways to define multiple actions:

1. Using an Array:

device/devices.sendAai({query:"…", actions:[<action 1>, <action 2>, …]})

2. Using a Semicolon-Separated String:

device/devices.sendAai({query:"…", actions:"<action 1>;<action 2>, …"})

Return Format for Multiple Actions

For multiple actions, the return value will be an array containing the output of each action, for instance:

>> device.sendAai({action:"version;getCurrentPackageName;getCount"})
{count: 3, list: [{retval: 19},{retval: 'com.solaredge.homeowner'},{retval: 59}]}

Handling Errors in Multiple Actions

If an error occurs, lastError() will indicate the 0-based index of the failed command along with the error message, for example:

>> device.sendAai({action:"version;getCurrentPackageNameX;getCount"});
null
>> lastError()
[1 - getCurrentPackageNameX] Invalid action: getCurrentPackageNameX

Data Type

FindNode supports the following data types in its arguments. In Version 17, support for function, array, and object data types was added. The data types follow JavaScript conventions and perform type coercion when necessary.

Supported Data Types

String – Can be enclosed in single quotes, double quotes, or left unquoted if it contains only non-special characters.

Integer – Any whole number.

Double – Any number with a decimal point.

Boolean –true or false.

Function – A function name with arguments inside parentheses (added in Version 17).

Array – Elements separated by commas inside brackets [] (added in Version 17). Arrays in "retval" also include a "count" property.

Object – Key-value pairs enclosed in curly braces {} (added in Version 17).

null – A special value, useful for comparisons.

Examples:

>> device.sendAai({action:"echo(this is a string)"})
{retval: 'this is a string'}
>> device.sendAai({action:"echo('this is a string')"})
{retval: 'this is a string'}
>> device.sendAai({action:'echo("this is a string")'})
{retval: 'this is a string'}
>> device.sendAai({action:'echo(1234567)'})
{retval: 1234567}
>> device.sendAai({action:'echo(1234567.89)'})
{retval: 1234567.89}
>> device.sendAai({action:"echo(true)"})
{retval: true}
>> device.sendAai({action:"echo(false)"})
{retval: false}
>> device.sendAai({action:"echo([101,102,103,104,105])"})
{count: 5, retval: [101,102,103,104,105]}
>> device.sendAai({action:"echo({a:1, b:2, c:[1,2,3,4,5]})"})
{retval: {a: 1, b: 2, c: [1,2,3,4,5]}}
>> device.sendAai({action:"echo(null)"})
{retval:null}
>> device.sendAai({action:"echo({a:100, b:[1,2,3,4]})"})
{retval: {a: 100, b: [1,2,3,4]}}
>> device.sendAai({action:"echo({a:100, b:[1,2,3,4]}.b[2])"})
{retval: 3}
>> device.sendAai({action:"echo([1,2,{x:100, y:200, z:300}, 3])"})
{count: 4, retval: [1,2,{x: 100, y: 200, z: 300},3]}
>> device.sendAai({action:"echo([1,2,{x:100, y:200, z:300}, 3][2])"})
{retval: {x: 100, y: 200, z: 300}}
>> device.sendAai({action:"echo([1,2,{x:100, y:200, z:300}, 3][2].y)"})
{retval: 200}
>> device.sendAai({action:"echo([1,2,3,[4,5,6, [7,8,9]]][3][3][1])"})
{retval: 8}

node.X

The node.X notation is used to retrieve properties of the first nodes in ML. In the filter command, node.X is evaluated one node at a time, allowing the filtering process to iterate through ML and retain only the nodes that meet the specified conditions. For other commands such as "if", "assert", and "echo", node.X retrieves only the first node in ML.

General Properties

Property	Description
class	Class name of the node
resourceId	Resource ID
text	Text content of the node
description	Content description
bounds	Returns 'Rect(left, top - right, bottom)'
bounds.left	Left coordinate
bounds.top	Top coordinate
bounds.right	Right coordinate
bounds.bottom	Bottom coordinate

Boolean Properties (Prefixed with "is")

Property	Description
isCheckable	Whether the node is checkable
isChecked	Whether the node is checked
isClickable	Whether the node is clickable
isEditable	Whether the node is editable
isEnabled	Whether the node is enabled
isFocusable	Whether the node is focusable
isFocused	Whether the node is focused
isLongClickable	Whether the node is long-clickable
isMultiLine	Whether the node supports multiple lines of text
isSelected	Whether the node is selected
isScrollable	Whether the node is scrollable
isVisibleToUser	Whether the node is visible to the user

Text-Related Boolean Properties

Property	Description
isNumber	Returns true if node.text is a valid number
isInteger	Returns true if node.text is a valid integer

The following will remove ML that is not ".Button" class name.

>> device.sendAai({action:"getCount;filter(node.class == '.Button');getCount"})
{count: 3, list: [{retval: 26},{retval: true},{retval: 20}]}

For this app (calculator), only button is clickable, so checking isClickable will work as well:

>> device.sendAai({action:"getCount;filter(node.isClickable);getCount"})
{count: 3, list: [{retval: 26},{retval: true},{retval: 20}]}
>> device.sendAai({action:"echo(node.class);echo(node.bounds)"})
{count: 2, list: [{retval: 'android.view.ViewGroup'},{retval: 'Rect(0, 113 - 1440, 2952)'}]}

Variables in Actions and Query

If arguments of an action are determined at runtime, there are 3 ways to do variable substitutions:

JavaScript String manipulation functions. Apply to action and query.
JavaScript template literals: Apply to action and query.
aaix helper function: Simple helper function to reconstruct string with variables.
buildAction: New and improved aaix to reliably send long or multi-line data and support raw format.

If you want to use variable in the action commands, can use "+" to concatenate multiple items, with quotes, it can become cumbersome:

JavaScript string functions:


var input = "Hello"
>> "setText('" + input + "')"
setText('Hello')
>> 'setText("' + input + '")'
setText("Hello")
>> 'setText(' + input + ')'
setText(Hello)

>> device.sendAai({query:"TP:textInput", action:["setText('" + input + "')", "click(OX:2)"]});
{count: 2, list: [{retval: true},{retval: true}]}

JavaScript template literals

Starting version 14, variable substitution (or expression) can be applied to string with backtick with ease, since it is basic construct of JavaScript, it can apply to both query and actions. With the introduction of multiple actions in string (separated by ";"), the substitution is much simpler, it also allow multiple actions across multiple lines.

Below is an example, it is good idea to surround ${input} with quotes in case input contains space, with the

>> device.sendAai({query:"TP:textInput", action:`setText("${input}");click(OX:2)`})
{count: 2, list: [{retval: true},{retval: true}]}

To retrieve all numbers in calculator:

>> var text = "/^[0-9]$/"
>> device.sendAai({query:`T:${text}`, action:"getText"})
{count: 10, retval: ['7','8','9','4','5','6','1','2','3','0']}

Calling function to obtain class and description:

>> var c=()=>".CheckBox"
>> var x="T,D"
T,D
>> var d=()=>"Sunday"
>> device.sendAai({query:`C:${c()}&&D:${d()}`,action:`getNodes("${x}")`})
{count: 1, list: [{description: 'Sunday', id: '509a7', text: 'S'}]}

>> var disp=()=>"getNodes('${x}')"
>> disp()
getNodes('T,D')
>> device.sendAai({query:`C:${c()}&&D:${d()}`,action:`${disp()}`})
{count: 1, list: [{description: 'Sunday', id: '509a7', text: 'S'}]}

Multiple actions (create a function called “enterText”):

>> device.sendAai({action:"function(enterText(Hello))", 
       enterText:`setText(BP:editable, %1);
                  if(getText(BP:editable) != %1, error(Not match));
                  click(BP:editable&&OX:2);
                  sendKey(Back)
`})

aaix:

A simple helper function on pre-version 14 to make actions easier to write, with the introduction of JavaScript template literals, there is no reason to use "aaix".

>> aaix("setText", input)
setText("Hello")
>> aaix("sendKey", tcConst.keyCodes.KEYCODE_HOME, 0)
sendKey(3, 0)

>> device.sendAai({query:"TP:textInput", action:[aaix("setText", input), "click(OX:2)"]})
{count: 2, list: [{retval: true},{retval: true}]}
>> device.sendAai({action:aaix("sendKey", tcConst.keyCodes.KEYCODE_HOME, 0)})
{retval: true}

If the argument is string, below 3 examples are identical:

action:["intersectX(OY:1, IX:2)", "getText"]
action:["intersectX('OY:1', 'IX:2')", "getText"]
action:[aaix("intersectX", "OY:1", "IX:2"), "getText"]

Most of the actions have optional query string in first argument, for instance:

{query:"T:John", action:"click"}
{actions:["newQuery(T:John)", "click"]}	→ {action:"click(+T:John)"}
{actions:["addQuery(T:Mary)", "click"]}	→ {action:"click(T:Mary)"}

The added query string will make command more concise. "newQuery" prefixed by "+" will clear ML, make the search and reconstruct the ML. "addQuery" will search based on the existing ML.

buildAction:

Builds a command string like fn(arg1, arg2, ...) with safe argument serialization.

Signature:buildAction(fn, ...args) → string

Rules:

Numbers/booleans → passed as literals (3, true).

Strings/other → safely quoted via JSON.stringify (handles quotes, \, newlines, Unicode) and escapes single quotes.

raw(x) → insert verbatim (unquoted) for tokens like null, identifiers, or pre-stringified JSON.

Use case: reliably send long or multi-line data (e.g., HTML) without breaking on quotes/newlines.

>> var action = buildAction('setClipData', 'htmlSend', 'Line 1\nLine 2\nLine 3', 'Line 1
Line 2
Line 3')
>> action
setClipData("htmlSend", "Line 1\nLine 2\nLine 3", "Line 1
Line 2
Line 3")
>> device.sendAai({action:action})
{retval: true}

>> var action = buildAction("setConfig", "restartApp:waitRenderingType", "event")
>> action
setConfig("restartApp:waitRenderingType", "event")
>> device.sendAai({action:action})
{retval: true}
>>device.sendAai({action:"getConfig(restartApp:waitRenderingType)"})
{retval: 'event'}

>> var action = buildAction("echo", raw('{"a":1, "b":[1,2,3]}'))
>> action
echo({"a":1, "b":[1,2,3]})
>> device.sendAai({action:action})
{retval: {a: 1, b: [1,2,3]}}

Special Character in Action Parameters

There are several characters in action parameters that are meaningful for FindNode parser, starting version 16, these 4 characters can be escaped by prefixed with "\\", "\\" will be translated to "\" in Java.

Single or double quotation marks to surround the text with space and comma. The parser cannot handle if one single or double quotation is used, even the text in surround by quotes. Text such as "I'm John", "John's book" will generate error in parser.
Comma: commas are used to separate multiple arguments. Text with comma needs to be surrounded by quotes.
Semicolon: Semicolon is used to separate multiple actions within a single string.
Any other character after "\\<char>" will retain its value (e.g. regular expression).

setText(I\\'m John)

setText(John\\'s Book)

Since FindNode parser consider unknown text type in parentheses is considered as text, you do not need to protect text inside parentheses, just use escaped character:

>> device.sendAai({query:"T:Field 7&&OX:1", action:"setText(I\\'m John\\, she is Mary\\;);getText"})
{count: 2, list: [{retval: true},{retval: 'I'm John, she is Mary;'}]}

Domino pizza's app is "Domino's", prior version 16, it cannot be run, now it can use escape character:

>> device.sendAai({action:"openApp(Domino\\'s)"})
{retval: true}

Use echo to escape special characters:

>> device.sendAai({action:"echo(John\\'s ball)"});
{retval: 'John's ball'}
>> device.sendAai({action:'echo(a\\, b\\, c)'});
{retval: 'a, b, c'}
>> device.sendAai({action:'echo(he said \\"it was me\\")'});
{retval: 'he said "it was me"'}

Synchronous Actions

Great efforts have been put in to ensure the actions are synchronous, when a "click" is called, not only ensure the click is pressed synchronously, it will wait for certain "events" to ensure the rendering "starts" to happen, cannot ensure the rendering is completed. FindNode offers several ways to perform such check:

Action commands such as "click": it should work most of the time.
"waitQuery": if the new window takes very long time to complete the rendering, use this command to check if the new node is available, add action command after waitQuery, this will ensure the node is found and ready to accept action.

Optional Query (OQ)

Actions require nodes will obtain from the ML, actions require one node, will obtain the first node from ML. However, users can change the behavior by adding option query (OQ) in actions.

E.g.{query:"T:OK", action:"click"} → {action:"click(T:OK)"}

All OQ is optional, if OQ is not specified, the actions will take the nodes from the ML. If actions require one node (e.g. click), the first node in the ML will be used, if the actions support multiple nodes (e.g. getNodes or getText), the entire ML is used. Limited capability in "TP", any TP requires parameter will generate error.

OQ is regular query language with different types identified by different prefixes, OQ likes other queries can generate one or multiple nodes, couple of query types will modify ML. Here are the supported 3 prefixes:

No prefix, apply the query from ML. ML is not change (Similar to "addQuery" action).
"+": New query, new ML (Similar to "newQuery" action).
"*": Same as regular query, the results will replace the ML.

Assuming calculator, the numbers 1 to 9 in 3 rows, "7,8,9", "4,5,6" and "1,2,3" (top to bottom):

>> device.sendAai({query:"T:/[3-9]/", actions:["getText(T:5)", "getText"]})
{count: 2, retval: [{retval: '5'},{retval: ['7','8','9','4','5','6','3']}]}
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(+T:5)", "getText"]})
{count: 2, list: [{retval: '5'},{retval: '5'}]}
>>  device.sendAai({query:"T:/[3-9]/", actions:["getText(*T:5)", "getText"]})
{count: 2, list: [{retval: '5'},{retval: '5'}]}

Here is the difference between regular and "*":

>>  device.sendAai({query:"T:/[3-9]/", actions:["getText(OX:1)", "getText(OX:1)", "getText"]})
{count: 3, list: [{retval: '8'},{retval: '8'},{retval: ['7','8','9','4','5','6','3']}]}
>>  device.sendAai({query:"T:/[3-9]/", actions:["getText(*OX:1)", "getText(*OX:1)", "getText"]})
{count: 3, list: [{retval: '8'},{retval: '9'},{retval: '9'}]}

Calculator, return printable and non-digit:

>>  device.sendAai({action:"getCount(+TP:anyText&&T:/[^0-9]/);getText(T:/^.$/)"})
{count: 2, list: [{retval: 11},{retval: ['÷','×','-','+','%','.','±','=']}]}

It looks like "*" and "+" is similar, consider the following, the "*" matches the list from "ML" (T:2 is not in ML), "+" search the entire screen content.

>>  device.sendAai({query:"T:/[3-9]/", actions:["getText(+T:1)", "getText"]})
{count: 2, list: [{retval: '1'},{retval: '1'}]}
>> device.sendAai({query:"T:/[3-9]/", actions:["getText(*T:1)", "getText"]})
null
>>  lastError()
[0 - getText] No match found

Conditional Expression (CondExp)

Conditional expressions (CondExp) are used in commands such as if, assert, and filter to determine execution flow. FindNode supports three types of conditional expressions:

Boolean Expression – Directly evaluates to true or false.

Comparison Expression – Uses comparison operators to evaluate two values.

Truthy/Falsy Evaluation – Converts values into true or false based on JavaScript-like rules.

All parameters can contain functions, since the function returned value is an object, if the function does not specify a key, the default key "retval" is used. "CondExp" will be used in the usage to represent Conditional Expression. The function name needs to include () to avoid ambiguity.

Boolean Expression

Can be the return value of function call, if no key is specified, the "retval" will be used.

>> device.sendAai({action:"if(true, echo(OK))"});
{retval: 'OK'}
>> device.sendAai({action:"if(exists(T:/^[0-9]$/), echo(OK), echo(Not))"});
{retval: 'OK'}
>> var query = "T:Username&&OX:1";
>> device.sendAai({action:`if(exists(${query}), echo(Found), error(Username not found))`});
{retval: 'Found'}
>> device.sendAai({action:`if(exists(${query}), echo(Found), error(Username not found))`});
null
>> lastError()
User Exception: Username not found
>> device.sendAai({query:query, action:`if(node.editable, setText(${username}), error(Username text field not found))`})
{retval: true}

Truthy/Falsy evaluation

Literal values are evaluated using JavaScript-like truthiness rules. For example, if (0) { ... } will not execute because 0 is falsy, whereas if (1000) { ... } will execute because 1000 is truthy.

Values that evaluate to false:

Boolean: false
Integer: 0
Double: 0.0
String: Empty ("" or '')
null

Values that evaluate to true:

Boolean: true
Integer: Any non-zero number
Double: Any non-zero number
String: Any string with a length greater than 0
Array: Always true
Object: Always true

>> device.sendAai({action:"function(isTrue)", isTrue:"if(%1, echo(is true), echo(is false))"})
{retval: true}
>> device.sendAai({action:"isTrue(0)"});
{retval: 'is false'}
>> device.sendAai({action:"isTrue(1)"});
{retval: 'is true'}
>> device.sendAai({action:"isTrue(Hello)"});
{retval: 'is true'}
>> device.sendAai({action:"isTrue('')"});
{retval: 'is false'}
>> device.sendAai({action:"isTrue(false)"});
{retval: 'is false'}
>> device.sendAai({action:"isTrue(null)"});
{retval: 'is false'}
>> device.sendAai({action:"isTrue([])"});
{retval: 'is true'}
>> device.sendAai({action:"isTrue({})"});
{retval: 'is true'}
>> device.sendAai({action:"isTrue([true,false][0])"});
{retval: 'is true'}
>> device.sendAai({action:"isTrue([true,false][1])"});
{retval: 'is false'}
>> device.sendAai({action:"isTrue(echo(100-100))"});
{retval: 'is false'}
>> device.sendAai({action:"isTrue(getText(BP:editable&&IX:0))"});
{retval: 'is true'}

Comparison Expression

A comparison expression consists of a left parameter, a comparison operator, and a right parameter. Each parameter can be a value or function. If a function is used as a parameter, it will be executed before the comparison is performed. Version 21 does not require "echo" on the right parameter.

Comparison Operators

FindNode supports the following comparison operators:

Operator	Description
= or ==	Checks if the left parameter is equal to the right parameter
>	Checks if the left parameter is greater than the right parameter
<	Checks if the left parameter is less than the right parameter
>=	Checks if the left parameter is greater than or equal to the right parameter
<=	Checks if the left parameter is less than or equal to the right parameter
!=	Checks if the left parameter is not equal to the right parameter

Several notes:

"null" can be used for comparison such as "== null", "!= null", return of "{retval:null}" is valid. It is useful for comparison such as "getText() != null".
Right parameter does not accept arithmetic expression or node.X, use echo() to get it, the limitation will be fixed in next version.

Examples:

You can use echo to test the comparison expression:

>> device.sendAai({action:"echo(40/2 == 20)"});
{retval: true}
>> device.sendAai({action:"echo(40/2 == 2*10)"});
{retval: true}
>> device.sendAai({action:"echo(40/2 == 2*10+1)"});
{retval: false}
>> device.sendAai({action:"if(getText(T:1&&TX)[-1] == '%', echo(matched), error(% not matched))"})
{retval: 'matched'}
>> device.sendAai({action:"if(saveImageInfo().format != 'png', saveImageSettings({format:'png'}))"})
{retval: true}
>> device.sendAai({action:"function(clickExists)", clickExists:"if(exists(T:%1), click(T:%1), return(false))"})
{retval: true}

Array and Object comparisons:

>> device.sendAai({action:"if([1,2,3] == [1,2,3],echo(true),echo(false))"})
{retval: true}
>> device.sendAai({action:"if([1,2,3] == [2,1,3],echo(true),echo(false))"})
{retval: false}
>> device.sendAai({action:"if({a:1, b:2} == {b:2, a:1},echo(true),echo(false))"})
{retval: true}
>> device.sendAai({action:"if({a:1, b:2} == {b:2, a:1, c:3},echo(true),echo(false))"})
{retval: false}
>> device.sendAai({action:"if([1,2,3][1] == 2,echo(true),echo(false))"})
{retval: true}
>> device.sendAai({action:"if({a:1, b:2}.a == 1,echo(true),echo(false))"})
{retval: true}
>> device.sendAai({action:"if({a:1, b:2}.b == 1,echo(true),echo(false))"})
{retval: false}
>> device.sendAai({action:"if([1,2,[4,5]][2] == [4,5],echo(true),echo(false))"})
{retval: true}

Check Android version 14 or above:

>> device.sendAai({action:"function(getAndroidVersion)", getAndroidVersion:`
    openAndroidSetting(DEVICE_INFO_SETTINGS);
    newQuery(T:Software*);
    click;
    newQuery(T:Android version&&OY:1);
    getText;
`})
{retval: true}
>> device.sendAai({action:"getAndroidVersion"});
{retval: '16'}
>> device.sendAai({action:"function(ensureAndroidVersion(14))", ensureAndroidVersion:"if(getAndroidVersion() >= %1, return(true), return(false))"})
{retval: true}
>> device.sendAai({action:"ensureAndroidVersion"})
{retval: true}
>> device.sendAai({action:"ensureAndroidVersion(17)"})
{retval: false}

Get Commands

getActivity

Returns the current foreground activity of the device.

The result is typically in the format:

com.package.name/.ActivityName

Usage:

getActivity

Return:

Return the activity that can be set via openActivity.

Examples:

>> device.sendAai({action:"getActivity"})
{retval: 'com.google.android.gm/.ui.MailActivityGmail'}
>> device.sendAai({action:"getActivity"})
{retval: 'com.android.chrome/com.google.android.apps.chrome.Main'}
>> var activity = device.sendAai({action:"getActivity"}).retval
>> device.sendAai({action:`openActivity(${activity})`})
{retval: true}

See Also: openActivity

getBounds (M)

Return the bounds (format: [left, top, right, bottom]) of the nodes in ML, it returns 2 arrays one with all the IDs the other one with all the bounds.

Usage:

getBounds[(<query>)]

Return:

The array of bounds, each bound is an array of 4-tuples: [x1, y1, x2. y2].

>> device.sendAai({query:"T:/[0-9]/", action:"getBounds"})
{bounds: [[18,1158,370,1521],[370,1158,722,1521],[722,1158,1074,1521],[18,1521,370,1884],[370,1521,722,1884],[722,1521,1074,1884],[18,1884,370,2247],[370,1884,722,2247],[722,1884,1074,2247],[18,2247,370,2610]], count: 10, ids: ['98d1','9c92','a053','a7d5','ab96','af57','b6d9','ba9a','be5b','c5dd']}

getBoolProp and getAllBoolProps (O)

Returns the Boolean property of a node. Possible values include "editable", "progressible", "clickable", "checkable", "scrollable", or "longClickable". If no matching property is found, it returns null.

getAllBoolProps returns all Boolean properties of the node.

Usage:

getBoolProp [(<query>)]
getAllBoolProps[()]

Return:

The array of string properties in "retval"

>> device.sendAai({action:"getBoolProp(T:Checkbox 2)"})
{count: 2, retval: ['checkable','clickable']}
>> device.sendAai({action:"getAllBoolProps(T:Checkbox 2)"})
{count: 7, retval: ['checkable','multiLine','clickable','visibleToUser','focusable','enabled','checked']}

getBoolPropInAncestors (O)

Similar to getBoolProp except it traverse parent to its parent until it reaches the top node tree.

Usage:

getBoolPropInAncestors [(<query>)]

Return:

The array of array of property and node ID or null if not found.

>> device.sendAai({query:"T:John", action:"getBoolPropInAncestors"})
{count: 2, retval: [['clickable','1b86f'],['scrollable','16d5b']]}

getChecked (M)

This command returns the boolean value of "checkable" nodes. Typically, toggle controls, checkboxes or radio buttons are checkable. For instance, class name of ".Switch" or ".Checkbox" are checkable.

Usage:

getChecked [(<query>)]

Return:

If input is one node, it will return true/false in "retval", if multiple nodes, retval will contain the array of true/false, for non-checkable nodes, it will show as "N/A". Return null if none of the checkable node is found. "BP:checkable" to find nodes that is checkable, "BP:checked" to find nodes that have been checked.

See also: setChecked

>> device.sendAai({action:"getChecked(D:Sunday&&TX)"})
{count: 7, retval: [false,true,true,true,true,true,false]}

If do not wish to use description, can use class to search, no other UI element is having class of ".CheckBox":

>>  device.sendAai({action:"getChecked(C:.CheckBox&&IX:0&&TX)"})
{count: 7, retval: [false,true,true,true,true,true,false]}

To find all the checked description:

>> device.sendAai({action:"getDescription(D:Sunday&&R:.day_button_0&&TX&&XBP:checked)"})
{count: 5, retval: ['Monday','Tuesday','Wednesday','Thursday','Friday']}

getCount (M)

This command returns the length of ML or OQ. If OQ is used, it will return zero if not matching is found in OQ.

Usage:

getCount [(<query>)]

Return:

Return the length of ML or OQ in "retval" (prior of version 17, it returns in "count").

Examples:

>> device.sendAai({action:"getCount"})
{retval: 26}
>> device.sendAai({action:"getCount(T:/\\d/)"})
{retval: 10}
>> device.sendAai({query:"T:Not there", action:"getCount"})
null
>> lastError()
No match found
>> device.sendAai({action:"getCount(T:Not there)"})
{retval: 0}

getDeviceName

This command returns the device name.

Usage:

getDeviceName

Return:

Return the device name.

Examples:

>> device.sendAai({action:"getDeviceName"})
{retval: 'samsung-SM-S938U1'}

getFocus

Return the node receiving focus ("focusable" in getNodes(BP)) (usually text field or slider) or null if no node gains focus. If node is found, it will set the ML to the found node.

Usage:

getFocus

Return:

Return "retval" with node ID is found otherwise return null with error.

Example:

To enter the next line from the existing line

>> device.sendAai({action:["getFocus", aaix("setText", username), "addQuery(OY:1)", aaix("setText", password)]})
{count: 4, list: [{retval: '67693'},{retval: true},{count: 1},{retval: true}]}

getFuncRetval

During the execution of a function, the outputs of each command are stored. To retrieve these outputs, use getFuncRetval. For optimal results, it is recommended to place getFuncRetval as the final command within the function. When called without arguments, getFuncRetval returns the outputs of all commands executed during the current function. However, if an argument is provided, it returns only the output of the specified command.

This functionality is particularly useful in functions with multiple commands, allowing precise retrieval of specific outputs, such as from getText. It is important to note that getFuncRetval must be executed within the function's context and does not apply to operations like forEach or repeat. When retrieving results from the search command, only the command name should be specified, without any arguments.

Usage:

getFuncRetval[(<search command>)]

Return:

Return "retval" with array of output in JSON object format. Will generate error if the search command cannot be found.

Examples:

No argument:

>> device.sendAai({action:"function(testEcho)", testEcho:`
    echo(This is a test to echo in different types);
    echo(true);
    echo(1000);
    echo(3.14159);
    echo({a:1, b:{x:1, y:'string'}, c:[100, 200, 300]});
    getFuncRetval()`})
{retval: true}
>> device.sendAai({action:"testEcho"})
{retval: [{retval: 'This is a test to echo in different types'},{retval: true},{retval: 1000},{retval: 3.14159},{retval: {a: 1, b: {x: 1, y: 'string'}, c: [100,200,300]}}]}

With argument:

>> device.sendAai({action:"function(testFR)", testFR:`
    echo(10*20);
    getDeviceName();
    getCount();
    getCurrentPackageName();
    getFuncRetval(%1)`})
{retval: true}
>> device.sendAai({action:"testFR(echo)"})
{count: 1, retval: [{retval: 200}]}
>> device.sendAai({action:"testFR(getDeviceName)"})
{count: 1, retval: [{retval: 'samsung-SM-S938U1'}]}
>> device.sendAai({action:"testFR(getCount)"})
{count: 1, retval: [{retval: 16}]}

To obtain the current location of Tesla:

>> device.sendAai({action:"function(getAddress)", getAddress:`
    click(T:Location);
    newQuery(R:.map_header_text, 1000);
    getText;
    sendKey(Back);
    getFuncRetval(getText)`});
{retval: true}
>> device.sendAai({action:"getAddress"})
{count: 1, retval: [{retval: '9999 Ocean Drive'}]

getIds (M)

This is the default action if action is not defined, this command will return count and array of the ML IDs.

Previously it returns as “ids”, for consistency, it has been changed to “retval”.

Usage:

getids [(<query>)]

Return:

count and array of IDs.

Examples:

>> device.sendAai({})
{count: 26, retval: ['7708','d8a2','e7a6','ef28','824b','860c','89cd','8d8e','914f','9510','98d1','9c92','a053','a414','a7d5','ab96','af57','b318','b6d9','ba9a','be5b','c21c','c5dd','c99e','cd5f','d120']}
>> device.sendAai({action:"getIds(C:.Button)"})
{count: 4, retval: ['8dc5','9547','9908','9cc9']}

getInstalledApps

Returns a list of all installed applications on the device, including system and preinstalled apps.

Each entry includes:

label – app label (the name shown to users)
packageName – unique package name of the app
version – app version

Usage:

getInstalledApps

Return:

Returns a list of all installed apps.

Examples:

>> var result = device.sendAai({action:"getInstalledApps"})
{
    count: 567,
    retval: [
        {
            label: 'Total Control',
            packageName: 'com.sigma_rt.totalcontrol',
            version: '11.0.30.77777'
        },
        {
            label: 'Maps',
            packageName: 'com.google.android.apps.maps',
            version: '26.17.05.901412876'
        }
        >> ... omitted
    ]
}

>> result.count
567
>> result.retval.length
567
>> var apps = result.retval;
>> apps[123]
{label: 'Maps', packageName: 'com.google.android.apps.maps', version: '26.17.05.901412876'}

Search: If a match is found, target contains the app object, if no match is found, target will be null. find() returns the first app whose label matches the specified value. If multiple apps have the same label, only the first match is returned:

var target = apps.find(a => a.label === "Total Control");
print(target)

output

{label: 'Total Control', packageName: 'com.sigma_rt.totalcontrol', version: '11.0.30.75362'}

Partial Matches: Returns all apps whose label contains "Google":

var matches = apps.filter(a => a.label.includes("Google"));
print(matches.length)

output

getLastError, clearLastError

getLastError returns the lastError() message. clearLastError resets to empty string.

Usage:

getlastError

clearLastError

Return:

Return the "lastError()" message in retval.

Examples:

>> device.sendAai({query:"Y:123"})
null
>> lastError()
Query Error: Invalid query type "Y"
>> device.sendAai({action:"getLastError"})
{retval: 'Query Error: Invalid query type "Y"'}
>> device.sendAai({action:"clearLastError"})
{retval: true}
>> device.sendAai({action:"getLastError"})
{retval: ''}

getNodes (M)

This is used to retrieve the information of nodes. The fields can determine what to display. "getNodes" will refresh the node before retrieving information.

Usage:

getNodes

getNodes([<query>] [,<fields>])

Arguments:

fields: The optional "fields" is used to define the fields to return, multiple fields in "fields" are separated by comma. Since it has comma inside the "field", you need to use the quote. String The valid field identifiers are:

P: Package name (String)
C: Class name (String)
R: Resource ID (String)
D: Description (String)
T: Text (String)
AL: Action List (Array)
IT: Input type (Integer)
CC: Child count (Integer)
RI: RangeInfo, provide more information about the type, min, max, current of widget such as SeekBar (slider), use "setProgress" command to change the value.(Object)
BP: All boolean properties in the node.(Array)
B: Return [left top][right bottom] bounds of the node (String)
DO: Drawing order (Integer)
PP: Parent node ID (Hex string)
all: Everything

Return:

Count and array of node information.

Examples:

>> device.sendAai({action:"getNodes(T:/^[0-9]$/,'R,T')"})
{count: 10, retval: [{id: 'a7d3', resourceId: '.button_seven', text: '7'},{id: 'ab94', resourceId: '.button_eight', text: '8'},{id: 'af55', resourceId: '.button_nine', text: '9'},{id: 'b6d7', resourceId: '.button_four', text: '4'},{id: 'ba98', resourceId: '.button_five', text: '5'},{id: 'be59', resourceId: '.button_six', text: '6'},{id: 'c5db', resourceId: '.button_one', text: '1'},{id: 'c99c', resourceId: '.button_two', text: '2'},{id: 'cd5d', resourceId: '.button_three', text: '3'},{id: 'd4df', resourceId: '.button_zero', text: '0'}]}

>> device.sendAai({query:"C:.SeekBar", action:"getNodes(RI)"})
{count: 3, retval: [{id: '5541c', rangeInfo: {current: 47, max: 100, min: 0, type: 0, typeString: 'int'}},{id: '566e1', rangeInfo: {current: 81, max: 100, min: 0, type: 0, typeString: 'int'}},{id: '579a6', rangeInfo: {current: 2, max: 10, min: 0, type: 0, typeString: 'int'}}]}

>> device.sendAai({query:"C:.SeekBar&&IX:-1", action:"getNodes(BP)"})
{count: 1, retval: [{booleanProperties: {checkable: false, checked: false, clickable: false, editable: false, enabled: true, focusable: true, focused: false, longClickable: false, multiLine: false, scrollable: false, selected: false, visibleToUser: true}, id: '579a6'}]}

getPackageName, getCurrentPackageName, getOnScreenPackageNames

“getPackageName” returns the current running package name, it is the same as “getCurrentPackageName”, “getOnScreenPackageNames” (previously "getAllPackageNames") will return the package names of apps currently present on the screen in array.

Usage:

getPackageName
getOnScreenPackageNames

Return:

Package name or array in "retval".

Examples:

>> device.sendAai({action:"getPackageName"})
{retval: 'com.google.android.gm'}
>> device.sendAai({action:"getOnScreenPackageNames"})
{count: 3, retval: ['com.android.systemui','com.sec.android.app.launcher','com.android.chrome']}

getProgress (O)

Return the values of the progress type of UI element such as slider (shorthand of "getNodes(RI)"):

Usage:

getProgress [query]

Return:

the RangeInfo in "retval" of a progressible node, return null if the node is not progressible node:

RangeInfo has one of 3 types: "float" – floating point between min and max, "int" integer between min and max and "percent" between 0 and 1. "current" contains the current value.

>> device.sendAai({action:"getProgress(C:.SeekBar)"})
{retval: {current: 41, max: 100, min: 0, type: 0, typeString: 'int'}}

See also: setProgress

getQuery/getUniqQuery (O)

Accept one node and attempt to generate a query to match the node (or multiple nodes). The query returned from "getQuery" can match multiple nodes, useful to get the text or description from multiple nodes. The query returned from "getUniqQuery" will match only one node, the query obtained will be sent to other devices to locate the nodes. So far, it is rudimentary, we will develop better query string in the future:

Will use the first node of the matching list (query or element) as the reference node.
The output is not fully optimized, for instance, it does not use template.
If multiple nodes are found, "getUniqQuery" will add "IX" to identify the single node, this is error prone especially for applications that do not use resource ID, IX can be large number, any change in UI may impact the accuracy.
UI Explorer’s "Code" button use this command to display the query string.
If offset (OX|OY) is used, it will search for neighboring node with text information (e.g. T:<text>).
-If the text or description information is dynamic, use skipText/skipDescription.
For text and description, if the length is longer than 30 characters, it will list the first 30 characters and add "*" at the end, use setConfig(text:maxQueryLength, <length>) to change it. E.g. If the text of the node is "The quick brown fox jumps over the lazy dog", the query generated will be: "T:The quick brown fox jumps over*".

Usage:

getUniqQuery/getQuery - Default is to use text and description as part of the query.

getUniqQuery/getQuery(<true/false>) – true to skip text in search query.

getUniqQuery/getQuery(<true/false>, <true/false>) – true on second argument to skip description in search query.

Return:

return query string.

Example:

To obtain query for each node:

>> device.sendAai({query:"T:Controls&&TX"}).retval.forEach(
    x => print(device.sendAai({elements:[x],action:"getUniqQuery"}).retval)
)
T:Controls&&OX:-1
T:Controls
T:Controls&&OX:1

getSystemProp

Convenience wrapper for Android’s getprop.

If the argument starts with =, treat the rest as an exact key and return that property’s value (null if missing).e.
Otherwise, perform a substring search (grep-style) over property names and values using the pattern, and return all matches.

Usage:

getSystemProp(<substring>)

Return:

return matching keys and values, if "=" key prefix is found, it will return the value, return null if not found.

Example:

>> device.sendAai({action:"getSystemProp(ro.product.model)"})
{count: 2, retval: [{'ro.product.model': 'SM-S938U1'},{'ro.product.model_for_attestation': 'SM-S938U1'}]}
>> device.sendAai({action:"getSystemProp('=ro.product.model')"})
{retval: 'SM-S938U1'}

getText/getDescription/getHintText/getTextFormats (O|M)

Simpler version of "getNodes", much easier to return obtain the content of text, description and hint text.

getTextFormats on unformatted text (non-Spanned), the output the same as "getText". For formatted text, it will generate html, return text in "retval" and html in "html". "getTextFormats" is mainly used by "Copy Text" in Total Control.

The actions will detect one or multiple nodes, it will return the value if one node is detected otherwise it will return the array of text/description or hint text.

Usages:

getText [(<query>)]

getDescription [(<query>)]

getHintText [(<query>)]

getTextFormats [(<query>)]

Return:

Return an array of text/description/hint text in retval if multiple nodes are found, return text/description/hint text in retval if one node is found. getTextFormats may return an object of "retval" of text and "html" of html.

Examples:

>>  device.sendAai({query:"TP:line,bottom,-1", action:"getText"})
{retval: ['Chats','Calls','Contacts','Notifications']} 
// One value will not return an array
>> device.sendAai({query:"TP:line,bottom,-1&&IX:1", action:"getText"})
{retval: 'Calls'} 

>> device.sendAai({query:"BP:editable", action:"getText"})
{retval: ['text','text','Number','Email','text password','Person Name']}
>> device.sendAai({query:"BP:editable", action:"setText(T:Number, '100');setText(T:text password, secretWord)"})
{count: 2, list: [{retval: true},{retval: true}]}
>> device.sendAai({query:"BP:editable", action:"getText"})
{retval: ['text','text','100','Email','••••••••••','Person Name']}
// getHintText will maintain the value of the original hints, getText does not
>> device.sendAai({query:"BP:editable", action:"getHintText"})
{retval: ['text','text','Number','Email','text password','Person Name']}

>> device.sendAai({query:"T:*Red*", action:"getTextFormats"})
{html: '<p dir="ltr" style="white-space:pre-wrap;">Colors: <span style="color:#FF0000;">Red</span> and <span style="color:#FFFFFF;"><span style="background-color:#0000FF;">Blue background</span></span></p>
', retval: 'Colors: Red and Blue background'}

>> device.sendAai({query:"T:/Red|mono/", action:"getTextFormats"})
{count: 2, retval: [{html: '<p dir="ltr" style="white-space:pre-wrap;"><tt>monospace (test: iiiWWW)</tt></p>
', retval: 'monospace (test: iiiWWW)'},{html: '<p dir="ltr" style="white-space:pre-wrap;">Colors: <span style="color:#FF0000;">Red</span> and <span style="color:#FFFFFF;"><span style="background-color:#0000FF;">Blue background</span></span></p>
', retval: 'Colors: Red and Blue background'}]}

Query Commands

The following commands with one exception (waitQuery) will involve ML.

newQuery/addQuery/waitQuery/refreshQuery/exists(M)

All commands accept a query string.

newQuery starts a new query and recreates the Matching List (ML), similar to using "+" in OQ. Convenient to do multiple screens in action, newQuery is used to refresh the nodes in the new screen.
refreshQuery is newQuery without condition. It will use default template to refresh the nodes in ML.
addQuery performs a search within the existing ML, placing the resulting nodes into the ML, similar to "*" in OQ.
waitQuery does not modify the ML but ensures the query is fulfilled before the timeout expires. If the condition is not met within the timeout, it returns null. Since waitQuery does not change the ML, if you need to modify it like newQuery, use newQuery with a timeout instead.
exists returns true or false, indicating whether the query finds matching nodes. When using exists with a timeout, it applies only to OQ with "+" (new query)—otherwise, an error will be generated. "exists" is the only command return false when node is not found, most command return null.

New Window

When a new window is displayed as a result of a click (or other actions), the node tree (ML) must be updated. If multiple commands involve different screens within a single action key, FindNode cannot automatically detect that a new window has appeared. This leads to mismatches because ML still contains nodes from the previous screen. Several methods can be used to handle this:

Break into two sendAai() calls — the second call will automatically read the current screen.
Use clickForNewWindow — this tells FindNode to expect a new window and re-read the screen.
Use refreshQuery — this forces FindNode to reread the window and update ML.
Use newQuery — if you do not want to specify a query, use refreshQuery instead.

Commands that include a timeout cannot switch to a different package before the timeout expires, or an error will be thrown. These commands will only search within the same package name (or app) during execution.

For examples:

"VG" is typically followed by addQuery rather than newQuery to refine the existing ML.
When performing an action that opens a new window, you need to use newQuery( <query>) to "reread" the nodes in the new window, if you intent to use OQ and do not have other condition, use newQuery(TP:default).

Usages:

newQuery(<query>[, <timeout>])

addQuery(<query>)

exists(<optional query>[, <timeout>])

waitQuey(<query>[, <timeout>]) // Default timeout = 2000ms

Return:

newQuery and addQuery return the size of resulting ML in "retval". When timeout expire waitQuery will return null otherwise "retval" will contain true.

Examples:

The following commands will obtain the same result but "newQuery" will change the ML and impact subsequent actions that use ML:

>> device.sendAai({action:`waitQuery(T:${text},10000);click(T:${text})`})
{count: 2, list: [{retval: true},{retval: true}]}
>> device.sendAai({action:`newQuery(T:${text},10000);click`})
{count: 2, list: [{retval: 1},{retval: true}]}
>>device.sendAai({action:`exists(+T:${text},10000);click`})
{count: 2, list: [{retval: 1},{retval: true}]}

This fail because click will bring up a new window, the ML stores the nodes on the old window.

>> device.sendAai({action:`openAndroidSetting(DEVICE_INFO_SETTINGS);
	click(T:Software information);
	getText('T:Android version&&OY:1')`})
null
>> lastError()
[2 - getText] No match found

There are 4 ways to cause it refresh the nodes on the new window:

Break to 2 "sendAai()", the new sendAai() will force to read the current screen.
Use clickForNewWindow, this will expect a new window to open and read the screen again.
Use "refreshWindow", this will force FindNode to reread the window.
Use "newQuery", if you do not specify query, use refreshWindow.

>> device.sendAai({action:`openAndroidSetting(DEVICE_INFO_SETTINGS);
    click(T:Software information);
    refreshQuery;
    getText('T:Android version&&OY:1')`})
{count: 4, list: [{retval: true},{retval: true},{retval: true},{retval: '16'}]}

intersectX/intersectY (O → M)

If query is specified, the first node of the query will be used for intersect otherwise ML first node will be used. This command will take one node and generate more nodes and store in ML.

Usage:

intersect?[(<query>)]

Return:

true /false: if the operation is successful. If the operation is successful, it will return "count"

If "post query" is specified, the output of post query (see "addQuery") will be stored as ML. If "post query" is not specified, the output of intersect will replace the ML.

Examples:

Assuming calculator app

>> device.sendAai({actions:["intersectX(T:3)", "getText"]})
{count: 2, list: [{retval: 4},{retval: ['1','2','3','%']}]}

"intersect Y" may include other none visible nodes such as result or viewgroup.

>>  device.sendAai({actions:["intersectY(T:3)", "getText"]})
{count: 2, list: [{retval: 8},{retval: [{},{},{},'×','9','6','3','±']}]}

Add a regular expression to ensure at least one character is visible:

>>  device.sendAai({actions:["intersectY(T:3)", "addQuery(T:/.+/)", "getText"]})
{count: 3, list: [{retval: 8},{retval: 5},{count: 5, retval: ['×','9','6','3','±']}]}

This will work too:

>> device.sendAai({actions:["intersectY(T:3)", "getText(T:/.+/)"]})
{count: 2, list: [{retval: 8},{count: 5, retval: ['×','9','6','3','±']}]}

Or use the filter to remove non-text nodes:

>> device.sendAai({actions:["intersectY(T:3)", "filter(node.text != null)", "getText"]})
{count: 3, list: [{retval: 8},{retval: true},{count: 5, retval: ['×','9','6','3','±']}]}

push/pop & load/save (M)

These commands store and restore the current ML (Match List). They are mainly useful when executing multiple commands within the same workflow.

Typical usage:

Save the current ML
Execute newQuery and perform actions
Restore the previous ML and continue processing

If a name is specified, save(<name>) stores the ML using that name, and load(<name>) restores the saved ML in a subsequent sendAai() call.

When loading a saved ML, the current foreground package name must match the package name associated with the saved ML; otherwise, an error will occur.

Usage:

push

pop

save([<name>])

load ([<name>])

Return:

push/pop return retval of true. save/load generate an error if the name is not found or the package names do not match.

Examples:


>> device.sendAai({query:"T:/^[0-9]$/", action:"nsClick(T:1);save(calc)"})
{count: 2, list: [{retval: true},{retval: true}]}

>> device.sendAai({action:"load(calc);nsClick(T:2);getText(R:.editText_result)"})
{count: 3, list: [{retval: true},{retval: true},{retval: '12'}]}

>> device.sendAai({action:"load(calc);nsClick(T:3);getText(R:.editText_result)"})
{count: 3, list: [{retval: true},{retval: true},{retval: '123'}]}

Faster, do not need to search the query in every sendAai():


>> device.sendAai({query:"TP:reduced&&T:/./&&IX:-1&&TX", action:"getText", execTime:true})
{count: 5, retval: ['Games','Apps','Search','Books','You'], timeFindNode: 161}

>> device.sendAai({query:"TP:reduced&&T:/./&&IX:-1&&TX", action:"getText(IX:0)", execTime:true})
{retval: 'Games', timeFindNode: 154}

>> device.sendAai({query:"TP:reduced&&T:/./&&IX:-1&&TX", action:"getText(IX:1)", execTime:true})
{retval: 'Apps', timeFindNode: 169}

>> device.sendAai({query:"TP:reduced&&T:/./&&IX:-1&&TX", action:"getText;save(playStore)", execTime:true})
{count: 2, list: [{count: 5, retval: ['Games','Apps','Search','Books','You']},{retval: true}], timeFindNode: 153}

>> device.sendAai({action:"load(playStore);getText(IX:0)", execTime:true})
{count: 2, list: [{retval: true},{retval: 'Games'}], timeFindNode: 13}

>> device.sendAai({action:"load(playStore);getText(IX:1)", execTime:true})
{count: 2, list: [{retval: true},{retval: 'Apps'}], timeFindNode: 12}

The current foreground package name must match the saved package name:


>> device.sendAai({action:"load(playStore);getText(IX:0)"})
null

>> lastError()
[0 - load] Node is saved in com.android.vending, but current package is com.sec.android.app.launcher

Name not found:


>> device.sendAai({action:"load(Noname);getText(IX:0)"})
null

>> lastError()
[0 - load] Cannot find saved nodes for Noname

reduceNodes (M → M)

Equivalent to the RN query filter. Takes no arguments. Reduces the size of the current ML using heuristic node reduction. The reduced node count is returned in count.

Usage:

reduceNodes

Return:

Return the number of resulting nodes in "retval".

See also: Query "RN".

>> device.sendAai({query:"TP:reduced"}).count
54
>> device.sendAai({action:"reduceNodes"})
{retval: 54}

See also: Query "TP:reduced", "RN".

Sort (M)

Sort based on the node bounds. It is identical to "ST" query, same argument as ST.

Usage:

sort(X|Y|YX)

Return:

Return true/false in "retval".

See also: Query "ST".

viewGroup (O → M)

This command is similar to "VG" query, will accept a level and query. Query if defined is used to identify the first node to seach for the group. Similar to "VG", it will change ML. Previously this command is called "getViewGroup".

Usage:

viewGroup

viewGroup([<query>,] <level>)

Return:

If ViewGroup is found, it will return the count in the ML, if ViewGroup is not found, it will return null.

See also: Query "VG".

>> device.sendAai({query:"T:Battery&&VG", action:"getText(T:/\\d%/)"})
{retval: '92% available'}
>> device.sendAai({actions:["viewGroup(T:Battery)","getText(T:/\\d%/)"]})
{count: 2, list: [{retval: 6},{retval: '92% available'}]}

Action Commands on Nodes

Commands in this section perform actions to the nodes.

click/nsClick/click2/longClick/ clickForNewWindow /clickDown/clickUp (O)

Various kinds of clicks on the node, this is synchronous click, will make sure screen is starting to refreshed when the control is returned.

click: Click in the center of the node.
longClick: Hold and click longer (500ms) to simulate a long click.
click2: Click at the random location in the node.
clickForNewWindow: Performs a click and expects a new window to appear as a result and refresh ML with the default template. Returns false if no new window is displayed or node is not found after refresh.
clickDown: Press (touch down) on the center of the node and keep the touch active until clickUp() is called. Used to begin a touch gesture such as a tap or long press.
clickUp: Release the previously pressed touch to complete the gesture. The node argument (if provided) is ignored — the touch is always released at the same position where clickDown() occurred.

"click", "click2", "longClick" and "clickForNewWindow" commands will monitor the events for the changes of the screen after the click is performed, if the screen is not changed, these clicks will fail and eventually timed out. If the screen is not changed (or very little change) after the click is performed, use "nsClick":

nsClick: Performed the click but does not monitor the events.

"nsClick" with also accepts 2 integers, it will perform a click on x and y coordinate if hardcoded x and y values are used the scripts are not portable will fail on new device, resolution or updated apps. Certain apps that use native interface, some of the nodes can be found (e.g. WebView), can use found nodes to calculate the missing node location and perform a click. Use "getBounds" or "getNodes(B)" to obtain the dimension (or bounds) of the node.

Usage:

click/nsClick/click2/longClick/clickForNewWindow/clickDown/clickUp[(<query>)]

nsClick(<x>, <y>)

Return:

true or false to indicate if the operation is successful.

Example:

>> device.sendAai({query:"T:5", action:"click"})
{retval: true}
>> device.sendAai({action:"nsClick(T:5)"})
{retval: true}

Calculator app, use button 4 dimension to calculate the location of button 5 and perform a click:

>> device.sendAai({action:"getBounds(T:4)"})
{bounds: [[18,1585,370,1981]], count: 1, retval: ['a641']}
>> var dim = device.sendAai({action:"getBounds(T:4)"}).bounds[0]
>> var x = (dim[2]-dim[0])/2 + dim[2]
>> var y = (dim[3]-dim[1])/2 + dim[1]
>> print(x + "," + y)
546,1783
>> device.sendAai({action:`nsClick(${x}, ${y})`})
{retval: true}
// Use clickDown and clickUp to simulate 3 seconds long click
>> device.sendAai({query:"T:Hold 3 seconds", action:"clickDown;sleep(3000);clickUp"})
{retval: true}

echo,log (O)

echo and log has exact same usage, echo display the output in "retval", log display the output in logcat. They accept the following types:

Value: Boolean, integer, double, string, function, array and object.
Function call: function call with the return value.
Comparison expression: If comparison operator is used, it will return true or false.
For string, if the string contains valid arithmetic expression, calculation will be performed and output results.
node.X, will return the information of the first node in ML.

Usage:

echo (<CondExp>)

log (<CondExp>)

Return:

echo will show return value in "retval".
log will always show retval: true.

Examples:

>> device.sendAai({action:"echo(string); echo(100); echo(3.14159); echo(true); echo(false); echo(null)"})
{count: 6, list: [{retval: 'string'},{retval: 100},{retval: 3.14159},{retval: true},{retval: false},{retval: 'null'}]}

>> device.sendAai({action:"echo(listAndroidSettings()[0])"});
{retval: 'ACCESSIBILITY_DETAILS_SETTINGS'}
>> device.sendAai({action:"echo(listAndroidSettings()[-1])"});
{retval: 'ZEN_MODE_PRIORITY_SETTINGS'}

Perform simple integer arithmetic, acceptable operators are: +, -, *, /, % and ().

>> var s = "(123 + 456) * (789-456)"
>> device.sendAai({action:`setText(BP:editable,${s}); echo(getText(BP:editable))`})
{count: 2, list: [{retval: true},{retval: 192807}]}
>> device.sendAai({action:`setText(BP:editable,${s});log(getText(BP:editable))`})
{count: 2, list: [{retval: true},{retval: true}]}

Logcat will show "Log: 192807".

filter (M)

The "filter" command will allow users to manipulate the ML based on the condition expression specified, the filter will iterate every node, remove all unmatched nodes, it will generate error if ML is empty. The conditional usually include "node.X".

Usage:

filter(<CondExp>)

Return:

return true on success or null if ML is empty.

Examples:

If entire screen has one input line, can use the following:

>> device.sendAai({action:`getCount;push;filter(node.isEditable);getCount; setText(${input});pop;getCount`})
{count: 7, list: [{retval: 227},{retval: true},{retval: true},{retval: 1},{retval: true},{retval: true},{retval: 227}]}
>> device.sendAai({action:"filter(node.text!=null); getText"})
{count: 2, list: [{retval: true},{count: 21, retval: ['Calculator','AC','÷','×','DEL','7','8','9','-','4','5','6','+','1','2','3','%', '0','.','±','=']}]}

node.text will return true if it is not null so it is a short form of node.text != null:

>> device.sendAai({action:"filter(node.text);filter(node.isNumber);getText"})
{count: 3, list: [{retval: true},{retval: true},{count: 10, retval: ['7','8','9','4','5','6','1','2','3','0']}]}

forEach (M)

The action loop through each node in ML and execute the provided actions, one at a time. Several rules apply:

Cannot execute actions that can be changed the ML or it will generate an error.
All optional query (OQ) with prefix ("+" or "*") will result with error.
All actions loop through will only see one node.
The return value:
- For single action in the argument, "retval" array will be used to store the output for each node.
- For multiple actions, the JSON name will contain the array of array, inner array will be the execution results of a node, output array will contain array of all nodes.

For multiple actions, "forEach" takes the name of JSON object, the value defines the actions either in array or string will semicolon delimiter. For single action, can put the action name in the argument.

Usage:

forEach(<action name>)

forEach(<JSON name>)

Return:

For single action in the argument, "retval" array will be used to store the output for each node.
For multiple actions, the "retval" will contain the array of array, inner array will be the execution results of a node, outer array will contain array of all nodes.

Examples:

The first line runs "getText" once, the second line run "getText" 4 times.

>> device.sendAai({query:"T:1&&TX", action:"getText"})
{count: 4, retval: ['1','2','3','%']}
>> device.sendAai({query:"T:1&&TX", action:"forEach(getText)"})
{count: 4, retval: [{retval: '1'},{retval: '2'},{retval: '3'},{retval: '%'}]}
>> device.sendAai({query:"R:/.day_button_[0-6]/", action:"getText"})
{count: 7, retval: ['S','M','T','W','T','F','S']}
>> device.sendAai({query:"R:/.day_button_[0-6]/", action:"forEach(setChecked(true))"})
{count: 7, retval: [{retval: true},{retval: false},{retval: false},{retval: false},{retval: false},{retval: false},{retval: true}]}

In the example document, for calculator, it fills in all operations in "elements" and use "forEach(nsClick)" to key in calculation.

If, assert (O)

"if" and "assert" commands can use condition to determine true/false (refer to "Conditional Expression" in Action section), for "if" command, if true, call "then" otherwise call "else", for "assert" true will return true, fal Several rules:

For both commands, conditional expression is required to determine true/false. Please refer to CondExp section in ACTIONS section.
For "if", if true, "then" will be called, if false, "else" will be called, null can be specified in "then" or "else" to do nothing. If "else" is specified or specified as null, the return value is {retval: true}.
For multiple nodes in ML, only the first node in ML will be used.
For "assert", if true will return {retval: true}, if false will cause the error, it will propagate the assert error to sendAai() and subsequently use RingoJS assert.fail() to generate AssertionError. If tests are running, AssertionError will cause the test to fail.
"if" can include "return" to exit the function.

Usage:

if(<CondExp>, <then>, <else>)

assert(<CondExp>[, <message>])

Return:

The return value of "then" or "else". Assert will either return true or will raise AssertionError in RingoJS assert.fail().

Examples:

In Tesla app, try to find out the % of the battery remain, it can show in range (suffix in "mi") or percentage (suffix "%"), click on it to change mode. Here is the function to obtain battery remains in %:

>> device.sendAai({action:"function(getPctg)", getPctg:"if(exists(+T:/^\\d+ mi/), nsClick);getText(+T:/^\\d+%/)"})
{retval: true}
>> device.sendAai({action:"getPctg"})
{retval: '79%'}

Create a function that return the low % remaining that needs to be charge:

>> device.sendAai({action:"function(getLowPctg)", getLowPctg:"echo(20)"})
{retval: true}

Create a function to return true or false that needs to be charge or not:

>> device.sendAai({action:"function(needCharge)", needCharge:"if(getPctg() <= getLowPctg(), return(true), return(false))"})
{retval: true}
>> device.sendAai({action:"needCharge"})
{retval: false}

Alternatively, use return:

>> device.sendAai({action:"function(needCharge)", needCharge:"return(getPctg() <= getLowPctg())"})
{retval: true}
>> device.sendAai({action:"needCharge"})
{retval: false}
>> device.sendAai({action:"function(getLowPctg)", getLowPctg:"echo(100)"})
{retval: true}
>> device.sendAai({action:"needCharge"})
{retval: true}

Also, replace "return" with "echo" will achieve the same result. You can also use object symbol:

You can also use the object to store the values. For example:

device.sendAai({action:"function(getNum)", getNum:"return({min:10, max:90})"})
device.sendAai({action:"if(getPctg() < getNum().min, chargeNow())"})
device.sendAai({action:"if(getPctg() > getNum().max, stopNow())"})

Assuming calculator app, resource of .formula will display the result, we can create a function for single-digit addition as follow:

>> device.sendAai({action:"function(add)", add:"click(T:%1);click(T:+);click(T:%2);click(T:=);getText(+R:.formula)"})
{retval: true}
>> device.sendAai({action:"add(8,9)"})
{retval: '17'}

function calculator app against arithmetic expression in echo:

>> device.sendAai({action:"function(equal)", equal:"if(add(%1,%2) == echo(%1+%2), return(true), return(false))"})
{retval: true}
>> device.sendAai({action:"equal(8,9)"})
{retval: true}

Use "assert":


>> device.sendAai({action:"assert(add(8,9)==10)"})
[AssertionError 'Assertion error on "add(8,9)==10"']
>> device.sendAai({action:"assert(echo(true))"})
{retval: true}
>> device.sendAai({action:"assert(echo(false))"})
[AssertionError 'Assertion error on "echo(false)"']
>> device.sendAai({action:"assert(echo(false), 'This should not happen')"})
[AssertionError 'This should not happen']

refresh (M)

The accessibility nodes information is cached, if the screen has been updated, the cache information may not be accurate, use "refresh" to force to reread the nodes information.

Usage:

refresh [(<query>)]

Return:

Return count and true (always return true)

Example:

>> device.sendAai({query:"T:Overall Cpu Usage&&OX:1", actions:["getText", "sleep(2000)", "refresh", "getText"]})
{count: 4, list: [{retval: '29'},{retval: true},{retval: true},{retval: '78'}]}

repeat (M)

This command is useful for repeated actions, it shares the action input (JSON or single action) and similar return values as "forEach", it will repeat the action/actions execution in given number of runs. The maximum repeat count is 20. Use "setConfig" on "action:repeat:maxCount" to increase the maximum repeat count.

Usage:

repeat (<count>, <action name>)

repeat (<count>, <JSON name>)

Return:

For single action in the argument, "retval" array will be used to store the output for each run.
For multiple actions, the JSON name will contain the array of array, inner array will be the execution results of a run, outer array will contain all runs.

Examples:

This command clicks 5 "1" in calculator:

>> device.sendAai({action:"repeat(5,click(T:1))"})
{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]}

To set the charging current of EV car:

Reduce by 1 Amp:

>> device.sendAai({action:"click(T:/[0-9]+ A/&&OX:-1)"})
{retval: true}

Reduce by 5 Amp:

>> device.sendAai({action:"repeat(5, click(T:/[0-9]+ A/&&OX:-1))"})
{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]}

Reduce and verify:

>> device.sendAai({query:"T:/[0-9]+ A/", action:"getText;repeat(5,click(OX:-1));refresh;getText"})
{count: 4, list: [{retval: '27 A'},{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]},{retval: true},{retval: '22 A'}]}

saveImage, saveImageInfo, saveImageSettings (M)

The saveImage command saves an image on all the nodes in ML to the device. It has several configurable properties that can be adjusted using saveImageSettings, and the current properties can be obtained with saveImageInfo.

Images can be saved in JPEG or PNG formats, either in the shared pictures folder or in the app's private folder. The supported properties are:

format: jpg or png. Specifies the image type. The default is jpg.
folder: share or private. "share" saves the image in the public pictures folder, while "private" saves the image in Total Control private folder "/storage/emulated/0/Android/data/com.sigma_rt.totalcontrol/files/Pictures" directory. The default is "share".
quality: 30 – 100. Specifies the quality of the image, applicable only to JPEG. The default value is 80.
resolution: 720p, 1080p, 2k, 4k, or native. Defines the target resolution after saving. For example, an image of a 2K screen can be saved as 720p to reduce file size. "native" saves the image at its original resolution. The default is 1080p.

By default, saveImage attempts to all nodes in ML, will enlarge region to accommodate all visible nodes in ML. It also accepts an optional query (OQ), such as "IX:0" to save only the first node in ML. If the operation is successful, the absolute path of the saved image file is returned.

To configure properties, saveImageSettings accepts settings in JSON format. For example:

"saveImageSettings({format: 'png', folder: 'private', resolution: '720p'})".

To retrieve the current settings, use saveImageInfo, which returns the configuration in JSON format.

Usage:

saveImage([query])

saveImageSettings({…})

saveImageInfo()

Return:

saveImage returns the absolute path of the image file. null on error.
saveImageSettings returns true on success. null on error.
saveImageInfo returns the current properties.

Examples:

Use calculator as example:

>> device.sendAai({query:"T:/^[0-9]$/", action:"saveImage"})
{retval: '/storage/emulated/0/Pictures/Samsung-SM-S908-2024-12-01-03-51-48.jpg'}
>> device.sendAai({action:"saveImageSettings({format:'png',folder:'private',
resolution:'720p'})"})
{retval: true}
>> device.sendAai({query:"T:/^[0-9]$/", action:"saveImage"})
{retval: '/storage/emulated/0/Android/data/com.sigma_rt.totalcontrol/files/Pictures/Samsung-SM-S908-2024-12-01-03-56-18.png'}

The image looks like this for T:0 to T:9, can use adb pull on the return image filename.

If you just want to have one node, use "IX":

>> device.sendAai({action:"saveImageSettings({format:'jpg',quality:50, folder:'private',resolution:'1080p'})"})
{retval: true}
>> device.sendAai({action:"saveImageInfo"})
{retval: {folder: 'private', format: 'jpeg', quality: 50, resolution: '1080p'}}
>> device.sendAai({query:"T:/^[0-9]$/", action:"saveImage(IX:0)"})
{retval: '/storage/emulated/0/Android/data/com.sigma_rt.totalcontrol/files/Pictures/Samsung-SM-S908-2024-12-01-04-04-45.jpg'}

The first node is 7 (top-left):

If you change from IX:0 to IX:-1, the last node will display as zero.

The following command is used to capture the navigation icons of Skype:

>> device.sendAai({query:"T:Calls&&VG", action:"saveImage"})
{retval: '/storage/emulated/0/Android/data/com.sigma_rt.totalcontrol/files/Pictures/Samsung-SM-S908-2024-12-01-04-14-47.jpg'}

"VG" will place the largest ViewGroup on the first node, query:"T:Calls&&VG&&IX:0" will obtain the same image.

On SigmaTestApp, if just want to save the Checkbox:

>> device.sendAai({query:"C:.CheckBox", action:"saveImage"})
{retval: '/storage/emulated/0/Android/data/com.sigma_rt.totalcontrol/files/Pictures/Samsung-SM-S908-2024-12-01-04-30-37.jpg'}

scrollIntoView (M)

Accepts a query string, an optional scrolling direction, and an optional maximum number of pages to scroll. The command attempts to locate a matching node by scrolling in the specified direction. If one or more matching nodes are found, it returns immediately. If the direction is invalid, the command returns null with an error. If the match cannot be found after the allowed number of pages, it also returns null with an error.

Scrolling is performed using page-up/page-down gestures. By default, the maximum number of pages is 30, which can be adjusted globally using.

setConfig(navi:maxPageScroll, <number>).

If the <max pages> argument is provided in the command, it overrides the configured default for this call.

Because the scroll process may take some time, this command automatically extends the timeout in sendAai until scrolling finishes.

To ensure scripts work across different screen resolutions and device sizes, scrollIntoView is more reliable than manually issuing Page Up/Down actions.

This action also ensures the first matched node is fully visible and updates ML. ML may contain more than one node.

Usage:

scrollIntoView(<query>)[,<direction>][,<number of pages>] // Default down

Arguments:

<direction> — optional. One of:

"down": Start from the current location and scroll downward.

"up": Start from the current location and scroll upward.

"top": Fast-scroll to the top, then search by scrolling down.

"bottom": Fast-scroll to the bottom, then search by scrolling up.

If omitted, the default is "down".

If any other value is used, the command returns null with an error.

<max pages> — optional.

Maximum number of pages to scroll. 0 will only match current screen.

If omitted, the default is the configured value from navi:maxPageScroll (default 30).<direction > can be one of the following, if not specify, the default is "down".

Return:

Returns null if the direction is invalid or if the query cannot be found. Otherwise returns true. ML will be updated to the found nodes.

Example:

For a long list, scroll 50 pages until "John" is found and click on the found node.

>> device.sendAai({actions:["scrollIntoView(T:John,50)", "click"]})
{count: 2, list: [{retval: true},{retval: true}]}

Find Tesla mileage:

>> device.sendAai({action:"scrollIntoView(T:/[0-9]+ miles/, topDown, 2);getText"})
{count: 2, list: [{retval: true},{retval: '9,999 miles'}]}

openActivity

Launches an activity using the Android am (Activity Manager) command.

am is a built-in Android shell tool used to start activities, send intents, and control app behavior. openActivity is a wrapper around am start.

Takes one parameter, which is passed to am start.

If the parameter does not start with "-", -n will be automatically added in front of it.

Usage:

openActivity(<activity>)

Returns:

May return true even if the activity does not actually open.

Example:

>> var activity = device.sendAai({action:"getActivity"}).retval;
>> activity
com.google.android.gm/.ComposeActivityGmailExternal
>> device.sendAai({action:`openActivity(${activity})`})
{retval: true}
>> device.sendAai({action:'openActivity(-a android.intent.action.VIEW -d https://www.sigma-rt.com)'})
{retval: true}
>> device.sendAai({action:"openActivity(-d https://www.sigma-rt.com)"})
>> device.sendAai({action:"openActivity(-a android.intent.action.MAIN -c android.intent.category.LAUNCHER -p com.google.android.youtube)"})
{retval: true}
>> device.sendAai({action:"openActivity(-a android.settings. -a android.settings.WIFI_SETTINGS)"})
{retval: true}
>> device.sendAai({action:"openActivity(-a android.intent.action.SENDTO -d mailto:support@sigma-rt.com)"})
{retval: true}

See Also: getActivity

setChecked (M)

This command applies to nodes with "checkable" nodes. Typically, toggle controls are checkable, checkboxes or radio buttons, example of class can be ".Switch" or ".Checkbox". For radio button, setChecked(false) will not work since FindNode does not know which radio button to click to disable the existing radio button.

Usage:

setChecked(true|false)

setChecked([query], true|false)

Arguments:

"setChecked(true|false)", since there is no permission to set the checked value, it will click the node to toggle the value if value needs to be changed. It accepts multiple nodes, will ignore non-checkable nodes.

Returns:

return true/false to indicate at least 1 node has been changed.

setProgress (O)

To set the value of the slider bar, the min, max, type, value can be obtained by "getNodes('RI’)",

Usage:

setProgress(<number>)

setProgress(<query>,<number>)

Arguments:

<number>: Integer or decimal to set the value determined by "type", "min" and "max"

Return:

true/false: return true if value has been successful set or false if set to the same value or out of range. Return null if the node is not progress node.

See also: getProgress

Example:

Set the display brightness to 50%:

>> device.sendAai({query:"T:Brightness&&VG&&XC:.SeekBar", action:"getProgress"})
{retval: {current: 85983232, max: 267386880, min: 0, type: 0, typeString: 'int'}}
>> var max = device.sendAai({query:"T:Brightness&&VG&&PQ:'C:.SeekBar'", action:"getProgress"}).retval.max
267386880
>> device.sendAai({query:"T:Brightness&&VG&&PQ:'C:.SeekBar'", action:aaix("setProgress", max/2)})
{retval: true}

In sound setting, set all 4 sound volumes to 50%:

>> device.sendAai({query:"C:.SeekBar"}).retval.forEach(
    x => { 
        var max = device.sendAai({elements:[x], action:"getProgress"}).retval.max;
        device.sendAai({elements:[x], action:`setProgress(${max/2})`});
    }
)

setText (O)

If the node is a text field, setText enters the specified input string programmatically. No input method is involved (the text is set directly on the node). If you need to use an input method, use sendKey instead.

For Android 11 or later, if the input string ends with "\n" or "\r\n":

In many apps, search fields do not have a search button. A newline will trigger the Search action after the text is entered.
In apps where Enter = Send, the message will be sent automatically without clicking a send button.

Additional behavior:

By default, setText replaces the existing text with the new value. If the new text starts with "+", it is appended to the end of the current text instead of replacing it.
If there is only one editable field on the screen, setText will use that field by default.
If the matched node is not editable (for example, a label of a text field), setText will attempt to locate a nearby text field using OX:1, then OY:1.
You can use BP:editable or TP:textInput to locate text fields. See examples below.

Usage:

setText([query], <input string>)

setText([query], +<input string>)

Arguments:

Input string: By default the text field will be cleared before entering the string, if the first letter is "+", it will add on to the existing string. For multiple lines of text field, use "\n" to skip to next line.

Return:

true or false to indicate if the operation is successful.

Examples:

Below is example of UI Explorer of "TP:textInput"

// Obtain all text field
>> device.sendAai({query:"TP:textInput", action:"getText"})
{count: 5, retval: ['Text','Number','Email','Text password','Text']}
// Hint Text starts support on API 26
>> device.sendAai({query:"TP:textInput", action:"getHintText"})
{count: 5, retval: ['Text','Number','Email','Text password','Text']}
// setText based on hint text
>> device.sendAai({query:"TP:textInput,Email", action:"setText(support@sigma-rt.com)"})
{retval: true}
>> device.sendAai({query:"BP:editable", action:"setText(HT:Number, 100);setText(HT:Text password, secretWord)"})
// Use "+" for concatenation
>> device.sendAai({query:"TP:textInput&&IX:-1", action:"setText('Hello');setText('+ World'); getText"})
{count: 3, list: [{retval: true},{retval: true},{retval: 'Hello World'}]}
 
// For Enter as Send, add "\n" at the end of text
>> device.sendAai({query:"TP:textInput", action: "setText('Hello\n')"})

// Add Emoji at the middle of string
>> device.sendAai({query:"TP:textInput&&IX:-1", action:`setText("hello ${String.fromCodePoint(0x1f643, 0x1f644)} world");getText`})


// Below are 5 ways to enter on "Number" field:
>> device.sendAai({query:"BP:editable&&T:Number", action:"setText(100)"})
{retval: true}
>> device.sendAai({query:"TP:textInput,Number", action:"setText(200)"})
{retval: true}
>> device.sendAai({query:"BP:editable&&HT:Number", action:"setText(300)"})
{retval: true}
>> device.sendAai({query:"T:Field 2", action:"setText(400)"})
{retval: true}
>> device.sendAai({query:"TP:textInput", action:"setText(IX:1,500)"})
{retval: true}

You can setup a function to enter multiple text fields a bit easier

>> device.sendAai({query:"TP:textInput", action:"function(setNextText)", setNextText:"setText(*OY:1, %1)"})
{retval: true}

>> device.sendAai({query:"TP:textInput", action:`
  setText(IX:0, Hello);
  setNextText(100);
  setNextText(support@sigma-rt.com);
  setNextText(password);
  setNextText(long long long word)
  `})
{count: 5, list: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: true}]}

>> device.sendAai({query:"TP:textInput", action:"getText"})
{count: 5, retval: ['Hello','100','support@sigma-rt.com','••••••••','long long long word']}

showOnScreen (O)

This will ensure the first node will appear fully on the screen, if associated UI element is partially displayed, it will scroll until the UI element is fully visible on the screen. If the node is too large to fit into the scrollable area, it will generate an error.

Usage:

showOnScreen [(<query>)]

Return:

true to indicate the screen has been scrolled, false to indicate the node is already fully visible on screen, nothing is done. Return null if the node is too large to fit into scrollable area.

Example:

>> device.sendAai({action:"showOnScreen(T:Ace)"})
{retval: true}

swipeAcross (O)

swipeAcross will take the first node in ML, swipe from end to end based on the direction specified, the optional number of steps determine the speed of swipe, low number is faster than high number. This command is useful to dismiss a message by swiping to left or right of the node.

Usage:

swipeAcross [query,] <left|right|up|down> [,<number of steps>]

Return:

true or false to indicate if the swipe is successful.

Examples:

>> device.sendAai({action:"swipeAcross(T:Alarm deleted, right)"})
{retval: true}

until (M)

until will wait for certain condition based on the option specified, if the condition is met, it will return true, if timeout expired it will return null with error. Currently 2 types are supported:

until([query], gone, <timeout>)

"gone" will wait until the specified node (first element in ML or OQ) disappear.

Example:

The following will wait until reset button is disappeared, both formats are supported:

>> device.sendAai({query:"R:.btn_img_reset_time", action:"until(gone, 20000)"})
{retval: true}
>> device.sendAai({action:"until(R:.btn_img_reset_time, gone, 20000)"})
{retval: true}

until([query], changed, T|D, <timeout>)

Will check until timeout or text (T) or description (D) content is changed.

Example:

The following example check if Nasdaq index is changed:

>> device.sendAai({query:"T:Nasdaq*&&OY:1", actions:["getDescription", "until(changed,D,30000)","getDescription"]})
{count: 3, list: [{retval: '10,989.25'},{retval: true},{retval: '10,988.50'}]}
>> device.sendAai({actions:["until(T:Nasdaq*&&OY:1, changed, D, 30000)","getDescription"]})
{count: 2, list: [{retval: true},{retval: '10,988.50'}]}

Action Commands

These commands do not involve nodes but useful for creating scripts.

error

Will stop function execution and return null on sendAai() with lastError() contain the “User Exception: ” with error message.

Usage:

error(<error message>)

Return:

Will stop execution, no return is necessary.

Examples:

>> device.sendAai({query:"P:com.android.systemui&&R:.battery_percentage_view",   
    action:`if(getText() < 10, 
                error(battery too low), 
                echo(battery level is adequate))
`})
null
>> lastError()
User Exception: battery too low

exec

The "exec" command executes the specified command using "sh", and returns the output. If an error occurs, the command will return null, and the error message can be retrieved using "lastError()". Note that Not all commands are allowed to run; check "lastError()" for "Permission denied" messages if execution fails.

Moreover, the command must not contain >, as it is reserved for capturing output and error messages.

Notes:

Android 14+ provide rudimentary exec, only single command is allowed. To revert back to full settings, use the settings command on the key to false, Total Control will set this flag during connection process.

adb shell settings put global settings_enable_monitor_phantom_procs false

Usage:

exec (<command line>)

Return:

return output as "retval" and return null if error occurs.

Examples:

>> device.sendAai({action:"exec(dumpsys activity activities | grep ResumedActivity | tail -1)"})
{retval: '  ResumedActivity: ActivityRecord{57c186 u0 com.veronicaapps.veronica.simplecalculator/.MainActivity t1027}'}
>> device.sendAai({action:"exec(ls /data/local)"})
null
>> lastError()
ls: /data/local: Permission denied
>> device.sendAai({action:"exec((cd /etc;ls|head -5)&&echo ----&&ls |head -5)"})
{retval: 'ADP.xml
ASKSB.xml
ASKSC.xml
ASKSHB.xml
ASKSP.xml
----
acct
apex
audit_filter_table
bin
bootstrap-apex'}

function/funcExists/funcDelete/funcReset/funcList

This action command will create a new action composed:

Once the function is created, the function can be called by actions/functions that accept function or action. Do not create the same name as the existing command or it will never be called.
It supports batch file style variable substitutions, %<number> corresponding to calling argument. Like normal action arguments, substitution can be string, floating point, Boolean (true/false) and integer. Insufficient arguments will use the defaults when "function" is called to create action. Any % follow without a number will not be interfered, if "%1" is needed as part of the argument (e.g. setText), use "%%" to represent "%", so, "%%1" will become "%1", no substitution will be performed.
It supports default value, when a function is setup, new argument will be the default values. The default value will be used if the action argument is less than the default number of arguments.
Do not use recursive (or function calls resulting in a loop), it will likely result in a crash.
Prior version 17, function will return the return values for each command in the function. Starting version 17, function will return the return value of the last command in the function. "return" command can change that behavior (see "return" action for more information). "forEach" and "repeat" will retain previous return values.

Usage:

Create new function:

function(<action name in JSON>(arguments)) <JSON name>:<actions> in [] or ";" or single action

Delete created function:

funcDelete(<function name>)

Return true/false whether the function exists:

funcExists(<function name>)

Return list of function names as array and count (version 17):

funcList

Reset – delete all functions

funcReset

Return:

Return null if error occur or it will return true in retval.

Examples:

Assuming we want to use calculator to add 2 single-digit values:

>> device.sendAai({action:"function(add(1,2))", add:"click(T:%1);click(T:+);click(T:%2);click(T:=);getText(R:.editText_result)"})
{retval: true}

This accepts 2 substitutions %1 and %2, the default value of %1 is "1", the default value of %2 is "2".

The following will substitute first argument value with %1 ("3"), second argument with %2 ("4"), thus click "3", "+", "4", "=".

Since function return the last command return value, which is getText:

>> device.sendAai({action:"add(3,4)"})
{retval: '=7'}

To bring back old behavior, use "getFuncRetval()":

>> device.sendAai({action:"function(add(1,2))", add:"click(T:%1);click(T:+);click(T:%2);click(T:=);getText(R:.editText_result);getFuncRetval()"})
{retval: true}
>> device.sendAai({action:"add(3,4)"})
{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: '=7'}]}

The following does not specify argument, both default values will be used, thus click "1", "+", "2", "=".

>> device.sendAai({action:"add"})
{retval: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: '=3'}]}

One argument is specified, %1 will be "9" and %2 will be the default value of "2", click "9", "+", "2", "=".

>> device.sendAai({action:"add(9)"})
{retvak: [{retval: true},{retval: true},{retval: true},{retval: true},{retval: '=11'}]}

Check for function existence and delete a function.

>> device.sendAai({action:"funcExists(add)"})
{retval: true}
>> device.sendAai({action:"funcDelete(add)"})
{retval: true}
>> device.sendAai({action:"funcExists(add)"})
{retval: false}
>> device.sendAai({action:"add(9)"})
null
>> lastError()
Invalid action: add

Another example: This function "sendText" is used to send text in Skype, enter text in text field and click send button:

>> device.sendAai({action:"function(sendText(Hello))", sendText:[
    "newQuery(TP:textInput)", 
    "setText(%1)",
    "click(OX:2)",
    "getFuncRetval"
]})
{retval: true}
>> device.sendAai({action:"sendText(How are you?)"})
{sendText: [{count: 1},{retval: true},{retval: true}]}
>> var input = "Good morning"
>> device.sendAai({action:`sendText(${input})`})
{sendText: [{count: 1},{retval: true},{retval: true}]}

The function below will find a name, click to chat screen, send a text using "sendText" function just created, click back key to go back to main screen:

>> device.sendAai({action:"function(findAndSend)", findAndSend:"scrollIntoView(T:%1);click;sendText(%2);sendKey(Back);getFuncRetval"})
{retval: true}
>> device.sendAai({action:"findAndSend(John Doe, Nice to meet you)"})
{findAndSend: [{retval: true},{retval: true},{sendText: [{count: 1},{retval: true},{retval: true}]},{retval: true}]}

Another example: To obtain the SolarEdge production from mySolarEdge application:

>> device.sendAai({actions:"function(getProduction)", getProduction:[
    "openApp(mySolarEdge)", 
    "waitQuery(T:Lifetime,20000)", 
    "newQuery(R:.pv_power_now)", 
    "getText",
    "sendKey(Back)", 
	"sendKey(Back)", 
    "getFuncRetval(getText)"]});
{retval: true}
>> var info = device.sendAai({action:"getProduction"})
>> info
{retval: [{retval: '5.89 kW
Solar Power Now'}]}

To obtain the production, use info.getProduction[3].retval, SolarEdge will return W or kW depending on the amount of output, can write a simple function to get solar production:

function getProduction() {
    var p = device.sendAai({action:"getProduction"});
    if (p != null) {
        var prodText = p.retval[3].retval;
        var matches = /^(\d+(\.?\d+)?) (W|kW)/.exec(prodText);
        if (matches != null) {
            return matches[3] == 'kW' ? Math.round(matches[1]*1000) : matches[1];
        }
    }
    return -1;
}

>> getProduction()
5890

installApp and uninstallApp

This command sends a request to Total Control on Windows to execute adb for app installation or uninstallation. To install an app, provide the Windows path to the APK file; to uninstall an app, provide the package name. The command monitors package-list changes on the device to confirm whether the app was installed or removed, and if no relevant change is detected before the timeout, the command fails and returns null. The default timeout is 10 seconds for installation and 3 seconds for uninstallation; you can override the timeout (in milliseconds) by providing it as the second argument.

Usage:

installApp(<path to Windows APK file>[, <timeout>])
uninstallApp(<package name>[, <timeout>])

Return:

If installApp is successful, it will return the name of the install app. uninstallApp will return true if app is successfully uninstalled. If failure within timeout, it will return null.

Example:

>> device.sendAai({action:"installApp(/tmp/APK Files/SigmaTestApp.apk)"})
{retval: 'com.sigma_rt.sigmatestapp'}
>> device.sendAai({action:"uninstallApp(com.sigma_rt.sigmatestapp)"})
{retval: true}
>> device.sendAai({action:"installApp(/tmp/APK Files/NoSuchFile.apk)"})
null
>> lastError()
Timeout waiting for any package change
>> device.sendAai({action:"uninstallApp(com.sigma_rt.notthere)"})
null
>> lastError()
Timeout waiting for uninstall: com.sigma_rt.notthere

listAndroidSettings

This command returns the list of settings as an array.

Usage:

listAndroidSettings

Return:

List of valid settings in array.

Example:

>> device.sendAai({action:"listAndroidSetting"})
{list: ['SETTINGS','ACCESSIBILITY_SETTINGS','ACTION_CONDITION_PROVIDER_SETTINGS', ..., 'ZEN_MODE_PRIORITY_SETTINGS']}
>> device.sendAai({action:"listAndroidSetting"}).list.length
70

openApp/restartApp /closeApp

The specified app name can be a package name or the application display name (name show on the launcher), if multiple applications with the same name is matched, the first matching application will be used. No partial match is allowed, the match is case insensitive. "closeApp" with no argument will close the existing application. "openApp" and "restartApp" will accept optional query and timeout, after the application is launched, it will enter into "waitQuery" to wait for the query to match before return the control back to the caller. The timeout is in milliseconds.

Work has been done to ensure the openApp or restartApp will return the control when app stop in stable state (rendering is done, waiting for user input), try "restartApp" and "openApp" in terminal, if open/restart take a long time, use the associated query and timeout.

"restartApp" = "closeApp" + "openApp"

"restartApp" is useful for automation script since it will always start the app in a known state.

"openApp" will run the application, if the application is resided on the memory, it will bring the app to the front, if the app is not in the memory, it will act as start the application anew. "closeApp" will close the application (force-close).

Usage:

openApp(<name>[,<query>[,<timeout>]])

restartApp(<name>[,<query>[,<timeout>]])

closeApp(<name>)

closeApp

Return:

retval: true for success or null with lastError() when application is not found, fail to start app or query does not match.

Example:

with time measurement on (in ms)

>> device.sendAai({action:"openApp(Skype)"})
{retval: true, timeFindNode: 3040}
>> device.sendAai({action:"openApp(Skype)"})
{retval: true, timeFindNode: 672}
>> device.sendAai({action:"restartApp(Skype)"})
{retval: true, timeFindNode: 2859}

"mySolarEdge" will always take a long time to open application, use query and timeout:

>> device.sendAai({action:"openApp(mySolarEdge,T:Lifetime,15000)"})
{retval: true, timeFindNode: 13098}

openAndroidSetting

Will open the android system setting window based the setting name specified, the setting name will be converted to uppercase letter, " " will be converted to "_" and name will add "_SETTINGS" to suffix if it is not already there:

Provide part of the setting in https://developer.android.com/reference/android/provider/Settings, use "listAndroidSetting" for list of settings available in this command. This command will open the setting window. If the setting is not found, it will return null.

Usage:

openAndroidSetting(<Setting name>)

Example:

For instance: The following command open wireless setting:

device.sendAai({action:"openAndroidSetting(wireless)"})

The following open Apps setting:

>> device.sendAai({action:"openAndroidSetting(manage applications)"})
{retval: true}

"MANAGE_APPLICATIONS_SETTINGS", "manage applications setting ", "Manage Application" are the same.

return

"return" is an important function that serves 2 purposes:

Stop running in current function.
Return the value specified.

Return can only be effective inside function, outside of function, it behaves like "echo". By default, the function will return the return value of the last command in the function, "return" along with "getFuncRetval" can change that. return acceptTP:appNavBars one argument, if no argument is specified, it will return true. "return(null)" is allowed. Prior version 17, the function execution will return the return values of each commands running in the function. To bring back the same functionality, use "return(getFuncRetval())".

Usage:

return[<CondExp>] // return true if no argument is specified.

Examples:

>> device.sendAai({action:"if(exists(+BP:editable), return(getText), return(null))"})
{retval: 'Hello'}

Return has no effect to main action, only inside function:

>> device.sendAai({action:"echo(1);echo(2);return;echo(3)"})
{count: 4, list: [{retval: 1},{retval: 2},{retval: true},{retval: 3}]}

Demonstrate it can stop in the middle of function execution:

>> device.sendAai({action:"function(testecho)", testecho:"echo(1);echo(2);return(getFuncRetval());echo(3);return(getFuncRetval())"})
{retval: true} 
>> device.sendAai({action:"testecho"})
{retval: [{retval: 1},{retval: 2}]}

Data Type	Description	Common Use Cases
Text	Plain, unformatted text. Used for copying simple strings like messages or labels.	Copying simple strings like messages or labels.
HTML	Formatted text using HTML tags. Ideal for copying rich content with formatting, like web pages or emails.	Copying rich content from web pages or emails.
URI	A reference to a resource (file, image, web link). Useful for copying links to content or files.	Copying references to files, images, or content.
Intent	A messaging object to trigger actions (e.g., open a webpage or start an app).	Copying actions like opening a webpage or starting an app.

Action Name	OQ	Arguments	Input	Output	ML Change

Get Commands
getAllBoolProps	Yes	-	One Node	retval:[<prop>]	-
getBounds	Yes	-	All Nodes	ids:[<id>], bounds:[<bounds>]	-
getBoolProp	Yes	-	First Node	retval:<prop>	-
getBoolPropInAncestors	Yes	-	First Node	retval:<prop>	-
getChecked	Yes	-	One Node All Nodes	retval:<boolean> retval:[boolean\|N/A]	-
getCount	Yes	-	All Nodes	retval:<count>	-
getDescription	Yes	-	One Node All Nodes	retval:<text> retval:[<text>]	-
getHintText	Yes	-	One Node All Nodes	retval:<text> retval:[<text>]	-
getFocus	No	-	None	retval:<ID>	-
getIds	Yes	-	All Nodes	retval:[<ID>]	-
getNodes	Yes	<field list>	All Nodes	retval:[<info>]	-
getProgress	Yes	-	First Node	retval:<rangeInfo>	-
getQuery	No	[SkipText] [SkipDesc]	First Node	retval:<query String>	-
getText	Yes	-	One Node All Nodes	retval:<text> retval:[<text>]	-
getTextFormats	Yes	-	One Node All Nodes	retval:<text>[, html:<html>] retval:[<text>[, html:<html>]	-
getUniqQuery	No	[SkipText] [SkipDesc]	First Node	retval:<query String>	-

Query Commands
addQuery	No	<query>	All Nodes	retval:<count>	Yes
exists	Yes	<query>[, <timeout>]	All Nodes	retval:<boolean>	-
intersectX	Yes	-	First Node	retval:<count>	Yes
intersectY	Yes	-	First Node	retval:<count>	Yes
newQuery	No	<query>[, <timeout>]	All Nodes	retval:<count>	Yes
load	No	[<name>]	None	retval:<boolean>	Yes
pop	No	-	None	retval:<boolean>	Yes
save	No	[<name>]	All Nodes	retval:<boolean>	No
push/save	No	-	All Nodes	retval:<boolean>	No
reduceNodes [TP:reduced] [RN]	No	-	All Nodes	retval:<count>	Yes
refreshQuery	No	-	All Nodes	retval:<boolean>	Yes
sort [ST]	No	<type>	All Nodes	retval:<boolean>	Yes
viewGroup	Yes	[level]	First Node	retval:<count>	Yes
waitQuery	No	<query>[, <timeout>]	None	retval:<boolean>	-

Action Commands on Nodes
assert	No	<CondExp>	None	retval:<boolean>	-
click/click2/clickForNewWindow/clickDown/clickUp	Yes	-	First Node	retval:<boolean>	-
echo	No	<CondExp>	First Node	retval:<value>	-
filter	No	<CondExp>	All Nodes	retval:<boolean>	-
forEach	No	<action\|JSON name>	All Nodes	retval:<name>:[ […] ]	-
if	No	<CondExp>,<then>,<else>	First Node	retval:<value>	-
log	No	<CondExp>	First Node	retval:<boolean>	-
longClick	Yes	-	First Node	retval:<boolean>	-
nsClick	Yes	[<x>, <y>]	First Node	retval:<boolean>	-
refresh	Yes	-	All Nodes	retval:<boolean>	-
repeat	No	<count, action\|JSON>	All Nodes	retval:<name>:[ […] ]	-
saveImage	Yes	-	All Nodes	retval:<text>	-
saveImageInfo	No	-	None	retval:<boolean>	-
saveImageSettings	No	-	None	retval:<boolean>	-
scrollIntoView	Yes	<query>[<direction>], [<num of pages>]	None	retval:<boolean>	Yes
setChecked	Yes	<boolean>	All Nodes	retval:<boolean>	-
setProgress	Yes	<value>	First Node	retval:<boolean>	-
setText	Yes	<text>	First Node	retval:<boolean>	-
showOnScreen	Yes	-	First Node	retval:<boolean>	-
swipeAcross	Yes	<direction>[,<steps>]	First Node	retval:<boolean>	-
until	Yes	<option> <arg>	None	retval:<boolean>	-

Device Commands
getActivity	No	-	None	retval:[<activity>]	-
openActivity	No	<activity>	None	retva:<boolean>	-
getCurrentPackageName	No	-	None	retval:<PackageName>	-
getOnScreenPackageNames	No	-	None	retval:[<pname>]	-
getSystemProp	No	<substring>	None	retval:[{<k>:<v>},…] retval: <value>	-
getDeviceName	No	-	None	retval:<text>	-
exec	No	<command>	<command>	None	retval:<text>
listAndroidSettings	No	-	None	retval:[<settings>]; count:<count>	-
openAndroidSetting	No	<setting>	None	retval:<boolean>	-
Apps Commands
installApp	No	<path to APK file>	None	retval:<installed package name>	-
uninstallApp	No	<package name>	None	retval:<boolean>	-
openApp	No	(<name>[,<query> [,<timeout>]])	None	retval:<boolean>	-
closeApp	No	[<pname>]	None	retval:<boolean>	-
restartApp	No	(<name>[,<query> [,<timeout>]])	None	retval:<boolean>	-
getInstalledApps	No	-	None	retval:[{app}]	-
Function Commands
function	No	<name([default arg])>	None	retval:<boolean>	-
funcExists	No	<name>	None	retval:<boolean>	-
funcDelete	No	<name>	None	retval:<boolean>	-
funcList	No	-	None	retval:[<names>]; count:<count>	-
funcReset	No	-	None	retval:<boolean>	-
getFuncRetval	No	-	None	retval:[] or retval: {}	-
return	No	<CondExp>	None	retval:<value>	-
Clipboard Commands
getClipData	No	-	None	retval:<data>, type:<type> [, html:<html>]	-
getClipText	No	-	None	retval:<text>	-
setClipData	No	<data>[, <type>]	None	retval:<boolean>	-
sendClipData	No	-	None	retval:<boolean>	-
clearClipData	No	-	None	retval:<boolean>	-
General Commands
getLastError	No	-	None	retval:<error message>	-
clearLastError	No	-	None	retval:<boolean>	-
error	No	<message>	None	null	-
setConfig	No	<name> <value(*)>	None	retval:<boolean>	-
getConfig	No	-	None	retval:<value>	-
unsetConfig	No	<name(S)>	None	retval:<boolean>	-
saveImageInfo	No	-	None	retval:<JSON>	-
saveImageSettings	No	<JSON>	None	retval:<boolean>	-
sendKey	No	<code> [meta]	None	retval:<boolean>	-
sleep	No	<duration in ms>	None	retval:<boolean>	-
version/getVersion	No	-	None	retval:<version number>	-

CondExp - Conditional Expression
device/devices.sendAai({query:<query>, action:<command>})

Name	Type	Default	Description
text:maxQueryLength	Integer	30	Length of text and description limit to be replaced with "*"
navi:maxPageScroll	Integer	30	Maximum number of page swipes to find a node
navi:scrollVertSteps	Integer	40	Number of steps on page swipes, low value is fast and incurrate, higher value is slow and accurate
navi:slowScrollSteps	Integer	100	Number of swipe steps to make node fully visible
sigmaInput:backKeyAutoDismiss	Boolean	true	Sigma input method, send extra back key to dismiss invisble keyboard
selector:defaultTemplate	String	more	The default template if template is not specified in query
selector:allowBqAfterEq	Boolean	false	If set to true, will allow basic query that defined after expanded query will be carried out as part of expanded query in sequence (left to right)
selector:allowDelayedSearch	Boolean	true	Enable dalayed search, search when it is needed
action:repeat:maxCount	Integer	20	Increase the number of repeat count
action:enhancedParser	Boolean	true	Enable parser to support escape characters
action:oldReturnKey	Boolean	false	Since version 18, all values in get commands will return as "retval", setting this to true will revert back to previous return keys.