Overview

Scripting

One of the biggest strengths in Total Control is the scripting capability. Total Control offers rich set of API to control one or multiple Android devices. It offers 2 sets of APIs: JavaScript and REST API. JavaScript will run the script locally, REST API offers flexibility in language and hosts. You can also use REST API to control multiple PCs running "Total Control" application.

JavaScript API

Language

Total Control provide the following framework for JavaScript (ECMAScript 5 with selected ES6 features)

Mozilla's Rhino 1.7.7 with Total Control 7 (or below), Rhino 1.7.12 with Total Control 8 and Rhino 1.7.14 with Total Control 9 (Update 20). See https://github.com/mozilla/rhino for more information.
RingoJS 2.0.

The bonus with Rhino is the ability to call Java API (Total Control 8 is using OpenJDK 15) directly if Rhino + RingoJS do not provide sufficient functionality.

Total Control offers many classes, several examples:

Device
DeviceArray
UiElement (Total Control 8)
UiElementArray (Total Control 8)
Notification
Keyboard
Excel

Device and DeviceArray Classes

When a device is connected (auto connect or "Connect" button is pressed), a device object from "Device" class is created, use static method "searchObject" to locate the device objects. For multiple device objects, it offers "DeviceArray" class (subclass from Array). Devices in DeviceArray objects will be perform the same task at the same time.

Device class offers close to 100 methods and properties to manipulate a device. DeviceArray class offers about 20+ commonly used methods. Please refer to {{JavaScript API documentation}} for all methods. You can extend Device and DeviceArray with ease, (See Extending Device and DeviceArray).

Use "Device.searchObject()" to locate created device objects, the device object format is "device@<10 digits>".

// Return one device object
var mySamsung = Device.searchObject('Samsung-S9');
// Return all device objects, DeviceArray
var allDevices = Device.searchObject(tcConst.DevAll);
// Return device objects belong to same group, DeviceArray
var allGroupX = Device.searchObject(tcConst.DevGroup, 'first row');
var allGroupY = Device.searchObject(tcConst.DevGroup, 'second row');
// Combine 2 groups to form a larger group, DeviceArray
var groupXY = allGroupX.concat(allGroupY);

Since DeviceArray() is a subclass of Array(), it inherits most of the methods from Array class, since all the TC specific methods are tied to DeviceArray, if you have an array, use "var ary = new DeviceArray().concat(ary)" to convert to DeviceArray.

// This will fail since there is no "click" method in Array
var device = Device.getMain();
var ary = [device];
ary.click(100,100);

// This will work
var device = Device.getMain();
var ary = new DeviceArray(device);
ary.click(100, 100);    // or ary.clickSync("OK");

Couple of examples:

// click location 100,200 on device name "Samsung-S10".
var device = Device.searchObject('Samsung-S10');
if (device) {
    device.click(100,200); 	// or device.clickSync("Start");
}

// click in the middle of the screen on all connected devices.
var devices = Device.searchObject(tcConst.DevAll);
if (devices) {
    devices.click(0.5, 0.5); 	// or device.clickSync("John");
}

Directory and Userlib.js

By default, the scripting directory is located at \Users\<user name>\Documents\Scripts directory, you can change it by clicking "Script" in Main Window, select "Script List", top line to change the directory.

Within it, you can create a file called "Userlib.js", this file will always be loaded before script execution, you can include commonly used functions, add prototypes to existing classes or bring in 3rd party software.

To debug "Userlib.js", use "Terminal", to reload "Userlib.js", click reload icon on the bottom right Terminal window.

Absolute vs Relative Coordinates

For x, y coordinates, TC offers "absolute" and "relative" coordinates (the system settings show coordinates will show both coordinates), absolute coordinates from (0, 0) to (width – 1, height – 1), relative coordinates are usually 4 decimal points from (0, 0) to (0.9999, 0.9999). The relative coordinates multiply by device width and height will get the absolute coordinates. Coordinates are not ideal, Professional supports AAI that use text in the UI elements to retrieve the coordinates in the runtime.

Extending Device and DeviceArray

The added bonus of JavaScript is the extensibility, you can extend the methods by adding prototype to the classes. Assuming you want to create a method "longPress" for both single and multiple devices, one way to write it is:

Device.prototype.longPress = function(x, y) {
    var retval = this.click(x, y, tcConst.STATE_DOWN);
    if (retval != 0) {
        return retval;
    }
    delay(500);
    return this.click(x, y, tcConst.STATE_UP);
}

DeviceArray.prototype.longPress = function(x, y) {
    for (let i = 0; i < this.length; ++i) {
        retval = this[i].longPress(x,y);
        if (retval != 0) {
            return retval;
        }
}
return 0;
}

This implementation not efficient for large number of devices in DeviceArray, the method can take up to 50 seconds if there are 100 devices in the array.

The best way is to send STATE_DOWN to all devices first, then wait till the remaining of 500 ms runs out. Fortunately, Device and DeviceArray (multiple devices) introduced "sendAll", it will execute the method specified and wait until time specified is expired. This way, the execution time is almost 500 ms. Since "sendAll" is available in Device and DeviceArray, the code to implement Device and DeviceArray is exactly the same.

Device.prototype.longPress = function(x, y) {
var retval = this.sendAll(Device.prototype.click, 
     [x, y, tcConst.STATE_DOWN], 500);
    if (retval != 0) {
        return retval;
    }
    return this.sendAll(Device.prototype.click, [x, y, tcConst.STATE_UP]);
}

DeviceArray.prototype.longPress = Device.prototype.longPress;

sendAll only applies to methods that returns 0 on success.

Scripting Tools

There are many tools available in Total Control, click "Script" will open a new window:

Terminal: Open Rhino + RingoJS command prompt, you can use it for testing or development purpose. Click button on the bottom right to reload the interpreter and Userlib.js. Can use load("filename.js") to run a script.

Script List ⇒ Path: Script default path, click icon to change the path to other directory.

Script List ⇒ JS source files: Use to quickly execute a JavaScript file, it provide a simple editor for quick change or click arrow key to execute the script.

Script List ⇒ Record Script: Record the actions on the devices into an Excel file, Excel file can also be used to generate JS script or JSON file (maybe useful for REST API). Users with no experience in JavaScript can use this tool to generate simple script.

Script List ⇒ Image Helper. Generate BMP file from main device screen content, the image is needed for seekImage(), it extends BMP file to provide additional information such as app name, activity, width and height information. seekImage() will utilize these information without providing lots of parameters.

Script List ⇒ Color Helper. Load the screen content from the main device (like image helper), a magnified color chooser to pick RGB values of the colors and generate seekColor() with complex parameters. Can support single and multiple colors.

Script List ⇒ UI Explorer. Open UI Explorer window, click capture to capture the existing main phone window and allow users to experience query language to select UI elements. A Helper window will provide helper description for each keys in query language.

Execute (will be renamed to Task). Create a task to run a script on various number of conditions such as date & time, number of iterations, regular intervals or the devices. The output and the results of the task execution will be saved for review. Task can be also be created or changed via JS API. Runner execution will be shown as a task here.

Inspect: Inspect the various internal variables related to scripts and engines.

Automation and Accessibility Integration (AAI) (Total Control 8+)

AAI identifies UI elements on screen as objects, traditional way of (x, y) coordinates take screen as one giant object, so seek image/color and OCR are required to identify an object on screen.

Accessibility is a feature that represent UI elements on screen as underlying nodes, a node includes many properties such as text/description, dimension, boolean properties such as clickable, editable or scrollable, underlying class name, etc. The text/description can be accessed easily (OCR is not needed), the dimension (and clickable) ensures the button can be clicked on certain location even the node is moved to another location.

A node can represent an UI element (e.g. button) or a group of UI elements or layout of certain elements. A node (element, group or layout) can be identified by a node ID (represent in string of hex). We integrate Accessibility, TC scripting framework and UI Automator library to achieve the following goals:

Coordinate independent makes the script more portable with different resolutions, multiple sizes and brands.
Synchronous API will wait until the screen is repaint, make the script simpler, do not need to guess the time to sleep.
Can retrieve the string from the app with ease, instead of using error-prone OCR.

The simplest case of AAI:

Click "OK" on the screen if found, far better than click(100, 100) for specific resolution: devices.clickSync("OK")

Enter text into text entry, AAI can find all text entry lines in current screen.

devices.inputTextSync([position], "text")    // Enter text, the position is used for multiple inputs

Run or restart application, without query, it will return on screen refresh; with query, it will match the query after screen is refreshed.
```
devices.runAppSync(<package name>, [query]) 
devices.restartAppSync(<package name>, [query])
```

The entire screen is composed by many nodes, a node can be a smallest UI element or a container of many nodes, some nodes are invisible. The entire screen is a tree structure starting from the single root node. Depending on the complexity of App, a screen can contain 50-300 nodes.

Since the users are only interested in small subset of nodes, the challenge is finding the correct nodes users want and extract information or perform actions on them.

The challenge is how to find nodes? We invented a query language to find the nodes, the FindNode program is installed in every device, the query language will be performed to obtain the nodes that meet the criteria, the intent is to reduce the large number of nodes to one or few intended nodes, users can obtain information or apply actions to the nodes.

For example: UI Automator in Java provides "UiSelector" and "BySelector" in UiDevice.findObject() or findObjects() to locate nodes, it can be complex for multiple conditions:

new UiSelector().className("android.widget.TextView").text("OK")

We created a simple query language, that is shorter and portable since the query will be send to many devices, the above code can be rewritten in our query language as:

"C: android.widget.TextView&&T:OK"

AAI project includes the following:

Query language, simple one line syntax language to search for intended nodes. Core of the AAI.
FindNode carry out the query or actions on each device. All the query and certain actions are done in FindNode, it contains few dozen commands. See FindNode documentation for more information.
Object mode in one-to-many synchronization, send the node (or UI object) to all devices instead of coordination, click "OK" can run on all devices with different resolutions than click(100,100).
UI Explorer to obtain node information, can visually test the query language, learning and exploration tool.
AAIS, a simple language to perform automation on multiple devices. Capture and replay generate this language, see AAIS documentation for more information.
REST and JS API includes accessibility to FindNode.
UiElement class on top of FindNode to access node with ease.

Query

Each query has one or multiple "<key>:<value>" pair, multiple keys can be added with "&&" as separator.

Each node is identified by a node ID. A query can divided into 3 phases:

Template ("TP"). This class "generates" the initiate nodes. For instance "TP:textInput" will return a list of editable text fields. This class is mandatory, if not specified, default template will be used.
Basic query (BQ). Each node contains information about itself, the classes, the text/description, the properties, etc. BQ will match node one at a time, will reject the nodes that do not meet the criterion, will not be passed into next phase. If BQ is not specified, the nodes generate from TP will be passed into EQ.
Expanded query (EQ). A set of keys that usually work with multiple nodes. Multiple EQ execute from left to right, same key can be specified multiple times. An example of EQ: "OX:1" find the element/node on the right of the current node.

After the query is performed, one/more nodes found are listed in "ML" (matching list), list of actions can be applied on the ML, the actions can be retrieving information or perform action on ML.

Template:

Generate the initial nodes for BQ or EQ.

TP:all All nodes

TP:more All nodes except nodes end with "Layout"

TP:basic All leaf node (child count is zero)

TP:reduced Optimize "TP:more" to return nodes that is important for screen

TP:anyText[,<min>[,<max>]] Nodes with content in "text" in certain length.

TP:anyDescription[,<min>[,<max>]] Nodes with content in "description" in certain length.

TP:textInput All "editable" fields that sorted from top-left to bottom-right.

TP:findText,<text> Nodes with the text in the arguments, can contain "*" and "/…/".

TP:line,top|bottom,<number> Return top/bottom nodes that is outside of scrollable nodes.

TP:scrollable,<position> Nodes inside scrollable container, position for multiple scrollable nodes.

Basic Query (BQ):

Query for node-level information obtained, each nodes from TP will match (if provided) in BQ to proceed.

P:<package name> -Should not be used, default to running app
C:<class name> Class name (S)
R:<resource ID> Resource ID (S)
D:<text> Description (S)
T:<text> Text (S)
IT:<number> Text input type (I)
CC:<number> Child count (I)
ID:<ID> Node ID in hex (S)
BI:[x, y] The nodes contain (x,y)
BI:[x1, y1, x2, y2] The nodes enclosed by the rectangle, if x or y is -1, it will be ignored
BP:<prop name> Boolean properties (S).
- checkable, checked, clickable, editable, enabled, focused, longClickable
- scrollable, visibleToUser.
TD:<text> Match text or description (S).

Expanded Query (EQ):

Queries here usually across multiple nodes.

The order of Expanded Query is important, all expanded queries execute from left to right. Commands with the same keys are allowed.

IX:<number> Obtain the one node from list of matching nodes based on position
OX:<number> Offset to neighbor nodes horizontally (positive – right, negative – left)
OY:<number> Offset to neighbor nodes vertically (positive – down, negative – up)
ON:<type> Different ways to pick one node out of list of matching nodes
- first, last, min
ST:<sort type> Return sorted nodes based on the position of the nodes on screen
- x, y, yx (or all)
TX Return nodes that intersect with reference node horizontally.
TY Return nodes that Intersect with reference node vertically.
VG:[level number] Return the group of nodes in a view group from first node in ML.
RN Return the optimized nodes from a list of matching nodes.
BQ:<query> Perform basic query.
X:<key in BQ> Basic query key prefixed with "X"

For BQ, the query syntax can contain "!" for not, ">", "<" for greater than or less than, "*" for wild card match and "/<regexp>/" for regular expression. It can match package name, class name, resource ID, text, description, child count and input type.

FindNode is installed into every device (part of Total Control App), it is the only program that recognize the query syntax, it parses query, locate nodes and perform actions to the nodes found. FindNode offload the complexity of JavaScript and CPU utilization of Total Control, all the search is conducted in the devices.

device.sendAAi() and devices.sendAai() are direct way to communicate with FindNode with one or list of devices. The JS object will be translated to JSON before sending to device, the returned value is in JS object format. If error is encountered, the return is null, the lastError() contains error message.

A simple query example, to obtain the text of Model name, use X offset of 1 (next to the right):

>> device.sendAai({query:"T:Model name&&OX:1", action:"getText"})
{retval: 'Galaxy S10+'}

FindNode can even detects the fixed icons on the top/bottom of the screen:

>> device.sendAai({query:"TP:line,bottom,-1", action:"getText"})
{retval: ['Chats','Calls','Contacts','Notifications']}

The following 3 commands, doing the same thing, click on the "Calls" text:

>> device.sendAai({query:"TP:line,bottom,-1&&T:Calls", action:"click"})
{retval: true}
>> device.sendAai({query:"TP:line,bottom,-1&&IX:1", action:"click"})
{retval: true}
>> device.sendAai({query:"TP:line,bottom,-1&&T:Chats&&OX:1", action:"click"})
{retval: true}

Click "Contacts" icon:

>> device.sendAai({query:"TP:line,bottom,-1&&T:Contacts&&OY:-1", action:"click"})
{retval: true}
>> device.sendAai({query:"TP:line,bottom,-1&&IX:2", action:"click"})
{retval: true}
// Find multiple "Contacts" are found on the screen, IX:-1 is to select the 
// last node found 
>> device.sendAai({query:"T:Contacts&&IX:-1&&OY:-1", action:"click"})
{retval: true}

Please read FindNode User Guide for complete information.

View all