Overview

Total Control is a large system consists of many components/modules and functionalities:

• Total Control application in Windows (TCW), main application of the system.
• Total Control application for Android (TCA). Can be downloaded in Google Play Store or bundled with the downloaded package, Total Control will install automatically.
• Total Control Viewer (TCV). With the purchase of optional "Device Control", it allows Android device to control part of the features in TCW, can access the screen of the main device.

There are many applications within the TCW:

• Connections. Establish the connectivities (and permissions) between TCW and TCA in one or mode Android devices.
• Control app. There are 2 ways to control connected Android devices. WDM – Windows Desktop Mode shows one Android device per window. MDCC – Multi-Device Control Center is a single big screen to access and control multiple devices.
• Coordinates vs objects. Click (x, y) vs a button with a text of OK. Coordinates are useful to control multiple devices with the same model & type. Objects are useful for devices with multiple resolutions and dimensions. Object model is recommended, easier to script when applications under tested are changed, obtaining the text is faster and accurately than OCR.
• Scripts. Provides JavaScript and REST API allowing users to control and access multiple devices both coordinate and object models. In-house developed AAIS is useful to generate scripts with ease, seamless integration with JavaScript and can controls multiple devices with single script.
• AAI. Umbrella name for all object models, it includes query language, UI Explorer, FindNode and AAIS.

Once the Total Control is installed, for every Android device, the first task is to establish a USB connection, the process is cumbersome but also is very important in 2 aspects:

• Grant adb/shell permission over USB so we can control the device.
• Install necessary software into the device.

In order obtain adb/shell permission, "USB Debugging" in "Developer Options" needs to be turned on, once it is turned on with the correct adb device driver, Total Control will be able to access the device and use adb to install "Total Control" App and NDK software into the device. Windows 10 can recognize most of the Android devices, Windows 7 and XP will need adb device driver, we provided a universal adb device driver that should work on majority of the devices, if it does not work, may need to visit device manufacturer Website to download adb device driver.

Total Control automatically detects the presence of USB devices and ask for confirmation to connect, it offers instructions on how to bring up "Developer Options" and turn on "USB Debugging" for various device models, during the setup process, the device (not PC) may popup a message to confirm the connection. Establish the connection via USB is the hardest part, this is a one-time setup, as soon as "Connect" button appears, you have everything installed correctly. Click "Connect" button, you will likely see a device window, go thru on-screen instructions to test if it works.

After the USB connection is established, the software has been granted the permission, you can now continue to use it or unplug USB and establish connection via WiFi or Ethernet.

TCP Connection

Once you have USB connection to bring up device window, the device has received adb permission, now you can unplug USB and choose to use WiFi or Ethernet (or other networking method with IP address). To add a device via TCP, either use "+ Connect new device" in main window or "No device found, Click here to begin", this will bring you a connection window. There are 2 ways to connects via TCP:

1. Scan QR Code. In Connection Window select TCP, it will bring up QR code, open the device "Total Control" app, select scan, to scan the QR code, if the network is reachable, it will start with initial setup process and it will bring up "Connect" button.
2. Input IP address. On the bottom right corner of "TCP" connection window is "Input IP address", you can click to enter IP address of the device manually, if you do not know the IP address, open "Total Control" app in device and click TCP tab. Ignore the password and click OK. The device will popup a confirmation for TC to connect, click OK to proceed. You can bypass confirmation by setting up a password in "Total Control" App and enter IP and password in connection window. The entered connection information will be saved in the pulldown menu on the top.

Device setting also supports "Auto Connect" for USB and TCP, it will connect automatically and open the device window. It does not offer Ethernet auto connect, WiFi auto connect allows the device to detect the presence of the Access Point, discover and connect to TC automatically.

As long as devices are granted with adb permission, TCP connection can very powerful, it allows TC to connect/control devices in any reachable networks, with VPN, it can connect devices in multiple remote locations. Total Control 8 have added TCP grouping for ease to connect multiple devices via TCP.

Screen Projection

If control is not your concern, you just want screen projection, select "Projection" in the connection window, it will display QR code, same as TCP connection, use "Total Control" app in your device to scan the code, if it is reachable, it will display "Connect" button, click to display. This mode does not allow users to control the device.

There two modes in Total Control, Windows Desktop Mode (WDM) and Multi-Device Control Center (MDCC), MDCC is a completely different beast, we will dedicate a section to describe it later. For simplicity, we will describe WDM in most of the sections.

There are several types and functions of different types of windows:

• Main Window: When Total Control is started, this window will always be present. It consists of several functions:
  • The device names of the connected devices, Lite will display the thumbnail of the device screen content, will update thumbnail regularly. Double click on the device name will bring the device to front. [Pro] For 2 devices or more: it has 3 sync modes: mouse, keyboard and function key, sync mode allows all devices follow the same action as main device. You can also change the device name that is meaningful to TC.
  • If notification is configured, it will show the notification summary at the bottom of device window. Click on it to bring up more detailed information, the icon will reflect number of unread notifications.
  • "Apply All" is seldom used, right now it only support screenshot to all devices.
  • "Script", scripting is a big part of Total Control, this popup a scripting window with many features. See later sections.
  • "MDCC" – Enter MDCC mode, for 4 device or less, advise to use WDM. MDCC is efficient for 10 devices or more.
• Device Window: This is where the mirroring and control take place. You can open as many devices as CPU and network can sustain, Total Control supports (need to pay extra for more than 2 devices) up to 50 devices, WDM is suitable for handful of devices. With many device windows, there is always one "main" device window, click the mouse cursor on another device window to switch the "main" device window. var device = Device.getMain().
• Connection Window, clicking main window "+" will bring up Connection Window, the connection window will also popup automatically when USB device is connected. Click "Connect new device" this is where you can connect via USB, TCP and QR Code to project device screen.

The device window will be opened either automatically (auto connect) or click "Connect" button in Connection Window. The device window consists of several components:

• Device screen content
• Optional hardware keys (recent app, home, back and menu) for device that has the hardware keys.
• Top row has a menu button menu button and 4 quick keys (can be configured in system settings).
• Title bar has 4 keys:
• Setting: Enter device settings.
• "i" information: Display information about the device, its name, resolution, etc.
• Always on top.
• Help.
• Optionally, it will display the coordinate of the mouse relative to the device (system settings ⇒ show coordinates), it is useful to locate coordinate of certain controls.
• If the device headphone jack is plugged with THA (our hardware device), it will show information here. THA will loopback device audio to microphone, TC will capture the sound from mic and send to PC. Currently, THA is not sell internationally.
• Return key from setting or back key (will be obsoleted).
• Icon with 4 small rectangles for debugging purpose.
• Can resize the device window by dragging the border of the device window, the position and size will be saved, if the same device is opened in the future, the window will be restored back to the same position and size.

Click 3 horizontal in the device window will bring up a menu of features:

• Screenshot to PC/Device and Record Video. See Screenshot and video recording.
• [Pro] Capture PC. See Copy-n-paste and file transfer.
• Zoom in/out and rotate left/right. Rotate if the landscape/portrait auto-detect does not work.
• Full screen. Will fill the device in entire screen, since Escape key is used as a "Back" key, to restore the screen, click the last button in the menu. If you don't see the menu, click the arrow on the left edge of the screen, it will show the menu.
• Max Height. Sacrifice the menu title and extend vertically to much as possible.
• Backup and Restore
• File Manager. Provide a rudimentary file manager on the device, useful to drag in/out files between device and PC.
• Pause screen. Will temporarily halt the screen transfer, click menu ">" to proceed (jump to the most recent screen).
• Change Keyboard. Change the device virtual keyboard.
• [Pro] Voice Control. To turn on/off Voice control.
• Device Settings. Bring up the device settings.
• Runner. To run AAIS and JavaScript scripts.

Device settings are used to fine tune how the device behave, it has many options, most of them run well with its default values:

• See screen device settings for various controls in the device settings.
• Control Brightness of the screen of the device ([Pro] can applies to all connected devices), this is temporary, when device is disconnected, it will restore back to its previous setting. Useful to dim the device screen to conserve power and retain privacy.
• Volume controls, can control up to 6 types of volumes ([Pro] apply to all connected devices). This change is permanent.
• Shortkeys. Configure 10 function keys to start application or run a script. So you can configure F1 for Facebook, anytime can you click F1 to bring up Facebook.
• Auto connect. If check will automatically connect and display device window, if you want to use TCP auto connect, you must be already connected with TCP.
• [Adv] Hardware key. If auto detect has problem detects the presence of hardware keys, you can override it in this setting.
• [Adv] Hotspot if enable and configure in an application, it has a red dot on the top menu, drag to clickable location and associate with a hot key (Ctrl + Alt + <letter>), it will record down the location, hot key and application, when application is started, click the hot key will trigger TC to click the location. Multiple hotspots can be configured in an application.
• [Adv] Input mode. We support thousands of Android devices, if the device is unresponsive to actions (mouse or keyboard) on the device screen, change to one of the 4 input modes.

Lots of effort has been put into ensure the efficient of the screen transfer, how to ensure 40-60 fps of transfer without sacrificing high PC's CPU (<10%). The efficient use of GPU on both device and PC is crucial, absence of GPU of any one side will slow down the mirroring significantly. We also have homegrown compression (SFC) to handle devices without GPU. We will perform image compression (H.264 or homegrown), transfer and decompress and render on PC, PC has to handle multiple devices. The image (quality + latency) are highly impacted by many factors: frames per second, data size (resolution, quality) per frame, USB transfer rate, adb transfer rate, CPU/GPU of devices and PC. We offer many parameters to tune.

Resolution

Lite version offers 640p resolution, Pro offers 4 resolutions, we will maintain the aspect ratio, the resize is done on the devices. For example, the device with 4K resolution has very little impact on screen quality on PC but 1080p does. 1080p will put heavy burden on both devices and PC (generate and move more data).

• 480p
• 640p
• 720p
• 1080p

Device Acceleration

There are 3 acceleration methods:

• HA1: This is transfer as fast as possible, this is done on the device. Usually can reach 40-60 frames per second (FPS) with average device and PC.
• HA2: This is rate limited to no more than 30 FPS, more device CPU/GPU is required to manually resize and compress screen image. This mode is useful for recording (will switch to HA2 when recording is turned on), 60 FPS recording will create huge MP4 file size.
• Compatibility mode: This mode is required for the devices SoC that does not offer GPU or H.264 compression. We developed our patented compression technology called SFC, that compression is viable in slow device CPU. Almost all recent SoC has offer GPU (and H.264), this mode is useful for headless devices without GPU. SFC utilizes CPU on the devices and PC, on average this mode can achieve 10-15 fps.

Image Quality

We offer 3 image quality: Low, Medium and High by tuning H.264 bitrate, the quality will not impact on static image (device will transfer high quality static image if image is not refreshed), more apparent in dynamic images.

[Adv] Device coding setting

Tune H.264 profile to achieve higher image quality, Android requires baseline profile, not all devices will support other profiles.

[Adv] PC Acceleration

TC is tuned to utilize Intel i-series CPU/GPU, if you have external graphics board, you can adjust if DirectX or OpenGL gives better performance. Usually, DirectX is good enough.

[Adv] Hardware Acceleration

For HA1 and HA2, it will let GPU to decompression and rendering in one step. If this mode is unchecked, it will do the decompression (GPU) and render as 2 distinct steps (moving lots of data).

[Adv] Native vs Projection

Prior of Android 12 we can do H.264 or SFC compression with our own code, we called it "native", with the introduction of "screen projection" feature in Android, we have added "projection" into Total Control. Android projection has similar speed (if not faster) than our native mode, it is difficult for us to maintain 2 modes for every new releases of Android. Starting Android 12, we will no longer offer "native" mode or SFC.

The system settings are settings that applies to all devices. It is accessible thru Main Window settings or Device settings ⇒ System Settings. It consists of subjects on the left and detail settings on the right, you can scroll on the right click on the title on the left. It contains the following settings:

Sigma keyboard

• Enable Sigma keyboard: Check this setting will always enable Sigma Keyboard automatically, disconnect connection will revert back to original keyboard setting.
• Paste directly to device: This applies to text, you can copy from other Windows applications and press Ctrl-v to paste the text into device directly.
• -Auto send enter key: For social media apps, most settings have Enter key to send. Check this setting will save an Enter. When you enter text in textbox and press Enter, with this option, this will send the text and send Enter automatically. Enter can also send search key if the keyboard is in search mode.
• Autotext input: See "text input", can configure mapping between short and long form of text.

Other options

Screenshot, Video recording – See Screenshot and Video Recording.

Capture PC – See Copy-n-paste and file transfer.

Copy – See Copy-n-paste and file transfer.

Save/Clipboard when Sharing images. See Copy-n-paste and file transfer.

Boss Key – Can define a function key to close all the device windows in Windows system tray.

Screen Control Mode – See Movements, click and swip.

adb/adb2 – adb2 is newer and much faster than adb, if encountered problem on adb2, can switch to adb.

Device Auto Sleep – By default, TC will keep device screen always on, check this option to use device display timeout setting.

Backup – Backup and restore path for contacts, messages and call logs.

Frame per second – Show number of frames per second.

Show Coordinates – Show absolute and relative coordinates of mouse cursor relative to device screen, useful to API that requires x and y coordinates.

Device Display Mode – MDCC or WDM. MDCC will be in different section.

Shortcuts – This allows users to define up to 4 shortcuts that will be display on top of the device windows for ease of access. Most of them are self-explanatory, "Sigma Input" will allow users to select input method.

Notification Management – Can define number of notification to store and the actions when notification is received.

Auto Start Up – It is a small PC software to allow users to start/exit/restart Total Control or if the device with auto connect is plugged to USB port, it will start automatically.

REST API – Define the username and password of REST API, by default, for security reason, the password is the last 8 characters of the TC ID. The "Duration setting" is the time required to send another "Authentication". Port number can be changed if you have port number conflicts on the PC. The rest is accounting information and connection list.

Voice – See Voice Control for more information.

Movement likely is the first thing you will test, we simulate movements via swipe or the scroll, keyboard 4-arrow keys, PgUp, PgDn, Home, End (menu) will move the screen, Shift with 4 arrow keys are long swipe in the middle of the screen, very useful for next page such as swiping thru the pictures or next page on the app cabinet. Shift-PgUp and Shift-PgDn will move to the front and bottom of the document (swipe as fast as possible).

Three keys are important: "Esc", "Tab" and "Enter". "Esc" key is Android's back key "<". Some apps offer Enter key as send, the Enter key will be handy instead of click a button to send. Since there is no cursor, you can use "Tab" and "Enter" to go thru some of the options. Enter key can be used to search by pressing extra Enter key.

Alternatively, mouse can be used to control the actions, the mouse wheel acts as "up" and "down" arrows, left mouse button is use to click or swipe. Right mouse button has 2 features:

• Click to go back.
• Drag to simulate pinch-to-zoom, useful to zoom in/out pictures. With right button is pressed, move away from center of the screen to zoom in, move towards the center to zoom out. When you are done, release the right mouse button. When mouse button is pressed at location (x, y), it will simulate another press at point (width – x, height – y).

In the system option, "screen control mode" you can select "scroll" which use Android Scroll functionality whenever possible (all movement keys except Shift-up/down/left/right).

The way Total Control handle input is to install a keyboard called "Sigma Keyboard" on the device, "Sigma Keyboard" is an invisible keyboard that in the background communicate with Total Control in PC. It is advised to always use Sigma keyboard since it allows users to use the PC's keyboard to enter text. System settings have an option always use Sigma Keyboard. When a text input is required, a textbox will appear on the top of the device window, users can enter text in the textbox via their preferred language and input system offered by Windows. There are 3 input options for users to choose:

• "a": Enter one character at a time, it does not require Sigma Keyboard, every character typed (need English keyboard) will be sent to the device. For instance, if device has been changed to Chinese input method, you can use alphabet and numeric keys to compose Chinese character. Copy-n-paste is not supported in this mode.
• "ab" (version 8 update 20): This allows users to type in any language, a short pause (default: 200 milliseconds) will made the typed text to send to the device. Copy-n-paste is supported, autotext is not supported.
• "abc": Line mode allow you to enter long text and enter key to send to device. This is the default mode allow both autotext and copy-n-paste.

There is another option "a", that is seldom used since the input is letter at a time, usually useful in terminal software. The textbox offers a lot of features:

• If password entry is required, enter the password will display with password mask of dot.
• Support to enter long text string that span multiple lines.
• Text based cut-n-paste to this textbox can span multiple lines without any problem.
• Support shortcut, you can use system settings under "Sigma keyboard", "Autotext input" and configure button to configure the autotext. For instance: "cyl" = "I will call you later", if you enter "cyl", it text will replace it with "I will call you later".
• Enter will search "Search" or "Enter" depending on the device text property, Many Android App offers "Enter to send", TC supports this, enter text, press enter, press enter again to send. Similarly, for search: enter text, press enter twice.

When textbox appears, the navigation keys such as up/down/left/right or PgUp/PgDn applies to textbox, you need to use Escape key to dismiss textbox.

When the USB is disconnected, device TC will restore its keyboard to its previous setting.

If you want to use specific keyboard, select menu "Change Keyboard" option in device window to change the keyboard.

With the support of Sigma Keyboard, you can cut-n-paste between PC and devices, TC also support transfer files:

• Copy-n-paste to the textbox. Ctrl-C in other app and Ctrl-v in textbox.
• Copy-n-paste directly to the device window (if it is in input mode), you need to enable in System Settings "Paste directly to the device" option to avoid paste accidentally.
• Copy the text in device, the device window will show an icon, press the icon to copy the text to Windows clipboard. If you do not want to press icon every time, enable "Clipboard auto sync to PC" in system settings.
• You can copy the one or multiple files from Windows to device by dragging files from Window explorer into device window, the files will be stored in "/sdcard/Total Control/files". If you drag APK file to device window, it will be automatically installed. If you have external SD card, it will prompt you the installation location.
• Open file Manager go to specific path in the device, you can drag in/out files to transfer files between device and PC.
• Select "Capture PC" in device window menu, will prompt users to crop a portion of PC screen, click "OK" to store the file in the device, the file will be stored in "/sdcard/Total Control/screenshots", alternatively, the system settings, under "Capture PC", press combination key ("Ctrl/Alt/Shift + letter") in textbox to set the hotkey to capture. See Screenshot and video recording for more information.
• Another way to transfer a file from device to PC is to use Android "Share" feature. When system setting "Clipboard Auto Sync to PC" is enabled, choose save to Clipboard or File, in the Gallery, click Share, select "Total Control", when image is sent, the image will be available either in clipboard or store in a file or both.

See system settings Screenshot and Video Recording for more options.

There are several ways to do screenshots:

• Screenshot to PC
• Screenshot to Device
• Screenshot to Device Memory (only available in JS/REST API).

Three formats are available: BMP, PNG and JPG. The screenshot is performed on the device (not existing device screen), it will be done in native resolution, if you have high resolution (e.g. 4K) device screen, do not use BMP, it will take a long time to transfer large BMP data to PC (or take a long time to store). Use lossless PNG or JPG (with slider to set quality) to save space.

The screenshot can store in file or clipboard, file is stored in file path, for the clipboard, you can paste on graphics program such as "Paint".

For PC with high resolution screen (2K or above), capture native resolution will result in large file to be transferred to device. "Resize to 1080p" default option will treat screen as 1080p and resize accordingly.

For video recording, when video recording is turned on, a timer will appear, click red rectangle to stop recording. A few notes:

• When in recording, the acceleration mode will be switched to HA2 automatically, to ensure low FPS (limit to maximum of 30 FPS) instead of 40-60 FPS in HA1, this will greatly reduce the size of MP4. After the recording stops, it will revert back to previous mode.
• The video will be split into different segments, the segments are split by duration, when the duration is reached, it will create a new video file, this will limit the video file size. The video filename with segment is "<device name>-<date>-<time>-<segment no>.mp4", the segment number starts from 1.
• Underneath of "Record Video" icon in device window menu item, there is a little arrow, when click, it will display a list of audio sources, microphone, line-in and THA, select the source, it will be mixed in the video.

TC utilized Windows 7 and 10 voice capability allowing users to use voice to open an application or run a function (see Userlib.js). By default, all applications in the device will be registered so you can use "open " to open an application, for instance, "open calendar" to open calendar. If the applications are not English word or difficult to pronounce (especially in game names), you can associate a phrase to an app.

You can also associate a phrase to a function name in Userlib.js. When a word is heard, it will execute a function with device ID as the only parameter. E.g. configure "leave now" to function leaveNow. You can create a function in Userlib.js. function leaveNow(device) {…}

The trigger phrase is "OK TC", a popup will open and ready for command, if the first word is "multiple", users can provide a list of commands, until "stop" is heard or 60 seconds without a usable command. The actions will be carried out in main device.

For security reason, the voice control capability needs to be "turned on" to listen for trigger phrase, turning on one device will turn on for all devices. If you in system settings, you can check "Always enable" to turn on automatically.

Feature	Voice
Trigger Phrase	OK TC
Multiple voice cmd	multiple
Stop multiple	stop
Application	open "application name"
	open <phrase>; phrase = "application name"
Function	run <phrase>; phrase = "function name"

Feature	Key	Voice	Key	Voice
Movement	up arrow	up	page up	page up
	down arrow	down	page down	page down
	left arrow	left	shift-PgUp	go top
	right arrow	right	shift-PgDn	go bottom
Swipe	shift-left	swipe left	shift-up	swipe up
	shift-right	swipe right	shift-down	swipe down
Special function	home	home	escape	back
	end	menu

Total Control provide device-based backup and restore on contacts, text messages and call logs. The backup located in PC can be changed in system settings ⇒ Backup. The backup is stored in JSON format, you can use JSON editor to make changes, erase all data in the device and restore back to device. There are also JavaScript APIs to manipulate contacts and messages. It is difficult to check-in to Google Play Store with these permissions, we have taken this feature out, if you have purchased professional, contact us if you want this feature.

Total Control provides Lite version which is free for non-commercial use. All Total Control paid features are annual PC-based subscription plan. To purchase, click the shopping cart icon, select the features you want to purchase, select checkout, and payment method, once the payment is completed, restart Total Control on PC, the license will be upgraded with the new purchased features.

Total Control offers 2 payment methods:

• Google Pay: Due to limited functionalities in Google Pay, this payment method allows users to purchase the new subscription up to TC-MDC-50.
• Stripe: Stripe provides dynamic pricing and higher limit, users can use this method to purchase all subscriptions or upgrade to higher number of supported devices.

MDC-x

Since Professional contains base features for other paid features, users must purchase Professional subscription before purchase other features. The Professional 1-year subscription expiration date is important if you plan to purchase additional paid features at a later time. For instance, you purchased the professional subscription 6 months ago and decide to purchase MDC-20, you only need to pay half of the price of MDC-20, the MDC-20 expiration date is the same as Professional. Furthermore, if you plan to purchase MDC-30 3 months later (after 9 months from professional), you only need to pay the difference (TC deduct the unused time):

In this case:

All MDC-x are upgradable to higher number of devices with Stripe payment method.

TC-DC

TC-DC (Device Control) is a separate android application (called viewer) allowing MDCC users to choose a device to control main device, some applications are easier to navigate via device instead of using mouse and keyboard to control them. The viewer application can be either be one of the devices connected to MDCC or it can run on device anywhere that is reachable to PC running TC (even over VPN thousand miles away). If you are running MDCC, if you are in MDCC, you can choose any device click DC icon, if you are using TCP to connect to MDCC, please download the application here.

After the purchase is completed, it will send an email confirmation of the subscription and attach an "activation code", if you plan to move Total Control to new PC or you need to reinstall the OS, you first install the lite version in the new PC/OS, use the activation code to activate the paid features, when a valid activation code is entered, a new activation code will be generated, keep it safe. You can only make 10 transfers per year.

we provide 2 payment methods:

Item Code	Description	# of device
TC-Lite	Free version, offer many basic features	2
TC-Pro	Professional, offer many additional features	4
TC-MDC-10	Total Control to support 10 devices	10
TC-MDC-20	Total Control to support 20 devices	20
TC-MDC-30	Total Control to support 30 devices	30
TC-MDC-50	Total Control to support 50 devices	50
TC-DC	Device to control PC-TC to control other devices	N/A

Total control implemented several kinds of shortcuts to improve user experience, there are 2 kinds of shortcuts, one applies to device (in device settings) and the other applies to the entire system (in system settings):

• [Device] Shortkey: function key to run a script or start an App.
• [System] Input textbox Autotext to replace short form with longer form.
• [System] Capture PC and boss key.
• [System] Keyboard class API.

One of the biggest strengths in Total Control is the scripting capability. Total Control offers rich set of API to control one or multiple Android devices. It offers 2 sets of APIs: JavaScript and REST API. JavaScript will run the script locally, REST API offers flexibility in language and hosts. You can also use REST API to control multiple PCs running "Total Control" application.

Language

Total Control provide the following framework for JavaScript (ECMAScript 5 with selected ES6 features)

• Mozilla's Rhino 1.7.7 with Total Control 7 (or below), Rhino 1.7.12 with Total Control 8 and Rhino 1.7.14 with Total Control 9 (Update 20). See https://github.com/mozilla/rhino for more information.
• RingoJS 2.0.

The bonus with Rhino is the ability to call Java API (Total Control 8 is using OpenJDK 15) directly if Rhino + RingoJS do not provide sufficient functionality.

Total Control offers many classes, several examples:

• Device
• DeviceArray
• UiElement (Total Control 8)
• UiElementArray (Total Control 8)
• Notification
• Keyboard
• Excel

By default, the scripting directory is located at \Users\<user name>\Documents\Scripts directory, you can change it by clicking "Script" in Main Window, select "Script List", top line to change the directory.

Within it, you can create a file called "Userlib.js", this file will always be loaded before script execution, you can include commonly used functions, add prototypes to existing classes or bring in 3rd party software.

To debug "Userlib.js", use "Terminal", to reload "Userlib.js", click reload icon on the bottom right Terminal window.

For x, y coordinates, TC offers "absolute" and "relative" coordinates (the system settings show coordinates will show both coordinates), absolute coordinates from (0, 0) to (width – 1, height – 1), relative coordinates are usually 4 decimal points from (0, 0) to (0.9999, 0.9999). The relative coordinates multiply by device width and height will get the absolute coordinates. Coordinates are not ideal, Professional supports AAI that use text in the UI elements to retrieve the coordinates in the runtime.

The added bonus of JavaScript is the extensibility, you can extend the methods by adding prototype to the classes. Assuming you want to create a method "longPress" for both single and multiple devices, one way to write it is:

Device.prototype.longPress = function(x, y) {
    var retval = this.click(x, y, tcConst.STATE_DOWN);
    if (retval != 0) {
        return retval;
    }
    delay(500);
    return this.click(x, y, tcConst.STATE_UP);
}

DeviceArray.prototype.longPress = function(x, y) {
    for (let i = 0; i < this.length; ++i) {
        retval = this[i].longPress(x,y);
        if (retval != 0) {
            return retval;
        }
}
return 0;
}

This implementation not efficient for large number of devices in DeviceArray, the method can take up to 50 seconds if there are 100 devices in the array.

The best way is to send STATE_DOWN to all devices first, then wait till the remaining of 500 ms runs out. Fortunately, Device and DeviceArray (multiple devices) introduced "sendAll", it will execute the method specified and wait until time specified is expired. This way, the execution time is almost 500 ms. Since "sendAll" is available in Device and DeviceArray, the code to implement Device and DeviceArray is exactly the same.

Device.prototype.longPress = function(x, y) {
var retval = this.sendAll(Device.prototype.click, 
     [x, y, tcConst.STATE_DOWN], 500);
    if (retval != 0) {
        return retval;
    }
    return this.sendAll(Device.prototype.click, [x, y, tcConst.STATE_UP]);
}

DeviceArray.prototype.longPress = Device.prototype.longPress; 
 

sendAll only applies to methods that returns 0 on success.

There are many tools available in Total Control, click "Script" will open a new window:

Terminal: Open Rhino + RingoJS command prompt, you can use it for testing or development purpose. Click button on the bottom right to reload the interpreter and Userlib.js. Can use load("filename.js") to run a script.

Script List ⇒ Path: Script default path, click icon to change the path to other directory.

Script List ⇒ JS source files: Use to quickly execute a JavaScript file, it provide a simple editor for quick change or click arrow key to execute the script.

Script List ⇒ Record Script: Record the actions on the devices into an Excel file, Excel file can also be used to generate JS script or JSON file (maybe useful for REST API). Users with no experience in JavaScript can use this tool to generate simple script.

Script List ⇒ Image Helper. Generate BMP file from main device screen content, the image is needed for seekImage(), it extends BMP file to provide additional information such as app name, activity, width and height information. seekImage() will utilize these information without providing lots of parameters.

Script List ⇒ Color Helper. Load the screen content from the main device (like image helper), a magnified color chooser to pick RGB values of the colors and generate seekColor() with complex parameters. Can support single and multiple colors.

Script List ⇒ UI Explorer. Open UI Explorer window, click capture to capture the existing main phone window and allow users to experience query language to select UI elements. A Helper window will provide helper description for each keys in query language.

Execute (will be renamed to Task). Create a task to run a script on various number of conditions such as date & time, number of iterations, regular intervals or the devices. The output and the results of the task execution will be saved for review. Task can be also be created or changed via JS API. Runner execution will be shown as a task here.

Inspect: Inspect the various internal variables related to scripts and engines.

AAI identifies UI elements on screen as objects, traditional way of (x, y) coordinates take screen as one giant object, so seek image/color and OCR are required to identify an object on screen.

Accessibility is a feature that represent UI elements on screen as underlying nodes, a node includes many properties such as text/description, dimension, boolean properties such as clickable, editable or scrollable, underlying class name, etc. The text/description can be accessed easily (OCR is not needed), the dimension (and clickable) ensures the button can be clicked on certain location even the node is moved to another location.

A node can represent an UI element (e.g. button) or a group of UI elements or layout of certain elements. A node (element, group or layout) can be identified by a node ID (represent in string of hex). We integrate Accessibility, TC scripting framework and UI Automator library to achieve the following goals:

• Coordinate independent makes the script more portable with different resolutions, multiple sizes and brands.
• Synchronous API will wait until the screen is repaint, make the script simpler, do not need to guess the time to sleep.
• Can retrieve the string from the app with ease, instead of using error-prone OCR.

The simplest case of AAI:

• Click "OK" on the screen if found, far better than click(100, 100) for specific resolution: devices.clickSync("OK")
• Enter text into text entry, AAI can find all text entry lines in current screen.

  devices.inputTextSync([position], "text")    // Enter text, the position is used for multiple inputs

• Run or restart application, without query, it will return on screen refresh; with query, it will match the query after screen is refreshed.

  devices.runAppSync(<package name>, [query]) 
  devices.restartAppSync(<package name>, [query])

The entire screen is composed by many nodes, a node can be a smallest UI element or a container of many nodes, some nodes are invisible. The entire screen is a tree structure starting from the single root node. Depending on the complexity of App, a screen can contain 50-300 nodes.

Since the users are only interested in small subset of nodes, the challenge is finding the correct nodes users want and extract information or perform actions on them.

The challenge is how to find nodes? We invented a query language to find the nodes, the FindNode program is installed in every device, the query language will be performed to obtain the nodes that meet the criteria, the intent is to reduce the large number of nodes to one or few intended nodes, users can obtain information or apply actions to the nodes.

For example: UI Automator in Java provides "UiSelector" and "BySelector" in UiDevice.findObject() or findObjects() to locate nodes, it can be complex for multiple conditions:

new UiSelector().className("android.widget.TextView").text("OK")

We created a simple query language, that is shorter and portable since the query will be send to many devices, the above code can be rewritten in our query language as:

"C: android.widget.TextView&&T:OK"

AAI project includes the following:

1. Query language, simple one line syntax language to search for intended nodes. Core of the AAI.
2. FindNode carry out the query or actions on each device. All the query and certain actions are done in FindNode, it contains few dozen commands. See FindNode documentation for more information.
3. Object mode in one-to-many synchronization, send the node (or UI object) to all devices instead of coordination, click "OK" can run on all devices with different resolutions than click(100,100).
4. UI Explorer to obtain node information, can visually test the query language, learning and exploration tool.
5. AAIS, a simple language to perform automation on multiple devices. Capture and replay generate this language, see AAIS documentation for more information.
6. REST and JS API includes accessibility to FindNode.
7. UiElement class on top of FindNode to access node with ease.

Query

Each query has one or multiple "<key>:<value>" pair, multiple keys can be added with "&&" as separator.

Each node is identified by a node ID. A query can divided into 3 phases:

1. Template ("TP"). This class "generates" the initiate nodes. For instance "TP:textInput" will return a list of editable text fields. This class is mandatory, if not specified, default template will be used.
2. Basic query (BQ). Each node contains information about itself, the classes, the text/description, the properties, etc. BQ will match node one at a time, will reject the nodes that do not meet the criterion, will not be passed into next phase. If BQ is not specified, the nodes generate from TP will be passed into EQ.
3. Expanded query (EQ). A set of keys that usually work with multiple nodes. Multiple EQ execute from left to right, same key can be specified multiple times. An example of EQ: "OX:1" find the element/node on the right of the current node.

After the query is performed, one/more nodes found are listed in "ML" (matching list), list of actions can be applied on the ML, the actions can be retrieving information or perform action on ML.

Template:

Generate the initial nodes for BQ or EQ.

TP:all All nodes

TP:more All nodes except nodes end with "Layout"

TP:basic All leaf node (child count is zero)

TP:reduced Optimize "TP:more" to return nodes that is important for screen

TP:anyText[,<min>[,<max>]] Nodes with content in "text" in certain length.

TP:anyDescription[,<min>[,<max>]] Nodes with content in "description" in certain length.

TP:textInput All "editable" fields that sorted from top-left to bottom-right.

TP:findText,<text> Nodes with the text in the arguments, can contain "*" and "/…/".

TP:line,top|bottom,<number> Return top/bottom nodes that is outside of scrollable nodes.

TP:scrollable,<position> Nodes inside scrollable container, position for multiple scrollable nodes.

Basic Query (BQ):

Query for node-level information obtained, each nodes from TP will match (if provided) in BQ to proceed.

• P:<package name> -Should not be used, default to running app
• C:<class name> Class name (S)
• R:<resource ID> Resource ID (S)
• D:<text> Description (S)
• T:<text> Text (S)
• IT:<number> Text input type (I)
• CC:<number> Child count (I)
• ID:<ID> Node ID in hex (S)
• BI:[x, y] The nodes contain (x,y)
• BI:[x1, y1, x2, y2] The nodes enclosed by the rectangle, if x or y is -1, it will be ignored
• BP:<prop name> Boolean properties (S).
  • checkable, checked, clickable, editable, enabled, focused, longClickable
  • scrollable, visibleToUser.
• TD:<text> Match text or description (S).

Expanded Query (EQ):

Queries here usually across multiple nodes.

The order of Expanded Query is important, all expanded queries execute from left to right. Commands with the same keys are allowed.

• IX:<number> Obtain the one node from list of matching nodes based on position
• OX:<number> Offset to neighbor nodes horizontally (positive – right, negative – left)
• OY:<number> Offset to neighbor nodes vertically (positive – down, negative – up)
• ON:<type> Different ways to pick one node out of list of matching nodes
  • first, last, min
• ST:<sort type> Return sorted nodes based on the position of the nodes on screen
  • x, y, yx (or all)
• TX Return nodes that intersect with reference node horizontally.
• TY Return nodes that Intersect with reference node vertically.
• VG:[level number] Return the group of nodes in a view group from first node in ML.
• RN Return the optimized nodes from a list of matching nodes.
• BQ:<query> Perform basic query.
• X:<key in BQ> Basic query key prefixed with "X"

For BQ, the query syntax can contain "!" for not, ">", "<" for greater than or less than, "*" for wild card match and "/<regexp>/" for regular expression. It can match package name, class name, resource ID, text, description, child count and input type.

FindNode is installed into every device (part of Total Control App), it is the only program that recognize the query syntax, it parses query, locate nodes and perform actions to the nodes found. FindNode offload the complexity of JavaScript and CPU utilization of Total Control, all the search is conducted in the devices.

device.sendAAi() and devices.sendAai() are direct way to communicate with FindNode with one or list of devices. The JS object will be translated to JSON before sending to device, the returned value is in JS object format. If error is encountered, the return is null, the lastError() contains error message.

A simple query example, to obtain the text of Model name, use X offset of 1 (next to the right):

>> device.sendAai({query:"T:Model name&&OX:1", action:"getText"})
{retval: 'Galaxy S10+'}

FindNode can even detects the fixed icons on the top/bottom of the screen:

>> device.sendAai({query:"TP:line,bottom,-1", action:"getText"})
{retval: ['Chats','Calls','Contacts','Notifications']}

The following 3 commands, doing the same thing, click on the "Calls" text:

>> device.sendAai({query:"TP:line,bottom,-1&&T:Calls", action:"click"})
{retval: true}
>> device.sendAai({query:"TP:line,bottom,-1&&IX:1", action:"click"})
{retval: true}
>> device.sendAai({query:"TP:line,bottom,-1&&T:Chats&&OX:1", action:"click"})
{retval: true}

Click "Contacts" icon:

>> device.sendAai({query:"TP:line,bottom,-1&&T:Contacts&&OY:-1", action:"click"})
{retval: true}
>> device.sendAai({query:"TP:line,bottom,-1&&IX:2", action:"click"})
{retval: true}
// Find multiple "Contacts" are found on the screen, IX:-1 is to select the 
// last node found 
>> device.sendAai({query:"T:Contacts&&IX:-1&&OY:-1", action:"click"})
{retval: true}

Please read FindNode User Guide for complete information.

Before version 8, all action or movement commands are asynchronous, when a command (e.g. device.click()) return, it means the command has been passed down to our mobile agent for execution, when the command returns, it is likely the command is not executed, so a sleep is required to give time for the action to be done and screen is refreshed:

	device.click(100, 100);
	sleep(300);

This can happen to all action and movement commands which make the coding cumbersome. UI Automator includes synchronous command and wait for window to update, when a click is pressed, a window has been updated, most likely the command is completed. All synchronous commands are suffixed by "Sync". All "Sync" commands are using AAI label instead of coordinate:

One drawback of the synchronous feature is the time it takes for the commands to complete is much longer than asynchronous commands, this can slow down the script executions for large number of devices, we have addressed this issue.

The subsequent versions will provide synchronous feature to all action and movement functions.

We also developed a mini script that utilize AAI capability to do automation, this is currently available in WDM and MDCC (the file extension is ".tst"), capture and replay (with object option) will generate in this script. It also offers seamless integration with JavaScript (enclosed in "{}").

E.g.

exec "teslaLib.js"
open "Tesla"
find "T:VIN:"
get "T:VIN:&&OX:1", "text"
{ saveVar("vin", getOutput().retval) }
get "T:/[0-9,]+ miles/", "text"
{ saveMileage(new Date(), loadVar("vin"), getOutput().retval) }
print "Done"

This script can be run on multiple devices concurrently, can be run on any screen size, "find" will scroll until the query is found. Any failure on any line will stop the script. It provides commands for both AAIS and JavaScript API.

Another example:

{
    var apps = ["Skype", "Whatsapp", "Telegram"];
    var arguments = getArg();
    var app = apps[0];
    if (arguments.length > 0) {
        var appNum;
        if(isNaN(appNum = parseInt(arguments[0]))) {
            throw "Need a number;"
        }
        if (appNum < 1 || appNum > apps.length) {
            throw "Option out of range";
        }
        app = apps[appNum-1];
    }
}
open "${app}"
print "${app} opened"

Please read AAIS User Guide for AAIS information.

Click on the "MDCC" icon in the main window of Total Control to enter the Multi-Device Control Center.

More content will available shortly.

The article will available shortly.

Please send email to support@sigma-rt.com for support and feedbacks.