Wrench: control your Android phone from your Desktop

Here’s a video of Wrench: playing monument valley (with phone screen mirrored), using Lua to script your phone, and ordering coffee (kinda).

Here’s another youtube video which introduced Wrench’s most useful functions: chatting with IM and Social Media Apps, finding contacts in IM and Social Media Apps, writing emails, and so on (inputing text is best done with the PC, then sent with the phone).

Install

There are prebuilt versions for Windows, Mac and Linux (ubuntu 14.04). They are released at https://github.com/SmartisanTech/Wrench-releases/releases .

Compilation

To compilen Wrench, you need make sure SnoreNotify is compiled and installed first. It’s in the SnoreNotify sub-directory. Please refer to the SnoreNotify/build-*.sh for your platform.

Compiling Wrench

• Under Linux, compile Wrench with the build-linux.sh script. You should be able to run this script directly.

  Because Wrench used QHotkey, you need to install libqt5x11extras5-dev first if you are under Debian/Ubuntu or something similar under other Linux distributions.

• For Windows version, I compile with Wine under Linux, with the Qt for msys installed as compiler. Please refer to the ./build-wine.sh script.
• For Mac version, please refer to the ./build-mac.sh script.

You need to adapt the ./build-wine.sh and ./build-mac.sh scripts, there are some hardcoded parameters (such as the PATH environment and the hostname of the Mac PC that I use for compilation).

How does it work

Android has provided quite some debugging abilities through adb, for example, you can

1. Inject keyboard and touch events with input
2. Start Activities and Services with am start|startservice
3. Take a screen shot with screencap
4. Record your screen with screenrecord
5. Dump window infomation with dumpsys window (where you can also get infomation about the Input Method)

Wrench just makes use of all the above, with a great deal of optimization.

Optimazations

better and faster adb shell input

Android’s builtin input command allows you to inject touch, keyboard, text events, but there’re some limitations.

1. You can use adb shell input text <string> to input a string, but the string is limited: it can’t contain spaces or any other special characters, such as quote and double quote, or Chinese characters.

   I solved this problem by using the Android clipboard and my own Input Method WrenchIme adapted from Google Pinyin. When Wrench need to input any string, it:

   • Put the string into the phone’s clipboard.
   • Trigger a special key (SCROLL_LOCK) with adb shell input
   • WrenchIme interpret the SCROLL_LOCK key to send the clipboard text to the current App, as if the text is an IME candidate and has been selected.
2. adb shell input is very slow, because every time input is run, a framework/input.jar is loaded and a new java virtual machine is created by running app_process.

   To solve this problem, Wrench comes with a Setclip.apk, which whileas being an Android Apk, it also provides the functionality of framework/input.jar (i.e., it can also be loaded and run by app_process), but instead of loading every time, it loads once and then runs as a daemon.

faster adb shell am

For the same reason as adb shell input, am is also very slow because an am.jar is loaded with a new java virtual machine each time. Wrench improved it using the same trick as input, in fact, Setclip.apk will load am.jar in turn when it is daemonized.

(Wrench’s am.jar is from AOSP, with userdebug build to make sure the .dex java class files are not optimized away and thus dalvik loadable.)

faster phone screen mirroring

At first, Wrench used screencap to make a screenshot of the phone screen, and displayed on the PC. This is very slow.

Now, with some phones (according to the Android SDK version and CPU arch), Wrench can use the VNC protocol to mirror your phone screen, which is a lot smoother as you can see from the video. In doing this, Wrench used source code from 3 projects: it combined the AOSP screenrecord project and the droidVncServer project to run a VNC server on the phone; and used the Qt VNC Viewer to talk to the phone’s VNC server.

How to use Wrench

Please refer to this youtube video.

There is also a blog explaining all the details, but it’s in Chinese.

Here’s the simple steps:

1. Start the Wrench program
2. Enable the phone’s USB debugging.
3. Connect the phone to Desktop with USB.

(Wrench is packaged with the AOSP adb program, so you need to make sure you are not running a different version of adb, as it may cause connection trouble.)

How to compile droidVncServer

Note

If your phone is deeply customized, it might not work if you compile the droidVncServer in AOSP source code. I found this out on Smartisan (the company that I’m working at)’s U1 phone, where I compiled droidVncServer with aosp-lollipop-mr1-dev, which has the same SDK version as U1, but sadly the program crashes when run. I also compiled the AOSP screenrecord program with aosp-lollipop-mr1-dev, then run it on U1, and found that it can do .mp4 file recording, but also crashes when run for streaming (with --output-format frames).

In this case, the only way to make it work is to compile under the same source code where your phone’s system image is compiled, which is unlikely unless you are a developper of this phone hardware company. Sorry.

If your phone’s Android SDK version/CPU arch is not supported, you can compile it yourself.

First you need get AOSP source code. SDK/NDK is not supported to compile droidVncServer because some system APIs are used which is not available in SDK/NDK.

Next copy the droidVncServer directory into AOSP’s external/droidVncServer, set up AOSP’s build appropriately, and run mma in external/droidVncServer.

Note that the source code for droidVncServer may need some porting if your phone’s SDK version is not 22 or 23. The changes may be small or big or even impossible depending on how different your phone’s SDK version is.

When build is finished, the binary is in external/droidVncServer, you need copy it back into your Wrench executable directory, with a rename (for e.g., androidvncserver-arm64.sdk23 should be renamed to androidvncserver-aarch64.sdk23, where aarch64 is the output from your phone’s adb shell uname -m ), and generate a .md5 file for it (Please refer to release/androidvncserver-aarch64.sdk23.md5).

Security Risks

Wrench need to use adb to connect to your phone. So it has all the security risks that comes with leaving adb open on your phone.

Wrench input text by putting it into the phone’s clipboard, so if there are clipboard sniffers running, you may accidentally leak sensitive infomation.

Contribute

Bug reports, patches, ideas are welcome.

Or, if you like Wrench, you may also donate some money to me with Paypal:

https://pledgie.com/campaigns/32699

or with alipay: