Blind monkey (#3)

* Add blind_monkey files

* Add TalkBack navigation to CLI

* Add TalkBack navigation with snapshots

* Refactor bm_explore

* Add python interface

* Add missing files

* Refactor files

* Refactor files
Add meaningful_actions to snapshot
Support proper physical click

* Add SightedTalkBack, Finalize report in py_code

* Add README, refactor files

* Make XML expandable

* Refactor (#2)

* Add Flask Visualizer

* Add Flask app, capture screenshots, visualize STB part

* Add timeout

* Add rectangle annotation to initial images

* Finalize blind monkey report page

* Refactor TalkBack Navigator, Add config file, add timeout

* Commit the latest

* update readme

* Refactor

* Add logger

* Refactor

* refactor

* Add a11y node tree

* Add retry, remove file in Android after reading, store logs in file

* Fix BroadcastReceiver message sanitization, add visible attribute to A11y Tree Node

* Change Result format, add version 2 visualizer, fix some bugs, WIP: STB

* Fix Sighted Explore

* fix a name

* finalize result writing and post analysis refactoring

* Finalize Version 2 visualization

* Update README, refactors in files

* Add some comments

* F

README.md

@@ -0,0 +1,69 @@
+# BlindMonkey
+## Setup
+-  Install python packages `pip install -r requirements.txt`
+- Initialize an Android Virtual Device (AVD) with SDK +28 and name it `testAVD_1`
+-	Disable soft main keys and virtual keyboard by adding `hw.mainKeys=yes` and `hw.kayboard=yes`  to `~/.android/avd/testAVD_1.avd/config.ini`
+	- If virtual device is not disabled, please follow this [link](https://support.honeywellaidc.com/s/article/CN51-Android-How-to-prevent-virtual-keyboard-from-popping-up)
+- Enable "Do not disturb" in the emulator to avoid notifications during testing (it can be found at the top menu)
+- Install TalkBack, the latest version (9.1) can be found in `Setup/talkback.apk` (`adb install Setup/talkback.apk`)
+- Build Latte Service APK by running `./build_latte_lib`, then install it (`adb install -r -g Setup/latte.apk`) or install from Android Studio
+	- To check if the installation is correct, first run the emulator and then execute `./scripts/enable-talkback.sh` (by clicking on a GUI element it should be highlighted). 
+	- Also, execute `./scripts/send-command.sh log` and check Android logs to see if Latte prints the AccessibilityNodeInfos of GUI element on the screen (`adb logcat | grep "LATTE_SERVICE"`)
+-  Save the base snapshot by `./scripts/save_snapshot.sh BASE`
+
+## Latte CLI
+You can interact with Latte by sending commands to its Broadcast Receiver or receive generated information from Latte by reading files from the local storage. First, you need to enable Latte by running `./scritps/enable-service.sh`, then you can send command by running `./scripts/send-command.sh <COMMAND> <EXTRA>`. If you want to work with TalkBack, first you need to enable it by running `./scritps/enable-talkback.sh`. If any command has an output written in a file, you can use `./scripts/wait_for_file.sh <FILE_NAME>` which prints the content of the file and removes it. It's encouraged to watch the logs in a separate terminal `adb logcat | grep "LATTE_SERVICE"`. Here is the list of all commands:
+- **General**
+	- `log`: Prints the current layout's xpaths in Android logs.
+	- `capture_layout`: Dumps the XML file of the current layout. Output's file name: `a11y_layout.xml`
+	- `report_a11y_issues`: Prints the accessibility issues (reported by Accessibility Testing Framework) in Android logs.
+
+- **TalkBack Navigation**
+	- `nav_next`: Navigates the focused element to the next element. Output's file name: `finish_nav_action.txt`
+	- `nav_select`: Selects the focused element (equivalent to Tap). Output's file name: `finish_nav_action.txt`
+	- `nav_interrupt`: Interrupt the current navigation action
+	- `nav_clear_history`: In case the last navigation result is not removed.
+- **UseCase Executor**
+	- `enable`/`disable`: Enable/Disable the use-case executor component
+	- `set_delay`: Sets the time for each interval (cycle).
+	- `set_step_executor`: Sets the driver (step_executor). The extra can be `talkback`, `regular` (touch based), `sighted_tb` (touch based TalkBack).
+	- `set_physical_touch`: If the extra is 'true', the regular executor emulates *touch*, otherwise it uses A11yNodeInfo events to perform actions.
+	- `do_step`: Performs a single step where the step is provided in extra.
+	- `interrupt`: Interrupts the current step execution
+	- `init`: Initializes a use case, the use case speicfication is provided in extra.
+	- `start`: Starts the use case (`init` must be called beforehand)
+	- `stop`: Stops the current use case execution
+
+## BlindMonkey
+To analyze a snapshot, first load the BASE snapshot `./scripts/load_snapshot.sh BASE`, then install the app under test, and go to the screen that you want to analyze. Next, creates a new snapshot by `./scripts/save_snapshot.sh <SNAPSHOT>`. Now you can run the BlindMonkey on this snapshot by running
+```
+python pt_src/main.py --app-name <APP_NAME> --output-path <RESLUT_PATH> --snapshot <SNAPSHOT> --debug
+```
+
+
+
+## OLD ---- Run SnapA11yIssueDetector
+- Load the base snapshot by `./scripts/load_snapshot.sh BASE`
+- Install the app you want to test, for example: `adb install -r -g Setup/yelp.apk`
+- Run the app and go to a state you want to test, then take a snapshot, for example: `adb shell monkey -p com.yelp.android 1` and `./scripts/save_snapshot.sh Yelp_0`
+- Run SnapA11yIssueDetector by executing `cd py_src && python main.py Yelp_0`
+- Once the script is done, you can analyze the result using following python script:
+
+```
+from snapshot import Snapshot
+snapshot = Snapshot("Yelp_0")
+different_behaviors, directional_unreachable, unlocatable, different_behaviors_directional_unreachable = snapshot.report_issues()
+```
+
+## OLD ------ Use Library
+- Create an app
+- Add dependency (AAR)
+- Add accessibility_service_config
+    - Add string in xml
+- Create a service and inherits from LatteService
+- if no activity, change the default launch option to nothing
+- TODO: Change enable-service and disable-service
+
+##  OLD ------- Communication Service
+- Http android:usesCleartextTraffic="true"
+-

UseCaseExecutor/src/main/AndroidManifest.xml

@@ -1,6 +1,6 @@
 <?xml version="1.0" encoding="utf-8"?>
 <manifest xmlns:android="http://schemas.android.com/apk/res/android"
-    package="dev.navids.latte">
+    package="dev.navids.latte.lib">
 
     <application>
 

UseCaseExecutor/src/main/java/dev/navids/latte/ActionUtils.java

@@ -3,13 +3,16 @@
 import android.accessibilityservice.AccessibilityService;
 import android.accessibilityservice.GestureDescription;
 import android.graphics.Path;
+import android.graphics.Rect;
 import android.os.Bundle;
 import android.util.Log;
+import android.util.Pair;
 import android.view.accessibility.AccessibilityNodeInfo;
 
-public class ActionUtils {
+import java.util.ArrayList;
+import java.util.List;
 
-    public static final int tapDuration = 100;
+public class ActionUtils {
     private static AccessibilityService.GestureResultCallback defaultCallBack= new AccessibilityService.GestureResultCallback() {
         @Override
         public void onCompleted(GestureDescription gestureDescription) {
@@ -24,7 +27,83 @@ public void onCancelled(GestureDescription gestureDescription) {
         }
     };
 
-    public static boolean performTap(int x, int y){ return performTap(x, y, tapDuration); }
+    public static boolean isFocusedNodeTarget(List<AccessibilityNodeInfo> similarNodes) {
+        if(similarNodes.size() == 0)
+            return false;
+        AccessibilityNodeInfo targetNode = similarNodes.get(0); // TODO: This strategy works even we found multiple similar widgets
+        AccessibilityNodeInfo firstReachableNode = targetNode;
+        boolean isSimilar = firstReachableNode != null && firstReachableNode.equals(LatteService.getInstance().getFocusedNode());
+        if(!isSimilar) {
+            AccessibilityNodeInfo it = targetNode;
+            while (it != null) {
+                if (it.isClickable()) {
+                    firstReachableNode = it;
+                    break;
+                }
+                it = it.getParent();
+            }
+            Log.i(LatteService.TAG, "-- FIRST REACHABLE NODE IS " + firstReachableNode);
+            isSimilar = firstReachableNode != null && firstReachableNode.equals(LatteService.getInstance().getFocusedNode());
+        }
+        return isSimilar;
+    }
+
+    public static Pair<Integer, Integer> getClickableCoordinate(AccessibilityNodeInfo node){
+        return getClickableCoordinate(node, true);
+    }
+
+    public static Pair<Integer, Integer> getClickableCoordinate(AccessibilityNodeInfo node, boolean fast){
+        int x, y;
+        if(fast)
+        {
+            List<AccessibilityNodeInfo> children = new ArrayList<>();
+            children.add(node);
+            for (int i = 0; i < children.size(); i++) {
+                AccessibilityNodeInfo child = children.get(i);
+                for (int j = 0; j < child.getChildCount(); j++)
+                    children.add(child.getChild(j));
+            }
+            Rect nodeBox = new Rect();
+            node.getBoundsInScreen(nodeBox);
+            children.remove(0);
+            int left = nodeBox.right;
+            int right = nodeBox.left;
+            int top = nodeBox.bottom;
+            int bottom = nodeBox.top;
+            for (AccessibilityNodeInfo child : children) {
+                if(!child.isClickable())
+                    continue;
+                Rect box = new Rect();
+                child.getBoundsInScreen(box);
+                left = Integer.min(left, box.left);
+                right = Integer.max(right, box.right);
+                top = Integer.min(top, box.top);
+                bottom = Integer.max(bottom, box.bottom);
+            }
+            Log.i(LatteService.TAG, " -------> " + nodeBox + " " + left + " " + right + " " + top + " " + bottom);
+            if(left > nodeBox.left)
+                x = (left+ nodeBox.left) / 2;
+            else if(right < nodeBox.right)
+                x = (right + nodeBox.right) / 2;
+            else
+                x = nodeBox.centerX();
+            if(top > nodeBox.top)
+                y = (top+nodeBox.top) / 2;
+            else if(bottom < nodeBox.bottom)
+                y = (top+nodeBox.top) / 2;
+            else
+                y = nodeBox.centerY();
+        }
+        else {
+            Rect box = new Rect();
+            node.getBoundsInScreen(box);
+            x = box.centerX();
+            y = box.centerY();
+        }
+        return new Pair<>(x,y);
+    }
+
+    public static boolean performTap(int x, int y){ return performTap(x, y, Config.v().TAP_DURATION); }
     public static boolean performTap(int x, int y, int duration){ return performTap(x, y, 0, duration); }
     public static boolean performTap(int x, int y, int startTime, int duration){ return performTap(x, y, startTime, duration, defaultCallBack); }
     public static boolean performTap(int x, int y, int startTime, int duration, AccessibilityService.GestureResultCallback callback){
@@ -46,26 +125,28 @@ public static boolean performType(AccessibilityNodeInfo node, String message){
         return node.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, arguments);
     }
 
-
     public static boolean performDoubleTap(){
+        return performDoubleTap(defaultCallBack);
+    }
+    public static boolean performDoubleTap(final AccessibilityService.GestureResultCallback callback){
         Log.i(LatteService.TAG, "performDoubleTap");
         try {
-            Thread.sleep(300); // TODO
+            Thread.sleep(300); // TODO: What is this?
         } catch (InterruptedException e) {
             e.printStackTrace();
         }
-        return performDoubleTap(0, 0); }
-    public static boolean performDoubleTap(int x, int y){ return performDoubleTap(x, y, tapDuration); }
-    public static boolean performDoubleTap(int x, int y, int duration){ return performDoubleTap(x, y, 0, duration); }
-    public static boolean performDoubleTap(int x, int y, int startTime, int duration){ return performDoubleTap(x, y, startTime, duration, defaultCallBack); }
+        return performDoubleTap(0, 0, callback);
+    }
+    public static boolean performDoubleTap(int x, int y, final AccessibilityService.GestureResultCallback callback){ return performDoubleTap(x, y, Config.v().TAP_DURATION, callback); }
+    public static boolean performDoubleTap(int x, int y, int duration, final AccessibilityService.GestureResultCallback callback){ return performDoubleTap(x, y, 0, duration, callback); }
     public static boolean performDoubleTap(final int x, final int y, final int startTime, final int duration, final AccessibilityService.GestureResultCallback callback){
         AccessibilityService.GestureResultCallback newClickCallBack = new AccessibilityService.GestureResultCallback() {
             @Override
             public void onCompleted(GestureDescription gestureDescription) {
                 Log.i(LatteService.TAG, "Complete Gesture " + gestureDescription.getStrokeCount());
                 super.onCompleted(gestureDescription);
                 try {
-                    Thread.sleep(100); // TODO
+                    Thread.sleep(Config.v().DOUBLE_TAP_BETWEEN_TIME);
                 } catch (InterruptedException e) {
                     e.printStackTrace();
                 }
@@ -76,6 +157,8 @@ public void onCompleted(GestureDescription gestureDescription) {
             public void onCancelled(GestureDescription gestureDescription) {
                 Log.i(LatteService.TAG, "Cancel Gesture");
                 super.onCancelled(gestureDescription);
+                if(callback != null)
+                    callback.onCancelled(gestureDescription);
             }
         };
         return performTap(x, y, startTime, duration, newClickCallBack);

UseCaseExecutor/src/main/java/dev/navids/latte/ActualWidgetInfo.java

@@ -1,7 +1,11 @@
 package dev.navids.latte;
 
+import android.graphics.Rect;
 import android.view.accessibility.AccessibilityNodeInfo;
 
+import org.json.JSONException;
+import org.json.JSONObject;
+
 import java.util.ArrayList;
 import java.util.List;
 
@@ -14,22 +18,31 @@ public ActualWidgetInfo(String resourceId, String contentDescription, String tex
     }
 
     public static ActualWidgetInfo createFromA11yNode(AccessibilityNodeInfo node){
+        return createFromA11yNode(node, true);
+    }
+
+    /**
+     * @param fix_text "If it's true, it will create content description or text for parent views like Layout
+     */
+    public static ActualWidgetInfo createFromA11yNode(AccessibilityNodeInfo node, boolean fix_text){
         if (node == null){
             return null;
         }
         String resourceId = String.valueOf(node.getViewIdResourceName());
         String contentDescription = String.valueOf(node.getContentDescription());
         String text = String.valueOf(node.getText());
         String clsName = String.valueOf(node.getClassName());
-        if (clsName.endsWith("Layout")){
-            if (text.equals("null") && contentDescription.equals("null")) {
-                String tmp = Utils.getFirstText(node);
-                if (tmp != null)
-                    text = tmp;
-                else {
-                    tmp = Utils.getFirstContentDescription(node);
+        if(fix_text) {
+            if (clsName.endsWith("Layout")) {
+                if (text.equals("null") && contentDescription.equals("null")) {
+                    String tmp = Utils.getFirstText(node);
                     if (tmp != null)
-                        contentDescription = tmp;
+                        text = tmp;
+                    else {
+                        tmp = Utils.getFirstContentDescription(node);
+                        if (tmp != null)
+                            contentDescription = tmp;
+                    }
                 }
             }
         }
@@ -96,6 +109,30 @@ public boolean isSimilar(WidgetInfo other, List<String> myMaskedAttributes) {
         return super.isSimilar(other, myMaskedAttributes);
     }
 
+    @Override
+    public String completeToString(boolean has_xpath) {
+        String base_path = super.completeToString(has_xpath);
+        Rect boundBox = new Rect();
+        node.getBoundsInScreen(boundBox);
+        String str = String.format("%s, bound= %d-%d-%d-%d",base_path, boundBox.left, boundBox.top, boundBox.right, boundBox.bottom);
+        return str;
+    }
+
+    @Override
+    public JSONObject getJSONCommand(String located_by, boolean skip, String action){
+        JSONObject result = super.getJSONCommand(located_by, skip, action);
+        if (result == null)
+            return result;
+        Rect boundBox = new Rect();
+        node.getBoundsInScreen(boundBox);
+        try {
+            result.put("bound", String.format("%d-%d-%d-%d", boundBox.left, boundBox.top, boundBox.right, boundBox.bottom));
+        } catch (JSONException e) {
+            e.printStackTrace();
+        }
+        return result;
+    }
+
     @Override
     public String toString() {
         return "ActualWidgetInfo{"