From dd210e2fd0c0e58496832d1501f058e03b2c6a03 Mon Sep 17 00:00:00 2001 From: Plamen Dimitrov Date: Sat, 27 Apr 2024 00:54:44 +0800 Subject: [PATCH] Update RTD documentation with recent developments RTD configuration file and a default index.rst is now required and some Sphinx settings have also changed accordingly. Let us also use the opportunity to simplify the API doc generation bash script. --- .readthedocs.yaml | 29 +++++ docs/api/README.rst | 126 ---------------------- docs/api/conf.py | 2 +- docs/api/generate_api.sh | 9 +- docs/api/index.rst | 98 +++++++++++++++++ docs/api/source/guibot.calibrator.rst | 6 +- docs/api/source/guibot.config.rst | 6 +- docs/api/source/guibot.controller.rst | 6 +- docs/api/source/guibot.desktopcontrol.rst | 6 +- docs/api/source/guibot.errors.rst | 6 +- docs/api/source/guibot.fileresolver.rst | 6 +- docs/api/source/guibot.finder.rst | 6 +- docs/api/source/guibot.guibot.rst | 6 +- docs/api/source/guibot.guibot_proxy.rst | 6 +- docs/api/source/guibot.guibot_simple.rst | 6 +- docs/api/source/guibot.imagelogger.rst | 6 +- docs/api/source/guibot.inputmap.rst | 6 +- docs/api/source/guibot.location.rst | 6 +- docs/api/source/guibot.match.rst | 6 +- docs/api/source/guibot.path.rst | 6 +- docs/api/source/guibot.region.rst | 6 +- docs/api/source/guibot.rst | 7 +- docs/api/source/guibot.target.rst | 6 +- packaging/pip_requirements_minimal.txt | 3 + 24 files changed, 191 insertions(+), 185 deletions(-) create mode 100644 .readthedocs.yaml delete mode 100644 docs/api/README.rst create mode 100644 docs/api/index.rst create mode 100644 packaging/pip_requirements_minimal.txt diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 00000000..ee1e2f70 --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,29 @@ +# .readthedocs.yaml +# Read the Docs configuration file +# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details + +# Required +version: 2 + +# Set the version of Python and other tools you might need +build: + os: ubuntu-22.04 + tools: + python: "3.12" + +# Build documentation in the docs/api directory with Sphinx +sphinx: + builder: html + configuration: docs/api/conf.py + fail_on_warning: true + +# Build docs in additional formats such as PDF and ePub +formats: all + +# Specify dependencies to enable reproducible builds: +# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html +python: + install: + - requirements: packaging/pip_requirements_minimal.txt + - method: pip + path: ./packaging/ diff --git a/docs/api/README.rst b/docs/api/README.rst deleted file mode 100644 index 61d2d31f..00000000 --- a/docs/api/README.rst +++ /dev/null @@ -1,126 +0,0 @@ -Guibot -====== - -A tool for GUI automation using a variety of computer vision and display control backends. - -.. note:: This is a complete guide on the concepts and usage of the Guibot python package and its interfaces. For a brief overview listing all backends and resources, see the project's README on Github. The resource links there are useful for information on installation, issue reporting, and advanced usage. - -Main concepts -------------- - -In order to do GUI automation you usually need to solve two problems: first, you need to have a way to control and interact with the interface and platform you are automating and second, you need to be able to locate the objects you are interested in on the screen. Guibot helps you do both. - -To interact with GUIs, Guibot provides the `controller `__ module which contains a common interface for different display backends, with methods to move the mouse, take screenshots, type characters and so on. The backend to use will depend on how your platform is accessible, with some backends running directly as native binaries or python scripts on Windows, macOS, and Linux while others connecting through remote VNC displays. - -To locate an element on the screen, you will need an image representing the screen, a `target `__ representing the element (an image or a text in the simplest cases) and a `finder `__ configured for the target. The finder looks for the target within the screenshot image and returns the coordinates to the region where that target appears. Finders, just like display controllers, are wrappers around different backends supported by Guibot that could vary from a simplest 1:1 pixel matching by controller backends, to template or feature matching mix by OpenCV, to OCR and ML solutions by Tesseract and AI frameworks. - -Finally, to bridge the gap between controlling the GUI and finding target elements, the `region `__ module is provided. It represents a subregion of a screen and contains methods to locate targets in this region using a choice of finder and interact with the graphical interface using a choice of controller. - -Main interfaces ---------------- - -There are three entry point interfaces that you can use to automate any GUI operations: - - * The guibot object from `guibot `__ - * The simple guibot module API from `guibot_simple `__ - * The remote guibot API from `guibot_proxy `__ - -The recommended option for all standard use cases is the first one while the second is a simpler procedural form of it. The third interface can be used in more special circumstances where you are automating GUI operations on a remote machine and want to use less data intensive connection through PyRO serialization. - -All three interfaces offer the same portfolio of functions: the functional capabilities of a region instance with a few added ones for file resolution. The function names must be self-explanatory so let's provide some examples: - -:: - - from guibot.guibot import GuiBot - - # initialize a starting region with default display controller and finder - guibot = GuiBot() - # add path to local ./images folder where we can find files needed for the target - guibot.add_path('images') - - # this will look for ok_button.png within the added images folder, then look for - # it on the screen and return true if it was found or false otherwise - if guibot.exists('ok_button'): - # find the button again and click on it - guibot.click('ok_button') - else: - # type a text with our disappointment from the outcome - guibot.type_text('Ok button does not exist') - -This example code performs the simple actions of looking for a button using default computer vision (CV) and display controller (DC) backends (listed and managed in the `config `__ module) and clicking on it if it was found or typing that it could not be found. - -:: - - from guibot.guibot import GuiBot - - guibot = GuiBot() - guibot.add_path('account_images') - - # select account type on the screen as the second item from a drop down list - # shifted 10 pixels below a text label that we provide an image of - guibot.select_at("account-type", 2, 0, 10) - # passively locate a big side panel region with 20s patience from animations - menu_panel = guibot.find("account-options", timeout=20) - # within the panel region click an option to show account credentials and - # stay idle for an extra second until the credentials show up - menu_panel.click("account-options-credentials").idle(1) - # fill in the username "dumbledore" at a text box 100 pixels to the right of - # a text label, pressing escape to avoid filled text suggestions - guibot.fill_at("account-username", "dumbledore", 100, 0, esc_flag=True) - # wait for a confirmation dialog to appear (more time tolerant) - guibot.wait("account-name-confirmation") - # pick the region to the topleft of the account login status - topright_menu = guibot.find("account-login-status", timeout=20).left().above() - # click on an exit button within the topleft region - topright_menu.click("account-logout") - -This example code performs a user login navigating through more complex controls and reusing previously matched regions. Since the guibot instance inherits these capabilities of a region instance and each one of these calls will return a resulting subregion, the region calls could be nested with each successive call operating on the result of the previous one. A more advance usage than this could simply import and directly initialize region instances with separate calls to a file resolver to add the necessary file paths instead of a single guibot instance. - -:: - - from guibot.region import Region - from guibot.fileresolver import FileResolver - from guibot.target import Image, Text - from guibot.finder import TextFinder - - # add the paths to all target files through a more general file resolver - file_resolver = FileResolver() - file_resolver.add_path('target_data/') - - # define more general targets like image instance (default from string before) - # and a text target to read out from the screen - some_image = Image('shape_blue_circle') - some_text = Text('Some text here') - - # define a region instead of the previous guibot object capable of the same - # calls with the exception of path handling from the file resolver - image_region = Region() - # define a region with a different CV backend, namely a text finder with its - # default OCR backends (also can be found in the config module) - text_region = Region(cv=TextFinder()) - # click on the image instance using all default settings and wait for it to vanish - image_region.click_vanish(some_image) - # hover the mouse on top of the text that was read from the screen but - # reusing the matched subregion - read_text = text_region.hover(some_text) - assert read_text.similarity == 1.0, "Text read completely accurately" - -This final example makes the step of using a more general API based entirely on regions and does a simple change of a CV backend to read a text string from the screen. It also makes use of targets different than images and makes use of direct target instances rather than default strings (which are otherwise used to guess the target type). The text finder here is used for text detection and recognition (OCR) and offers further choice of backend implementations as well as specific parameters that can be used for fine-tuning (even calibrated) but this is the subject of more advanced tutorials. - -Advanced tutorials ------------------- - -For more advanced usages feel free to explore our API here or visit some of the following tutorials: - - * A `tutorial `__ on match settings explaining how to configure CV backends and fine tune CV parameters building on the last example above. - * A `tutorial `__ on image logging explaining how to debug GUI scripts written with Guibot. - * A `tutorial `__ on the most advanced types of targets and finders that rely on fallback chains and hybrid finder methods. - -Example scripts ---------------- - -There are also additional examples for very specific use cases within the *examples* folder of the repository with examples about: - - * A `similarity sampling `__ and a `match calibrating `__ script that can be used for testing out and optimizing how a certain target is matched in a controlled background image. - * A `deep transfer learning `__ script that could be used to retrain pretrained R-CNN models used for object detection. - * A `custom finder `__ and a `custom controller `__ modules that extend the original hierarchy, are mostly kept as examples of customization, and are not yet fully finished and functional. diff --git a/docs/api/conf.py b/docs/api/conf.py index 344a35b9..f9b81e66 100644 --- a/docs/api/conf.py +++ b/docs/api/conf.py @@ -82,7 +82,7 @@ # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. -language = None +language = 'en' # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: diff --git a/docs/api/generate_api.sh b/docs/api/generate_api.sh index 86ecbd0b..73c6cc1f 100644 --- a/docs/api/generate_api.sh +++ b/docs/api/generate_api.sh @@ -1,11 +1,13 @@ #!/bin/bash +# Run this script when producing new documentation to submit to RTD. + +set -e # produce rst files for the modules sphinx-apidoc -e -f -o . ../../guibot || die "No rst files could be generated" # move all rst files to source directory to integrate with RTD -mv README.rst README.rst.bak -rm -fr source README.rst +rm -fr source index.rst mkdir source || die "No source directory to move rst files to" mv *.rst source @@ -13,5 +15,4 @@ mv *.rst source make html # use README as index page for RTD (needs MD-RST compatibility) -# pandoc ../../README.md --from markdown --to rst -s -o README.rst -mv README.rst.bak README.rst +pandoc ../../README.md --from markdown --to rst -s -o index.rst diff --git a/docs/api/index.rst b/docs/api/index.rst new file mode 100644 index 00000000..f9dca0df --- /dev/null +++ b/docs/api/index.rst @@ -0,0 +1,98 @@ +Guibot |GH Actions| |Documentation Status| |CodeQL| |codecov| +============================================================= + +A tool for GUI automation using a variety of computer vision and display +control backends. + +Introduction and concepts +------------------------- + +In order to do GUI automation you usually need to solve two problems: +first, you need to have a way to control and interact with the interface +and platform you are automating and second, you need to be able to +locate the objects you are interested in on the screen. Guibot helps you +do both. + +To interact with GUIs, Guibot provides the +```controller`` `__ +module which contains a common interface for different display backends, +with methods to move the mouse, take screenshots, type characters and so +on. The backend to use will depend on how your platform is accessible, +with some backends running directly as native binaries or python scripts +on Windows, macOS, and Linux while others connecting through remote VNC +displays. + +To locate an element on the screen, you will need an image representing +the screen, a +```target`` `__ +representing the element (an image or a text in the simplest cases) and +a +```finder`` `__ +configured for the target. The finder looks for the target within the +screenshot image and returns the coordinates to the region where that +target appears. Finders, just like display controllers, are wrappers +around different backends supported by Guibot that could vary from a +simplest 1:1 pixel matching by controller backends, to template or +feature matching mix by OpenCV, to OCR and ML solutions by Tesseract and +AI frameworks. + +Finally, to bridge the gap between controlling the GUI and finding +target elements, the +```region`` `__ +module is provided. It represents a subregion of a screen and contains +methods to locate targets in this region using a choice of finder and +interact with the graphical interface using a choice of controller. + +Supported backends +------------------ + +Supported Computer Vision (CV) backends are based on + +- `OpenCV `__ + + - Template matching + - Contour matching + - Feature matching + - Haar cascade matching + - Template-feature and mixed matching + +- `Tesseract OCR `__ + + - Text matching through pytesseract, tesserocr, or OpenCV’s bindings + +- `PyTorch `__ + + - R-CNN matching through Faster R-CNN or Mask R-CNN + +- `autopy `__ + + - AutoPy matching + +Supported Display Controller (DC) backends are based on + +- `PyAutoGUI `__ +- `AutoPy `__ +- `VNCDoTool `__ +- `XDoTool `__ + +Further resources +----------------- + +Homepage: http://guibot.org + +Documentation: http://guibot.readthedocs.io + +Installation: https://github.com/intra2net/guibot/wiki/Packaging + +Issue tracking: https://github.com/intra2net/guibot/issues + +Project wiki: https://github.com/intra2net/guibot/wiki + +.. |GH Actions| image:: https://github.com/intra2net/guibot/actions/workflows/ci.yml/badge.svg + :target: https://github.com/intra2net/guibot/actions/workflows/ci.yml +.. |Documentation Status| image:: https://readthedocs.org/projects/guibot/badge/?version=latest + :target: http://guibot.readthedocs.io/en/latest/?badge=latest +.. |CodeQL| image:: https://github.com/intra2net/guibot/actions/workflows/codeql.yml/badge.svg + :target: https://github.com/intra2net/guibot/actions/workflows/codeql.yml +.. |codecov| image:: https://codecov.io/gh/intra2net/guibot/branch/master/graph/badge.svg + :target: https://codecov.io/gh/intra2net/guibot diff --git a/docs/api/source/guibot.calibrator.rst b/docs/api/source/guibot.calibrator.rst index d80a8b24..18470bc3 100644 --- a/docs/api/source/guibot.calibrator.rst +++ b/docs/api/source/guibot.calibrator.rst @@ -2,6 +2,6 @@ guibot.calibrator module ======================== .. automodule:: guibot.calibrator - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.config.rst b/docs/api/source/guibot.config.rst index ba1632ae..78b905df 100644 --- a/docs/api/source/guibot.config.rst +++ b/docs/api/source/guibot.config.rst @@ -2,6 +2,6 @@ guibot.config module ==================== .. automodule:: guibot.config - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.controller.rst b/docs/api/source/guibot.controller.rst index 22c454be..cf9d4213 100644 --- a/docs/api/source/guibot.controller.rst +++ b/docs/api/source/guibot.controller.rst @@ -2,6 +2,6 @@ guibot.controller module ======================== .. automodule:: guibot.controller - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.desktopcontrol.rst b/docs/api/source/guibot.desktopcontrol.rst index 1ca4eda7..a744ac31 100644 --- a/docs/api/source/guibot.desktopcontrol.rst +++ b/docs/api/source/guibot.desktopcontrol.rst @@ -2,6 +2,6 @@ guibot.desktopcontrol module ============================ .. automodule:: guibot.desktopcontrol - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.errors.rst b/docs/api/source/guibot.errors.rst index aeb2fce1..8601c1c9 100644 --- a/docs/api/source/guibot.errors.rst +++ b/docs/api/source/guibot.errors.rst @@ -2,6 +2,6 @@ guibot.errors module ==================== .. automodule:: guibot.errors - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.fileresolver.rst b/docs/api/source/guibot.fileresolver.rst index 8534ba73..369c7c33 100644 --- a/docs/api/source/guibot.fileresolver.rst +++ b/docs/api/source/guibot.fileresolver.rst @@ -2,6 +2,6 @@ guibot.fileresolver module ========================== .. automodule:: guibot.fileresolver - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.finder.rst b/docs/api/source/guibot.finder.rst index 49a59a73..ad0e6f98 100644 --- a/docs/api/source/guibot.finder.rst +++ b/docs/api/source/guibot.finder.rst @@ -2,6 +2,6 @@ guibot.finder module ==================== .. automodule:: guibot.finder - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.guibot.rst b/docs/api/source/guibot.guibot.rst index 10e3f261..0e3acc6c 100644 --- a/docs/api/source/guibot.guibot.rst +++ b/docs/api/source/guibot.guibot.rst @@ -2,6 +2,6 @@ guibot.guibot module ==================== .. automodule:: guibot.guibot - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.guibot_proxy.rst b/docs/api/source/guibot.guibot_proxy.rst index 55f6fb3b..7e134228 100644 --- a/docs/api/source/guibot.guibot_proxy.rst +++ b/docs/api/source/guibot.guibot_proxy.rst @@ -2,6 +2,6 @@ guibot.guibot\_proxy module =========================== .. automodule:: guibot.guibot_proxy - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.guibot_simple.rst b/docs/api/source/guibot.guibot_simple.rst index b9654ea1..9194877b 100644 --- a/docs/api/source/guibot.guibot_simple.rst +++ b/docs/api/source/guibot.guibot_simple.rst @@ -2,6 +2,6 @@ guibot.guibot\_simple module ============================ .. automodule:: guibot.guibot_simple - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.imagelogger.rst b/docs/api/source/guibot.imagelogger.rst index 5b6d945c..67afccd0 100644 --- a/docs/api/source/guibot.imagelogger.rst +++ b/docs/api/source/guibot.imagelogger.rst @@ -2,6 +2,6 @@ guibot.imagelogger module ========================= .. automodule:: guibot.imagelogger - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.inputmap.rst b/docs/api/source/guibot.inputmap.rst index 40aa845f..55c0425e 100644 --- a/docs/api/source/guibot.inputmap.rst +++ b/docs/api/source/guibot.inputmap.rst @@ -2,6 +2,6 @@ guibot.inputmap module ====================== .. automodule:: guibot.inputmap - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.location.rst b/docs/api/source/guibot.location.rst index c523612b..f8f62531 100644 --- a/docs/api/source/guibot.location.rst +++ b/docs/api/source/guibot.location.rst @@ -2,6 +2,6 @@ guibot.location module ====================== .. automodule:: guibot.location - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.match.rst b/docs/api/source/guibot.match.rst index c5c46f8f..744fea8f 100644 --- a/docs/api/source/guibot.match.rst +++ b/docs/api/source/guibot.match.rst @@ -2,6 +2,6 @@ guibot.match module =================== .. automodule:: guibot.match - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.path.rst b/docs/api/source/guibot.path.rst index 02c21e3d..2316169b 100644 --- a/docs/api/source/guibot.path.rst +++ b/docs/api/source/guibot.path.rst @@ -2,6 +2,6 @@ guibot.path module ================== .. automodule:: guibot.path - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.region.rst b/docs/api/source/guibot.region.rst index e564aa17..25f1867c 100644 --- a/docs/api/source/guibot.region.rst +++ b/docs/api/source/guibot.region.rst @@ -2,6 +2,6 @@ guibot.region module ==================== .. automodule:: guibot.region - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.rst b/docs/api/source/guibot.rst index d3f05103..addfd1b9 100644 --- a/docs/api/source/guibot.rst +++ b/docs/api/source/guibot.rst @@ -5,6 +5,7 @@ Submodules ---------- .. toctree:: + :maxdepth: 4 guibot.calibrator guibot.config @@ -28,6 +29,6 @@ Module contents --------------- .. automodule:: guibot - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/source/guibot.target.rst b/docs/api/source/guibot.target.rst index 87d75b7e..4f134792 100644 --- a/docs/api/source/guibot.target.rst +++ b/docs/api/source/guibot.target.rst @@ -2,6 +2,6 @@ guibot.target module ==================== .. automodule:: guibot.target - :members: - :undoc-members: - :show-inheritance: + :members: + :undoc-members: + :show-inheritance: diff --git a/packaging/pip_requirements_minimal.txt b/packaging/pip_requirements_minimal.txt new file mode 100644 index 00000000..75850e45 --- /dev/null +++ b/packaging/pip_requirements_minimal.txt @@ -0,0 +1,3 @@ +# minimal +Pillow==9.3.0; python_version >= '3.7' and python_version < '3.12' +Pillow==10.3.0; python_version >= '3.12'