-
Notifications
You must be signed in to change notification settings - Fork 53
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add docs for 2 new AI commands (#76)
* add docs for 2 new AI commands * add usage example * update
- Loading branch information
1 parent
f99e8e9
commit 166acca
Showing
5 changed files
with
122 additions
and
2 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
# assertNoDefectsWithAI | ||
|
||
{% hint style="warning" %} | ||
|
||
This is an **experimental** feature powered by LLM technology. | ||
|
||
{% endhint %} | ||
|
||
```yaml | ||
- assertNoDefectsWithAI | ||
``` | ||
Takes a screenshot, uploads it to an LLM with a pre-made prompt, and asks the | ||
model if it sees any obvious defects in the provided screenshot. Common defects | ||
include text and UI elements being cut off, overlapping, or not being centered | ||
within their containers. | ||
### Output | ||
Output is generated in HTML and JSON formats in the folder for the individual | ||
test run: | ||
``` | ||
~/.maestro | ||
└── tests | ||
├── 2024-08-20_213616 | ||
│ ├── ai-(My first flow).json | ||
│ ├── ai-(My second flow).json | ||
│ ├── ai-report-(My first flow).html | ||
│ ├── ai-report-(My second flow).html | ||
``` | ||
|
||
{% content-ref url="ai-configuration.md" %} | ||
[ai-configuration.md](ai-configuration.md) | ||
{% endcontent-ref %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
# assertWithAI | ||
|
||
{% hint style="warning" %} | ||
|
||
This is an **experimental** feature powered by LLM technology. | ||
|
||
{% endhint %} | ||
|
||
### When to use? | ||
|
||
{% hint style="warning" %} | ||
|
||
As all things generative AI, `assertWithAI` can be very helpful, but don't trust | ||
it blindly. | ||
|
||
{% endhint %} | ||
|
||
`assertWithAI` is useful when it's hard (or even impossible) to write the | ||
assertion using default assertion commands. | ||
|
||
Asserting the presence of a two-factor authentication prompt is a good example. | ||
|
||
<figure><img src="../.gitbook/assets/uber_2fa.png" alt=""><figcaption></figcaption></figure> | ||
|
||
```yaml | ||
- assertWithAI: | ||
assertion: A two-factor authentication prompt, with space for 6 digits, is visible. | ||
``` | ||
### Output | ||
Output is generated in HTML and JSON formats in the folder for the individual | ||
test run: | ||
``` | ||
~/.maestro | ||
└── tests | ||
├── 2024-08-20_213616 | ||
│ ├── ai-(My first flow).json | ||
│ ├── ai-(My second flow).json | ||
│ ├── ai-report-(My first flow).html | ||
│ ├── ai-report-(My second flow).html | ||
``` | ||
|
||
{% content-ref url="ai-configuration.md" %} | ||
[ai-configuration.md](ai-configuration.md) | ||
{% endcontent-ref %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
# AI features configuration | ||
|
||
Some commands, such as `assertWithAI` and `assertNoDiffWithAI`, use generative | ||
AI models, which are not built directly in Maestro CLI. Therefore, to use such | ||
commands, additional configuration is required. | ||
|
||
### Model | ||
|
||
The default model is the latest GPT-4o. | ||
|
||
You can configure the model to use with the `MAESTRO_CLI_AI_MODEL` env var, for example: | ||
|
||
```console | ||
export MAESTRO_CLI_AI_MODEL=claude-3-5-sonnet-20240620 | ||
``` | ||
|
||
Currently supported: | ||
– GPT family of models from OpenAI | ||
- Claude family of models from Anthropic | ||
|
||
Support for more models and providers is tracked [in this issue](https://github.com/mobile-dev-inc/maestro/issues/1957). | ||
|
||
### API key | ||
|
||
To use this command, an API key for the LLM service is required. To set it, export the | ||
`MAESTRO_CLI_AI_KEY` env var. | ||
|
||
For example, to set the key for OpenAI: | ||
|
||
```console | ||
export MAESTRO_CLI_AI_KEY=sk-4NXxdLXY4H9DZW0Vpf4lT3HuBaFJoz1zoL21eLoLRKlyXd69 | ||
``` | ||
|
||
or for Anthropic: | ||
|
||
```console | ||
export MAESTRO_CLI_AI_KEY=sk-ant-api03-U9vWi8GDrxRAvA2RL2RMCImYCQr8BFCbNOq2woeRXLNz2Iy4PbY1X2137leSm92mitI7F9IwxKIrXtXgTIzj7A-2AvgbwAA | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters