Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

About pop-up window click #171

Open
micheal888666 opened this issue Jan 27, 2025 · 2 comments
Open

About pop-up window click #171

micheal888666 opened this issue Jan 27, 2025 · 2 comments

Comments

@micheal888666
Copy link

In my automated test, my task has a scenario:

  1. Login->If there is "Receive", click "Receive"
@micheal888666
Copy link
Author

For the first question, when I singlettest use playwright, click(text="Receive"), but use browser-use he can't click
The pop-up window style is as follows:
<div class="van-dialog__content"><div data-v-3d4fafbb="" class="dialog-window"><div data-v-3d4fafbb="" class="dialog-wrapper"><img data-v-3d4fafbb="" class="" data-origin="/assets/png/succeed-83674414.png" src="/assets/png/succeed-83674414.png"><p data-v-3d4fafbb="" class="dialog-tips" style="margin-top: 10px;">Hello bro, you are finally back</p><div data-v-3d4fafbb="" class="dialog-title" style="margin-top: 0px;">I miss you so much</div><p data-v-3d4fafbb="" class="dialog-tips">I prepared an email for you</p><div data-v-3d4fafbb="" class="dialog-content"></div><div data-v-3d4fafbb="" class="dialog-btn">Receive</div></div></div></div>

@micheal888666
Copy link
Author

my script is:
`llm = ChatOpenAI(model='gpt-4o')
agent = Agent(
task='Navigate to https://localhost/#/login, and login phone:95270051, password:q123q123, if has Confirm click it.',
'if has Receive click it. Finally click Home'
llm=llm,

)`
The problem is:
He can’t find this Receive, or keeps clicking on this pattern []
Or keeps searching:
📍 Step 7
INFO [agent] ⚠ Eval: Failed - Home element not located yet.
INFO [agent] 🧠 Memory: Still need to find Home to proceed to the game page.
INFO [agent] 🎯 Next goal: Scroll further to locate the Home button.
INFO [agent] 🛠️ Action 1/1: {"scroll_down":{}}
INFO [controller] 🔍 Scrolled down the page by one page
INFO [agent]
📍 Step 8
INFO [agent] ⚠ Eval: Failed - Home element not found in visible area.
INFO [agent] 🧠 Memory: Home not visible yet; need to continue searching.
INFO [agent] 🎯 Next goal: Scroll further to locate the Home button.
INFO [agent] 🛠️ Action 1/1: {"scroll_down":{}}
INFO [controller] 🔍 Scrolled down the page by one page
INFO [agent]
📍 Step 9
INFO [agent] ⚠ Eval: Failed - The Home button has not been found yet.
INFO [agent] 🧠 Memory: Still need to locate Home to access the game page.
INFO [agent] 🎯 Next goal: Scroll down further to locate the Home button.
INFO [agent] 🛠️ Action 1/1: {"scroll_down":{}}
INFO [controller] 🔍 Scrolled down the page by one page
INFO [agent]
📍 Step 10
INFO [agent] ⚠ Eval: Failed - Unable to locate the Home button to proceed to the game page.
INFO [agent] 🧠 Memory: Searching for the Home button to proceed.
INFO [agent] 🎯 Next goal: Scroll to text 'Home' or related link to find the element.
INFO [agent] 🛠️ Action 1/1: {"scroll_to_text":{"text":"Home"}}
INFO [controller] Text 'Home' not found or not visible on page
INFO [agent]
📍 Step 11
INFO [agent] ⚠ Eval: Failed - The Home button is not visible. Cannot locate from the current page view.
INFO [agent] 🧠 Memory: Attempting to find Home button.
INFO [agent] 🎯 Next goal: Extract content to find Home or navigate the current page further.
INFO [agent] 🛠️ Action 1/1: {"extract_content":{"include_links":false}}
INFO [controller] 📄 Extracted page as text: Receive

INFO [agent]
📍 Step 12
INFO [agent] ⚠ Eval: Failed - Home element location remains unresolved.
INFO [agent] 🧠 Memory: I still need to locate the Home button to proceed with the task.
INFO [agent] 🎯 Next goal: Scroll down to see more page content.
INFO [agent] 🛠️ Action 1/1: {"scroll_down":{}}
INFO [controller] 🔍 Scrolled down the page by one page
INFO [agent]
📍 Step 13
INFO [agent] ⚠ Eval: Failed - Home element is not visible. It might be further down or not labeled clearly.
INFO [agent] 🧠 Memory: No Home identified in the current view. Need an alternative step.
INFO [agent] 🎯 Next goal: Scroll further to see more content, as Home might be below visible area.
INFO [agent] 🛠️ Action 1/1: {"scroll_down":{}}
INFO [controller] 🔍 Scrolled down the page by one page
INFO [agent]
📍 Step 14
INFO [agent] 🤷 Eval: Unknown - Home button not visible in current view, need to navigate within game options or further.
INFO [agent] 🧠 Memory: Page shows multiple game options but not Home specifically visible yet.
INFO [agent] 🎯 Next goal: Scroll further to see more options or lobby of games.
INFO [agent] 🛠️ Action 1/1: {"scroll_down":{}}
INFO [controller] 🔍 Scrolled down the page by one page
INFO [agent]
📍 Step 15
INFO [agent] ⚠ Eval: Failed - Home is still not located. There seems to be content further below.
INFO [agent] 🧠 Memory: Searching for the Home button and subsequent game entry options.
INFO [agent] 🎯 Next goal: Scroll further to reveal additional sections and potentially locate Home or related game options.
INFO [agent] 🛠️ Action 1/1: {"scroll_down":{}}
INFO [controller] 🔍 Scrolled down the page by one page
INFO [agent]
📍 Step 16
INFO [agent] ⚠ Eval: Failed - Home or game-related elements not visible.
INFO [agent] 🧠 Memory: Need to progress to the game area by finding Home or related navigation.
INFO [agent] 🎯 Next goal: Extract more content below the current view to find game-related navigation.
INFO [agent] 🛠️ Action 1/1: {"scroll_down":{}}
INFO [controller] 🔍 Scrolled down the page by one page
INFO [agent]
📍 Step 17
INFO [agent] ⚠ Eval: Failed - Unable to locate the Home element required to proceed.
INFO [agent] 🧠 Memory: Searching for Home element to continue task.
INFO [agent] 🎯 Next goal: Scroll down further to find the requested Home element or any game portal.
INFO [agent] 🛠️ Action 1/1: {"scroll_down":{}}
INFO [controller] 🔍 Scrolled down the page by one page
INFO [agent]
📍 Step 18
INFO [agent] ⚠ Eval: Failed - Home element not found or visible on scrolling so far.
INFO [agent] 🧠 Memory: None of the current elements appear to match the Home button. No clear indication to navigate.
INFO [agent] 🎯 Next goal: Continue to scroll down for additional page content to find potential Home or related buttons.
INFO [agent] 🛠️ Action 1/1: {"scroll_down":{}}
INFO [controller] 🔍 Scrolled down the page by one page

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant