More telemetry #1008
Merged
More telemetry #1008
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LeonOstrez
requested changes
Jun 12, 2024
core/agents/task_completer.py
Outdated
| @@ -25,5 +26,8 @@ async def run(self) -> AgentResponse: | |||
| self.current_state.get_source_index(source), | |||
| tasks, | |||
| ) | |||
| await telemetry.trace_code_event( | |||
| "task-end", {"task-num": current_task_index1, "num-iterations": len(self.current_state.iterations)} | |||
There was a problem hiding this comment.
we should either update how trace_code_event works or add data to each trace. Here are some suggestions of what data might be useful:
- number of epics (is it the initial app or feature?)
- number of all tasks (is it the last task? Could be that we have most iterations on the last task because everyone clicks "continue" until it is the last task)
- some data so we can find and analyze the db
core/agents/architect.py
Outdated
| @@ -4,6 +4,7 @@ | |||
|
|
|||
| from core.agents.base import BaseAgent | |||
| from core.agents.convo import AgentConvo | |||
| from core.agents.mixins import SystemDependencyCheckerMixin | |||
There was a problem hiding this comment.
In my opinion, we shouldn't use this as mixin. We should always go though whole flow and each agent should do it's own job.
core/agents/developer.py
Outdated
| @@ -195,6 +196,7 @@ async def breakdown_current_task(self) -> AgentResponse: | |||
| self.next_state.modified_files = {} | |||
| self.set_next_steps(response, source) | |||
| self.next_state.action = f"Task #{current_task_index + 1} start" | |||
| await telemetry.trace_code_event("task-start", {"task-num": current_task_index + 1}) | |||
There was a problem hiding this comment.
add:
- number of epics
- total number of tasks
| @@ -53,6 +54,14 @@ async def run(self) -> AgentResponse: | |||
| spec = response.text | |||
|
|
|||
| complexity = await self.check_prompt_complexity(spec) | |||
| await telemetry.trace_code_event( | |||
There was a problem hiding this comment.
add
- initial project description (before spec writer)
| @@ -129,12 +135,21 @@ async def analyze_spec(self, spec: str) -> str: | |||
| buttons={"continue": "continue"}, | |||
| ) | |||
| if confirm.cancelled or confirm.button == "continue" or confirm.text == "": | |||
| await self.telemetry.trace_code_event( | |||
| "spec-writer-questions", | |||
| { | |||
There was a problem hiding this comment.
add:
- initial user description (before spec writer)
core/agents/tech_lead.py
Outdated
| @@ -151,6 +152,7 @@ async def plan_epic(self, epic) -> AgentResponse: | |||
| } | |||
| for task in response.plan | |||
| ] | |||
| await telemetry.trace_code_event("development-plan", {"num-tasks": len(response.plan)}) | |||
There was a problem hiding this comment.
add:
- number of epics (is it the initial app or feature?)
core/agents/troubleshooter.py
Outdated
| @@ -182,6 +182,7 @@ async def get_user_feedback( | |||
| return False, False, "" | |||
|
|
|||
| if user_response.button == "loop": | |||
| await telemetry.trace_code_event("stuck-in-loop", {"clicked": True}) | |||
There was a problem hiding this comment.
add:
- number of epics (is it the initial app or feature?)
- number of all tasks (is it the last task? Could be that we have most iterations on the last task because everyone clicks "continue" until it is the last task)
- current task index
- some data so we can find and analyze the db
- number of iterations
- number of steps
- tech stack? Then we can see easily on which stack is this button used the most
|
@LeonOstrez comments addressed in the 4 new commits (recommend checking one by one) Example project and package installation status info in final telemetry event:
Extra data in trace events:
Also, instead of SpecWriter handling everything for an example project, each agent now does its part: As as side-effect, this also allows future expansion to add more/different example projects. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes system dependency checks to be done even if example project is started, and to wait for user confirmation before continuing. It also stores the information on whether the dependency is installed or not in telemetry data.
It also disables telemetry in unit tests and adds new trace events for: