-
Notifications
You must be signed in to change notification settings - Fork 802
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
repo-gardening: Update AI prompt for labelling #39558
base: trunk
Are you sure you want to change the base?
Conversation
Following further tests, I updated the OpenAI prompt to ensure the 'Product,' 'Platform', and 'Type' labels are added.
Thank you for your PR! When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:
This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖 The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available. 🔴 Action required: Please add missing changelog entries for the following projects: Use the Jetpack CLI tool to generate changelog entries by running the following command: Follow this PR Review Process:
Still unsure? Reach out in #jetpack-developers for guidance! |
@anomiex 👋 This is my first time submitting a PR, and any help/guidance is appreciated. This change updates only the prompt used to generate issue labels. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since you asked for feedback 🙂 regarding the formatting of the PR itself,
- I adjusted the title for you to be more descriptive.
- You should fill in the template in the description. What you added at the top is decent for the "Proposed changes" section, but "Testing instructions" would be helpful.
You'll need to add a change entry. If you haven't done a pnpm install
in your local copy of the monorepo yet, do that. Then you should be able to run pnpm jetpack changelog add
to generate one.
As for the PR itself, I'm not terribly familiar with LLM prompt engineering but I left a bunch of inline comments about stuff that would confuse me if trying to follow the instructions. 😅
@@ -66,11 +66,21 @@ ${ repoLabels | |||
.join( '\n' ) } | |||
|
|||
Analyze the issue and suggest relevant labels. Rules: | |||
# Updated prompt: Ensures 'Product', 'Platform' and 'Type' issues are always added. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the AI supposed to ignore this line because it looks like a comment?
- Use only existing labels provided. | ||
- Include 1 '[Product]' label. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like it will confuse the AI, since no "Product" labels are included in the list above and the AI was instructed to only use labels from the list above.
- Include the "[Platform] Simple" AND/OR "[Platform] Atomic" labels as appropriate. | ||
- Only if NOT present: Add 1 of the following labels based on your assessment of the issue type: '[Type] Bug', '[Type] Enhancement', or '[Type] Feature'. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here, neither "Platform" nor "Type" are included in the list above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, I wonder whether the AI will have any clue about what either of the platforms are and when they might apply.
- Briefly explain each label choice in 1 sentence. | ||
- Format your response as a JSON object, with each suggested label as a key, and your explanation of the label choice as the value. | ||
- Format your response as a JSON object, with 'labels' and 'explanations' keys.- Use only existing labels provided. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This changed instruction does not match the example response format below.
# Original prompt | ||
#- Include 1 '[Feature Group]' label. | ||
#- Include 1 to 3 '[Feature]' labels. | ||
#- Briefly explain each label choice in 1 sentence. | ||
#- Format your response as a JSON object, with each suggested label as a key, and your explanation of the label choice as the value. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More comments the AI is supposed to figure out to ignore?
@anomiex, thanks for the feedback! I submitted the changes far, far sooner than I should have. For some context, the addition of the "Product", "Platform" and "Type" label prompts was WPcom specific. Removing them negates the need for this PR, so I'm tempted to go ahead and close it. Everything else you pointed out is jump that should not have been included by me. @jeherve, testing with "Product", "Platform" and "Type" label prompts resulted in more consistently labeled issues during my testing. I need clarification on how best to proceed here, even with the context that we're aiming to sync labels across repos at some point. Do you have any suggestions that might unblock me? Thanks both! |
This PR has been marked as stale. This happened because:
If this PR is still useful, please do a [trunk merge or rebase](https://github.com/Automattic/jetpack/blob/trunk/docs/git-workflow.md#keeping-your-branch-up-to-date) and otherwise make sure it's up to date and has clear testing instructions. You may also want to ping possible reviewers in case they've forgotten about it. Please close this PR if you think it's not valid anymore — if you do, please add a brief explanation. If the PR is not updated (or at least commented on) in another month, it will be automatically closed. |
Following further tests, I updated the OpenAI prompt to ensure the 'Product,' 'Platform', and 'Type' labels are added.
Fixes #
Proposed changes:
Other information:
Jetpack product discussion
Does this pull request change what data or activity we track or use?
Testing instructions: