-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Restrict TestSpark action to suitable code types and fix generation for a line #344
Restrict TestSpark action to suitable code types and fix generation for a line #344
Conversation
It does not solve #352 If I ask to generate test for package line in a kotlin project it throws the following exception:
|
java/src/main/kotlin/org/jetbrains/research/testspark/java/JavaPsiHelper.kt
Outdated
Show resolved
Hide resolved
As @pderakhshanfar metioned above, if you attempt generation on an
It is because we assert that a surrounding method exists. In general, we should quit using We should hide the action if there is no valid code construct under test. For a given line, we need to ensure it is within a method, class, or top-level function. The check should be performed in this order: 1) search for a surrounding method; if not found, 2) check for a surrounding class; if not found, 3) check for a top-level function. If none exist, the line is top-level, and we cannot generate tests, so the action should not appear in the right-click menu. |
The same error as here occurs if I start generation for a line public class Calc {
int a = 10; // (**)
public int sum(int a, int b) {
return a + b;
}
public int multiply(int a, int b) {
return a * b;
}
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are important comments above that identify unfixed bugs. Those should be addressed in this PR to close the issue #352.
java/src/main/kotlin/org/jetbrains/research/testspark/java/JavaPsiHelper.kt
Outdated
Show resolved
Hide resolved
langwrappers/src/main/kotlin/org/jetbrains/research/testspark/langwrappers/PsiComponents.kt
Outdated
Show resolved
Hide resolved
langwrappers/src/main/kotlin/org/jetbrains/research/testspark/langwrappers/PsiComponents.kt
Outdated
Show resolved
Hide resolved
src/main/kotlin/org/jetbrains/research/testspark/actions/TestSparkAction.kt
Outdated
Show resolved
Hide resolved
b40a077
to
3079173
Compare
Fixed. |
Fixed. |
Fixed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Introduced modifications that solve the described problems.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I must approve the PR to allow its further merge. This approval does not count.
I've found several problems connected with line generation:
class Calc {
fun sum(a: Int, b: Int): Int {
return a + b
}
} LLM is struggling with generating This problem became even more crucial if attempted to generate a test for a line inside a companion object, or it's method. LLM just don't know anything about how to call such function. Solution: includes information about the class surrounding the line if such is present.
/// 123 -- not allowed for line generation
// 123 -- allowed for line generation
fun multwice(a: Int) = a * 2 Solution idea: For each line generation, check if it includes something besides comments, open/closed brackets and allow generation only in such case |
@Hello-zoka, I agree with the 2nd and 3rd points; I'll address them in the PR or explain why they should be addressed in a different PR. At first glance, the first point does not appear to be an easy task, and it does not represent a bug; it is a lack of context. It is a stand-alone issue that should be addressed in a different PR (possibly after the upcoming release). @Hello-zoka, could you create an issue for it? My main focus, for now, is to close as many issues as possible that yield runtime exceptions and fail the plugin. The aspects regarding line-based test generation mentioned appear minor to me as very few people use line-based test generation compared to method/class-based generation (a good thing to calculate it as a metric! @arksap2002). CC: @pderakhshanfar |
@Hello-zoka |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
...c/main/kotlin/org/jetbrains/research/testspark/core/generation/llm/prompt/PromptGenerator.kt
Outdated
Show resolved
Hide resolved
src/main/kotlin/org/jetbrains/research/testspark/actions/TestSparkAction.kt
Show resolved
Hide resolved
src/main/kotlin/org/jetbrains/research/testspark/tools/llm/generation/PromptManager.kt
Outdated
Show resolved
Hide resolved
src/main/kotlin/org/jetbrains/research/testspark/tools/llm/generation/PromptManager.kt
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit (and we dont need to resolve it in this issue) TestSpark option is still available when we try to generate test for a test file:
open a test file right click and ask for test generation using TestSpark
The bug is reflected in the issue #375.
…umbers Before, the `KotlinPsiHelper` returned a 0-based line number which caused an issue with line-based test generation. The generated prompt contained a line above the selected one.
When there is no surrounding method about the selected line, we use the CUT as a context for this line. The CUT must always be present. Otherwise, the generation action should have been disabled for this line.
The line-based test generation that has a method as a context of the line now also accepts constructors of the containing class.
eb19a43
to
7d99e58
Compare
The major version increased due to the change of the public API of `PromptGenerator.generatePromptForLine` method.
Description of changes made
The PR fixes the following issues:
PsiHelper.getSurroundingLineNumber
; Kotlin used 0-based numbering which attempted the generated prompt, namely it contained a line before the selected one.See commit subjects and bodies for details.
What is left
The PR introduces a modification to
core
module (namely, extends the API ofPromptGenerator
with an overload ofgeneratePromptForLine
method that uses CUT as a line context). Therefore, I (@Vladislav0Art) need to publish a new version of thecore
module.TODOs:
4.0.0
is published.Other notes
Closes #336
Closes #352
Closes #247
Closes #249
Closes #377