-
Notifications
You must be signed in to change notification settings - Fork 65
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improve static analysis on jinja prompts (#1102)
* invalid prompt syntax now provides a compiler error * unknown variables with check / assert expressions will raise a warning <!-- ELLIPSIS_HIDDEN --> ---- > [!IMPORTANT] > Enhance static analysis for Jinja templates by adding compiler errors for invalid syntax and warnings for unknown variables, updating validation functions and tests accordingly. > > - **Behavior**: > - Invalid Jinja prompt syntax now triggers a compiler error. > - Unknown variables in `check`/`assert` expressions raise a warning. > - **Validation**: > - Updated `validate()` functions in `classes.rs`, `enums.rs`, and `functions.rs` to use `PredefinedTypes::default(JinjaContext::Prompt)`. > - Added error handling for parsing errors in `validate_template()` and `validate_expression()` in `lib.rs`. > - **Types**: > - Introduced `JinjaContext` enum in `types.rs` to differentiate between `Prompt` and `Parsing` contexts. > - Updated `PredefinedTypes` to handle different contexts. > - **Tests**: > - Added tests for malformed and valid-but-invalid expressions in `malformed_expression.baml` and `valid_but_invalid_expressions.baml`. > - Updated test cases in `test_expr.rs` and `test_stmt.rs` to reflect new validation behavior. > > <sup>This description was created by </sup>[<img alt="Ellipsis" src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=BoundaryML%2Fbaml&utm_source=github&utm_medium=referral)<sup> for 6e59805. It will automatically update as commits are pushed.</sup> <!-- ELLIPSIS_HIDDEN -->
- Loading branch information
Showing
16 changed files
with
328 additions
and
79 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
File renamed without changes.
39 changes: 39 additions & 0 deletions
39
engine/baml-lib/baml/tests/validation_files/constraints/malformed_expression.baml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,42 @@ | ||
class Foo { | ||
bar string @check(bar_check, {{ ) }}) | ||
} | ||
|
||
|
||
function FunctionName(arg:string) -> "foo" { | ||
client "openai/gpt-4o" | ||
prompt #" | ||
Your prompt here in jinja format | ||
{{ ) }} | ||
"# | ||
} | ||
|
||
|
||
function FunctionName2(arg:string) -> "foo" { | ||
client "openai/gpt-4o" | ||
prompt #" | ||
Your prompt here in jinja format | ||
{{ if foo }} | ||
{{ foo }} | ||
{{ endif }} | ||
"# | ||
} | ||
|
||
// error: Error validating: Error parsing jinja template: syntax error: unexpected `)` (in <expression>:1) | ||
// --> constraints/malformed_expression.baml:2 | ||
// | | ||
// 1 | class Foo { | ||
// 2 | bar string @check(bar_check, {{ ) }}) | ||
// | | ||
// error: Error validating: Error parsing jinja template: syntax error: unexpected `)` (in FunctionName:3) | ||
// --> constraints/malformed_expression.baml:10 | ||
// | | ||
// 9 | Your prompt here in jinja format | ||
// 10 | {{ ) }} | ||
// | | ||
// error: Error validating: Error parsing jinja template: syntax error: unexpected identifier, expected end of variable block (in FunctionName2:3) | ||
// --> constraints/malformed_expression.baml:19 | ||
// | | ||
// 18 | Your prompt here in jinja format | ||
// 19 | {{ if foo }} | ||
// | |
10 changes: 10 additions & 0 deletions
10
engine/baml-lib/baml/tests/validation_files/constraints/valid_but_invalid_expressions.baml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
class Foo { | ||
bar string @check(bar_check, {{ bar }}) | ||
} | ||
|
||
// warning: Variable `bar` does not exist. Did you mean `this`? | ||
// --> constraints/valid_but_invalid_expressions.baml:2 | ||
// | | ||
// 1 | class Foo { | ||
// 2 | bar string @check(bar_check, {{ bar }}) | ||
// | |
Oops, something went wrong.