feat: implement maps in BAML #797

sxlijin · 2024-07-17T23:12:15Z

Add support for maps in the BAML language. We currently only support maps with string keys.

Here's the work item list that we verified (fortunately, most of these were already pre-implemented):

integ-tests (python, ts, ruby)
parser for the compiler
- grammar
- schema-ast
parser for codemirror
- lezer
vscode: textmate grammar
diagnostics
- not sure what this means concretely
in-Rust implementation, i.e. BamlValue
value syntax
- ie how you define a map in a test
input layer (lang-native object → BamlValue)
ctx.output_format
output layer (BamlValue → lang-native object)
codegen (python, ts, ruby)

We also added:

a new entry point for validate_type_allowed where we apply refinements to the lexed types
textmate grammar snapshot testing using vscode-tmgrammar-snap

Fixes #787

vercel · 2024-07-17T23:12:18Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
baml	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 19, 2024 5:39pm

greptile-apps

PR Summary

Added support for maps with string keys in BAML language
Implemented coerce_map function in engine/baml-lib/jsonish/src/deserializer/coercer/coerce_map.rs
Updated docs/docs/snippets/supported-types.mdx with map syntax and usage
Added new test files for map types in integ-tests/baml_src/test-files/functions/input/named-args/single/
Modified engine/baml-lib/jsonish/src/deserializer/score.rs to handle map parsing errors

_{18 file(s) reviewed, 4 comment(s)}
_{Edit PR Review Bot Settings}

engine/baml-lib/baml/tests/validation_files/class/map_types.baml

greptile-apps · 2024-07-17T23:16:29Z

engine/baml-lib/baml/tests/validation_files/class/map_types.baml

+  a2 map<string, int>
+  a2 map<string, MapDummy>


Logic: Duplicate field name 'a2'. Rename one of these fields.

greptile-apps · 2024-07-17T23:16:48Z

engine/baml-lib/jsonish/src/deserializer/coercer/coerce_array.rs

+                    // TODO(vbv): this seems like a bug - it causes more penalization the deeper into an array it is?
                    Err(e) => flags.add_flag(Flag::ArrayItemParseError(i, e)),


Logic: Consider addressing this potential bug in error penalization for nested arrays

greptile-apps · 2024-07-17T23:23:55Z

...g-tests/baml_src/test-files/functions/input/named-args/single/named-map-string-to-class.baml

+    myMap {
+      "key" "example string"
+    }


Logic: Test case doesn't match function parameter type. Should be StringToClassEntry object, not string

wait isnt greptile correct here?

myMap {
"key" { word "example string" }
}

greptile-apps

PR Summary

(updates since last review)

Updated docs/docs/snippets/supported-types.mdx: Maps now only support string keys
Modified validation in engine/baml-lib/baml-core/src/validate/validation_pipeline/validations/*.rs
Added engine/baml-lib/baml/tests/validation_files/class/map_types.baml for map type testing
Introduced engine/baml-lib/jsonish/src/tests/test_maps.rs for map deserialization tests
Updated error messages in engine/baml-lib/baml/tests/validation_files/class/secure_types.baml

_{9 file(s) reviewed, 2 comment(s)}
_{Edit PR Review Bot Settings}

engine/baml-lib/jsonish/src/tests/test_maps.rs

greptile-apps

PR Summary

(updates since last review)

Added support for maps in BAML language
Implemented string-only keys for maps
Updated validation for map types in engine/baml-lib/baml-core/src/validate/validation_pipeline/validations/*.rs
Added map type testing in engine/baml-lib/baml/tests/validation_files/class/map_types.baml
Introduced map deserialization tests in engine/baml-lib/jsonish/src/tests/test_maps.rs

_{No file(s) reviewed, no comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps

PR Summary

(updates since last review)

No major changes found since last review.

_{No file(s) reviewed, no comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps

PR Summary

(updates since last review)

Implemented map types with string keys in BAML language
Updated secure_types.baml to use new map<> syntax
Replaced curly braces {} with map<> for defining map types
Added new entry point for validate_type_allowed with refinements
Introduced textmate grammar snapshot testing using vscode-tmgrammar-snap

_{1 file(s) reviewed, 1 comment(s)}
_{Edit PR Review Bot Settings}

engine/baml-lib/baml/tests/validation_files/class/secure_types.baml

hellovai · 2024-07-18T03:57:50Z

engine/baml-lib/jsonish/src/tests/test_maps.rs

@@ -0,0 +1,45 @@
+use super::*;


@sxlijin some more tests would be nice

map with new lines, for keys,

partial maps
nested partial maps
maps with ints as keys to ensure they cast to strings
maps with arrays as key to ensure they cast to strings
maps in json markup
nested maps with new lines
union of map vs class (would the map correctly let the class have it?)

engine/baml-lib/baml/tests/validation_files/class/map_types.baml

hellovai · 2024-07-18T04:02:25Z

engine/baml-lib/baml/tests/validation_files/class/secure_types.baml

@@ -17,207 +17,45 @@ class ComplexTypes {
  o (((int | string) | bool[]), (float, double) | long_long_identifier_123)
 }

-// error: Type `apple_pie` does not exist. Did you mean one of these: `ComplexTypes`, `float`, `bool`, `string`, `int`?
+// error: Error validating: This line is not a valid field or attribute definition. A valid class property looks like: 'myProperty string[] @description("This is a description")'


this error got much worse.

seems we broke the grammar :(

you reviewed an outdated version of the PR - the grammar wasn't broken, the test was still using the {key: value} map syntax

hellovai

almost, just one grammar bug it seems.

also what does the prompt look like for a map type? I don't see how the code changes for that

greptile-apps

PR Summary

(updates since last review)

Updated .envrc to include LLVM binaries in PATH
Added new test cases for map types in engine/baml-lib/baml/tests/validation_files/class/map_types.baml
Updated dependencies in engine/baml-lib/jinja/Cargo.toml to support map functionality
Introduced map_style keyword in engine/baml-lib/jinja/src/output_format/mod.rs
Added support for dictionary types in engine/language-client-codegen/src/python/templates/partial_types.py.j2

_{19 file(s) reviewed, 4 comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps · 2024-07-18T20:12:26Z

...g-tests/baml_src/test-files/functions/input/named-args/single/named-map-string-to-class.baml

+  word string
+}
+// test string
+function TestFnNamedArgsSingleMapStringToClass(myMap: int) -> map<string, StringToClassEntry> {


Logic: Function parameter type changed from map<string, StringToClassEntry> to int, which is inconsistent with the expected map structure.

greptile-apps · 2024-07-18T20:12:27Z

...g-tests/baml_src/test-files/functions/input/named-args/single/named-map-string-to-class.baml

+      "key" {
+        word "lorem ipsum"
+      }
+    }


Logic: Test case updated to match the new parameter type, but this does not align with the function's intended purpose of handling a map.

greptile-apps · 2024-07-18T20:20:01Z

integ-tests/ruby/baml_client/partial-types.rb

@@ -159,6 +162,11 @@ class Person < T::Struct
      const :name, T.nilable(String)
      const :hair_color, T.nilable(Baml::Types::Color)
    end
+    class Quantity < T::Struct
+      include T::Struct::ActsAsComparable
+      const :amount, T.nilable(T.any(T.nilable(Integer), T.nilable(Float)))


Style: Consider validating that amount is either an Integer or Float to avoid potential type issues.

greptile-apps · 2024-07-18T20:25:11Z

integ-tests/typescript/baml_client/types.ts

@@ -248,6 +248,12 @@ export interface Person {
  [key: string]: any;
 }

+export interface Quantity {
+  amount: number | number


Spelling: Duplicate type in amount: number | number. Should be number.

greptile-apps

PR Summary

(updates since last review)

Updated .envrc to include LLVM binaries in PATH.
Added docs/package.json with fern-api dependency.
Introduced map_types.baml for map validation tests.
Modified Cargo.toml in baml-lib/jinja to add strum dependency.
Added coerce_map.rs for JSON to BAML map coercion.

_{35 file(s) reviewed, 3 comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps · 2024-07-18T23:35:28Z

...g-tests/baml_src/test-files/functions/input/named-args/single/named-map-string-to-class.baml

+  word string
+}
+// test string
+function TestFnNamedArgsSingleMapStringToClass(myMap: int) -> map<string, StringToClassEntry> {


Logic: Function parameter type changed from map<string, StringToClassEntry> to int, which is inconsistent with the expected map structure.

greptile-apps · 2024-07-18T23:35:29Z

...g-tests/baml_src/test-files/functions/input/named-args/single/named-map-string-to-class.baml

+      "key" {
+        word "lorem ipsum"
+      }
+    }


Logic: Test case updated to match the new parameter type, but this does not align with the function's intended purpose of handling a map.

greptile-apps · 2024-07-18T23:49:28Z

integ-tests/typescript/baml_client/types.ts

@@ -248,6 +248,12 @@ export interface Person {
  [key: string]: any;
 }

+export interface Quantity {
+  amount: number | number


Spelling: Spelling: Duplicate type in amount: number | number. Should be number.

Suggested change

amount: number | number

amount: number

sxlijin · 2024-07-19T04:55:49Z

todo: lex maps of arbitrary parity

add support in the pest syntax dict->map rename implement maps do more

greptile-apps

PR Summary

(updates since last review)

Added engine/baml-lib/baml/tests/validation_files/class/map_types2.baml for additional map type validation tests.
Renamed tools/update_macos_llvm.sh to automate LLVM and Rust setup on macOS, ensuring WASM support.
Watch out for potential issues with Rust reinstallation affecting existing configurations.

_{2 file(s) reviewed, no comment(s)}
_{Edit PR Review Bot Settings}

vercel bot temporarily deployed to Preview July 17, 2024 23:17 Inactive

greptile-apps bot reviewed Jul 17, 2024

View reviewed changes

vercel bot temporarily deployed to Preview July 18, 2024 01:29 Inactive

vercel bot temporarily deployed to Preview July 18, 2024 01:31 Inactive