Npgsql: Use `NpgsqlPoint` .NET type for marshalling `GEO_POINT` types. Explore communicating and marshalling GeoJSON types. #782

amotl · 2024-12-21T11:55:03Z

About

Explore CrateDB+Npgsql type support for geometry and geospatial types.

Status

NpgsqlPoint works well for GEO_POINT types. Thanks, @simonprickett.
Marshalling from/to .NET's GeoJSON types apparently needs to be done manually ¹.

Trivia

Please note 762b488 includes some chore adjustments to remedy warnings, which are unrelated to the main topic of the patch.

... because GEO_SHAPE types are communicated as strings? I don't actually know the reason, but would like to investigate why it doesn't work fluently, optimally/optionally using the PostGIS/GeoJSON Type Plugin? ↩

Currently, this has to be conducted "manually".

amotl · 2024-12-21T12:12:27Z

This patch offers a 70/30 test failure flakyness, where I don't know where it is coming from. It can be observed both on my machine, and on CI. In a worst case, it could be CrateDB itself, but we would need to investigate to find out. As with each of such occasion, there is a chance that a discovery of a subtle downstream flaw can help to improve CrateDB.

[xUnit.net 00:00:03.39]     demo.tests.DemoProgramTest.TestGeoJsonTypesExample [FAIL]
  Failed demo.tests.DemoProgramTest.TestGeoJsonTypesExample [173 ms]
  Error Message:
   Newtonsoft.Json.JsonReaderException : error parsing coordinates
---- Newtonsoft.Json.JsonReaderException : Unexpected character encountered while parsing value: [. Path 'coordinates', line 1, position 17.

   at demo.tests.DemoProgramTest.TestGeoJsonTypesExample() in /home/runner/work/cratedb-examples/cratedb-examples/by-language/csharp-npgsql/tests/DemoProgramTest.cs:line 222

https://github.com/crate/cratedb-examples/actions/runs/12444934016/job/34745706235?pr=782#step:7:105

amotl · 2024-12-21T17:26:31Z

by-language/csharp-npgsql/DemoTypes.cs

+        public async Task InsertGeoJsonTyped()
+        {
+            /***
+             * Verify Npgsql PostGIS/GeoJSON Type Plugin with CrateDB.
+             * https://www.npgsql.org/doc/types/geojson.html
+             *
+             * TODO: Does not work yet, because CrateDB communicates GEO_SHAPE as string?
+             *       The error message is:
+             *
+             *       System.NotSupportedException : The NpgsqlDbType 'Geometry' isn't present in your
+             *       database. You may need to install an extension or upgrade to a newer version.
+             */
+            Console.WriteLine("Running InsertGeo");
+
+            // Insert single data point.
+            await using (var cmd = new NpgsqlCommand("""
+            INSERT INTO testdrive.example (
+                "geoshape"
+            ) VALUES (
+                @geoshape
+            );
+            """, conn))
+            {
+                var point = new Point(new Position(85.43, 66.23));
+                cmd.Parameters.AddWithValue("geoshape", NpgsqlDbType.Geometry, point);
+                cmd.ExecuteNonQuery();
+            }
+
+            // Flush data.
+            await RefreshTable();
+        }


It would be sweet to gain typed GeoJSON support, like the Npgsql PostGIS/GeoJSON Type Plugin might be providing it when talking to PostGIS. I don't know why it isn't working, the error message is:

System.NotSupportedException : The NpgsqlDbType 'Geometry' isn't present in your database. You may need to install an extension or upgrade to a newer version.

Maybe it does not work, because CrateDB communicates GEO_SHAPE as exclusively as string when using the PostgrSQL wire protocol? Please advise if you see any options for improvements here.

/cc @seut, @surister, @kneth

amotl · 2024-12-21T17:31:50Z

by-language/csharp-npgsql/DemoTypes.cs

+                var point = new Point(new Position(85.43, 66.23));
+                var poly = new Polygon([
+                    new LineString([
+                        new Position(longitude: 5.0, latitude: 5.0),
+                        new Position(longitude: 5.0, latitude: 10.0),
+                        new Position(longitude: 10.0, latitude: 10.0),
+                        new Position(longitude: 10.0, latitude: 5.0),
+                        new Position(longitude: 5.0, latitude: 5.0),
+                    ])
+                ]);
+                // TODO: Can GEO_SHAPE types be directly marshalled to a .NET GeoJSON type?
+                //       Currently, `InsertGeoJsonTyped` does not work yet.
+                cmd.Parameters.AddWithValue("geoshape", NpgsqlDbType.Json, JsonConvert.SerializeObject(point));
+                cmd.ExecuteNonQuery();
+
+                cmd.Parameters.Clear();
+
+                cmd.Parameters.AddWithValue("geoshape", NpgsqlDbType.Json, JsonConvert.SerializeObject(poly));
+                cmd.ExecuteNonQuery();


What works is to communicate GeoJSON data using the NpgsqlDbType.Json type, but it needs manual marshalling like JsonConvert.SerializeObject(point).

Contrary to that, as mentioned above, the Npgsql PostGIS/GeoJSON Type Plugin enables to communicate .NET's GeoJSON types natively.

@simonprickett: Please let me know if you find any way do that already, which I might not have discovered yet. Thanks!

amotl · 2024-12-21T17:34:13Z

by-language/csharp-npgsql/DemoTypes.cs

+            // Query back data.
+            await using (var cmd = new NpgsqlCommand("SELECT * FROM testdrive.example", conn))
+            await using (var reader = cmd.ExecuteReader())
+            {
+                reader.Read();
+                // TODO: Can GEO_SHAPE types be directly marshalled to a .NET GeoJSON type?
+                //       Currently, `InsertGeoJsonTyped` does not work yet.
+                var obj = reader.GetFieldValue<JsonDocument>("geoshape");
+                var geoJsonObject = JsonConvert.DeserializeObject<Point>(obj.RootElement.ToString());
+                return (Point?) geoJsonObject;
+            }


Dito: Manual procedures are currently needed when working with .NET's native GeoJSON types.

Here, the code uses reader.GetFieldValue<JsonDocument> for retrieval, and JsonConvert.DeserializeObject<Point>(...) for unmarshalling and type casting.

amotl · 2024-12-21T17:35:51Z

by-language/csharp-npgsql/tests/DemoProgramTest.cs

+            // Validate the outcome.
+            var coords = new Point(new Position(85.43, 66.23)).Coordinates;
+            Assert.Equal(coords.Latitude, point?.Coordinates.Latitude);
+            Assert.Equal(coords.Longitude, point?.Coordinates.Longitude);


Can't directly compare GeoJSON object instances. Need to run the comparison on the individual coordinates inside. 🤷

amotl added 2 commits December 21, 2024 12:44

Npgsql: Use NpgsqlPoint .NET type for marshalling GEO_POINT types

6770795

Npgsql: Explore communicating and marshalling GeoJSON types

ba814ca

Currently, this has to be conducted "manually".

amotl requested review from simonprickett and kneth December 21, 2024 12:14

amotl marked this pull request as ready for review December 21, 2024 12:14

Npgsql: Mitigate warnings (chore)

762b488

amotl commented Dec 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Npgsql: Use `NpgsqlPoint` .NET type for marshalling `GEO_POINT` types. Explore communicating and marshalling GeoJSON types. #782

Npgsql: Use `NpgsqlPoint` .NET type for marshalling `GEO_POINT` types. Explore communicating and marshalling GeoJSON types. #782

amotl commented Dec 21, 2024 •

edited

Loading

amotl commented Dec 21, 2024 •

edited

Loading

amotl Dec 21, 2024

amotl Dec 21, 2024

amotl Dec 21, 2024

amotl Dec 21, 2024

Npgsql: Use NpgsqlPoint .NET type for marshalling GEO_POINT types. Explore communicating and marshalling GeoJSON types. #782

Are you sure you want to change the base?

Npgsql: Use NpgsqlPoint .NET type for marshalling GEO_POINT types. Explore communicating and marshalling GeoJSON types. #782

Conversation

amotl commented Dec 21, 2024 • edited Loading

About

Status

Trivia

Footnotes

amotl commented Dec 21, 2024 • edited Loading

amotl Dec 21, 2024

Choose a reason for hiding this comment

amotl Dec 21, 2024

Choose a reason for hiding this comment

amotl Dec 21, 2024

Choose a reason for hiding this comment

amotl Dec 21, 2024

Choose a reason for hiding this comment

Npgsql: Use `NpgsqlPoint` .NET type for marshalling `GEO_POINT` types. Explore communicating and marshalling GeoJSON types. #782

Npgsql: Use `NpgsqlPoint` .NET type for marshalling `GEO_POINT` types. Explore communicating and marshalling GeoJSON types. #782

amotl commented Dec 21, 2024 •

edited

Loading

amotl commented Dec 21, 2024 •

edited

Loading