Add release notes for upcoming 32-bit floating points

website/docs/hyper-api/hyper_process.md

@@ -169,24 +169,34 @@ These settings control Hyper's database files.
 
 #### default_database_version
 
+Specifies the default database file format version that will be used to
+create new database files.
+Every version builds on the improvements of the previous version(s) and
+adds some new functionality, like new data types.
+
+Default value: `0`
+
+Accepted values: `0`, `1` (writing this version is deprecated in favor
+of version 2 and will be removed in a future Hyper API release), `2`, `3`, and `4`.
+
 :::note
 Newer database file format versions than the initial version `0` are
 unsupported in older product versions. This means that you can use newer
 database versions with the latest Hyper API and newer product versions
-but you cannot open them in older product versions. For example, the new
+but you cannot open them in older product versions. For example, the
 database file format version `2` can be opened in Tableau Desktop
 2020.4.15 but it cannot be opened in Tableau Desktop 2020.3. The
 complete compatibility matrix is documented in the version sections
 below.
-:::
 
-Specifies the default database file format version that will be used to
-create new database files.
+Opening a database file with an unsupported
+Tableau product version will produce an error message similar to:
 
-Default value: `0`
+"There was an error during loading database '[...]/file.hyper':
+unsupported version 3 (max supported version: 2). To open this database,
+please update your product. (error code 0AS01)"
+:::
 
-Accepted values: `0`, `1` (writing this version is deprecated in favor
-of version 2 and will be removed in a future Hyper API release), `2`
 
 ##### version 0
 
@@ -219,18 +229,11 @@ and will be removed in a future Hyper API release.
 The database file format version `1` is supported by Tableau
 Desktop/Server 2019.2.10, 2019.3.6, 2019.4.5, 2020.1.1 and newer product
 versions. It is supported by Tableau Prep 2020.2 and newer versions.
-Opening a database file with an unsupported Tableau product version will
-produce an error message similar to:
-
-"There was an error during loading database '[...]/file.hyper':
-unsupported version 1 (max supported version: 0). To open this database,
-please update your product. (error code 0AS01)"
 :::
 
 ##### version 2
 
-Database file format version `2` includes the improvements of file
-format version `1`. Additionally, it supports storing and querying
+Database file format version `2` adds support for storing and querying
 textual data with arbitrary versions of the Unicode collation tables.
 
 To create a new Hyper database file with this version, set
@@ -240,12 +243,7 @@ To create a new Hyper database file with this version, set
 The database file format version `2` is supported by Tableau
 Desktop/Server 2020.4.15, 2021.1.12, 2021.2.9, 2021.3.8, 2021.4.4,
 2022.1.2 and newer product versions. It is supported by Tableau Prep
-2022.3 and newer versions. Opening a database file with an unsupported
-Tableau product version will produce an error message similar to:
-
-"There was an error during loading database '[...]/file.hyper':
-unsupported version 2 (max supported version: 1). To open this database,
-please update your product. (error code 0AS01)"
+2022.3 and newer versions.
 :::
 
 ##### version 3
@@ -260,12 +258,24 @@ To create a new Hyper database file with this version, set
 :::note
 The database file format version `3` is supported by Tableau Desktop
 2022.4.1 and Server 2023.1 and newer product versions. It is supported by
-Tableau Prep 2022.4.1 and newer versions. Opening a database file with an
-unsupported Tableau product version will produce an error message similar to:
+Tableau Prep 2022.4.1 and newer versions. 
+:::
 
-"There was an error during loading database '[...]/file.hyper':
-unsupported version 3 (max supported version: 2). To open this database,
-please update your product. (error code 0AS01)"
+#### version 4
+Database file format version `4` was introduced to support
+persisting and reading the new 32-bit floating point type.
+
+Starting with release (#TODO), Hyper uses 32-bit floats for
+the SQL types `real`, `float4`, and `float(p)` with `p <= 24`.
+The types `double precision`, `float`, `float8`, and `float(p)` with `p >= 25`
+still use 64-bit doubles.
+
+To create a new Hyper database file with this version, set
+`default_database_version=4`.
+
+:::note
+The database file format version `4` will be supported
+in a not-yet-determined future version of Tableau.
 :::
 
 <!-- ### Experimental Settings {#experimentalsettings}
@@ -275,9 +285,7 @@ This section describes pre-release features that are not supported and
 should not be used in production code. Their interfaces, semantics, and
 performance characteristics are subject to change or they could be
 *removed at any time without prior notice.* There may also be bugs. If
-you encounter an issue, please [report
-it](https://github.com/tableau/hyper-db/issues).
-
+you encounter an issue, please [report it](https://github.com/tableau/hyper-db/issues).
 If you use an experimental feature in your test environment, we
 encourage you to enable telemetry in the Hyper API to help us improve
 these features. You can do so by passing the "send usage data to

website/docs/releases.md

@@ -28,7 +28,23 @@ In case you are wondering why all our releases start with `0.0`, read [this FAQ
 
 ### Upcoming Release
 
-* Document the new and improved database file format version 3 that was introduced in version 0.0.16123. The new format supports 128-bit numerics. Refer to [Hyper Database Settings](/docs/hyper-api/hyper_process#default_database_version) for more information.
+!!!
+
+TODO: Update release number (#TODO) in database version [database version 4](hyper-api/hyper_process#version-4) when doing this release!!!
+
+!!!
+
+* Introduced a new 32-bit floating point data type for `REAL`.
+  * Previously, `REAL` was internally mapped to 64-bit `DOUBLE PRECISION`.
+  * Introduced new [database file format version 4](hyper-api/hyper_process#version-4) to support reading and persisting the new 32-bit floats.
+  * A `CAST(… AS double precision)` is needed to store such columns in older file formats.
+* Documented the new and improved [database file format version 3](hyper-api/hyper_process#version-3) that was introduced in version 0.0.16123. The new format supports 128-bit numerics. Refer to [Hyper Database Settings](/docs/hyper-api/hyper_process#default_database_version) for more information.
+
+:::warning
+Queries using `REAL`, `FLOAT4`, or `FLOAT(p)` with `p <= 24` are now treated as 32-bit floating points.
+This can lead to different results due to the reduced precision of 32-bit floating points.
+To preserve the old behavior, you need to use the types `DOUBLE PRECISION`, `FLOAT8`, or `FLOAT(p)` with `p >= 25`. These continue to be treated as 64-bit floating points.
+:::
 
 ### 0.0.18825 [March 6, 2024]
 

website/docs/sql/datatype/index.md

@@ -14,7 +14,8 @@ Name|Aliases|Description
 `CHARACTER [ (n) ]`|`CHAR [ (n) ]`|fixed-length character string
 `CHARACTER VARYING (n)`|`VARCHAR (n)`|variable-length character string with limit
 `DATE`||calendar date (year, month, day)
-`DOUBLE PRECISION`|`FLOAT8`|double precision floating-point number (8 bytes)
+`REAL`|`FLOAT4`|single precision floating-point number (4 bytes)
+`DOUBLE PRECISION`|`FLOAT`, `FLOAT8`|double precision floating-point number (8 bytes)
 `INTEGER`|`INT`, `INT4`|signed four-byte integer
 `INTERVAL`||time span; not supported in Tableau
 `NUMERIC [ (p, s) ]`|`DECIMAL [ (p, s) ]`|exact numeric of selectable precision
@@ -25,10 +26,19 @@ Name|Aliases|Description
 `TIMESTAMP WITH TIME ZONE`|`TIMESTAMPTZ`|date and time, including time zone
 `GEOGRAPHY`||a geography object
 
+:::note
+Persisting `NUMERIC`s with a precision greater than 18 requires at least [database version 3](/docs/hyper-api/hyper_process#version-3).
+:::
+:::note
+Persisting 32-bit floating point values (e.g., type `REAL`) requires at least [database version 4](/docs/hyper-api/hyper_process#version-4).
+Up until Hyper API release [0.0.18825](/docs/releases#0.0.18825) Hyper used 64-bit floating points for all float types (i.e., also for `REAL`).
+:::
+
+
 Links to detailed documentation:
 
 ```mdx-code-block
 import DocCardList from '@theme/DocCardList';
 
 <DocCardList />
-```
+```

website/docs/sql/datatype/numeric.md

@@ -9,6 +9,7 @@ Name | Description
 `integer`|typical choice for integer: -2147483648 to +2147483647 (4 bytes)
 `bigint`|large-range integer: -9223372036854775808 to +9223372036854775807 (8 bytes)
 `numeric`|exact, fixed-length representation of numbers with decimal point: up to decimal 38 digits
+`real`|variable-precision, inexact: 6 decimal digits precision
 `double precision`|variable-precision, inexact: 15 decimal digits precision
 
 ## Integer Types {#int}
@@ -39,6 +40,10 @@ Precisions over 18 require 128-bit for internal storage. Processing
 so it is advisable to use a sensible precision for the use case at hand
 instead of always using the maximum precision by default.
 
+:::note
+128-bit numerics can only be stored using [database version 3](/docs/hyper-api/hyper_process#version-3) or newer.
+:::
+
 Both the maximum precision and the maximum scale of a `numeric` column
 can be configured. To declare a column of type `numeric` use the syntax
 `NUMERIC(precision, scale)`. The precision must be positive, the scale zero or positive.
@@ -107,9 +112,9 @@ is not supported, yet.
 
 ## Floating-Point Type {#float}
 
-The data type `double precision` is an inexact, variable-precision
-numeric type. On all currently supported platforms, these types are
-implementations of IEEE Standard 754 for Binary Floating-Point
+The data types `real` and `double precision` are inexact, variable-precision
+numeric types. On all currently supported platforms, these types are
+implementations of the IEEE Standard 754 for Binary Floating-Point
 Arithmetic.
 
 Inexact means that some values cannot be converted exactly to the
@@ -131,13 +136,20 @@ into account when using floating-point types:
     work as expected. Using difference to a small epsilon value is
     recommended instead.
 
-On all currently supported platforms, the `double precision` type has a
+On all currently supported platforms, the `real` type has a range of around
+1E-37 and 1E+37 with a precision of at least 6 digits.
+The `double precision` type has a
 range of around 1E-307 to 1E+308 with a precision of at least 15 digits.
 Values that are too large or too small will cause an error. Rounding
 might take place if the precision of an input number is too high.
 Numbers too close to zero that are not representable as distinct from
 zero will cause an underflow error.
 
+:::note
+32-bit floating points can only be stored with
+[database version 4](/docs/hyper-api/hyper_process#version-4) or newer.
+:::
+
 By default, floating point values are output in text form in their
 shortest precise decimal representation; the decimal value produced is
 closer to the true stored binary value than to any other value
@@ -166,7 +178,8 @@ with PostgresQL and many other database systems.
 
 Hyper also supports the SQL-standard notations `float` and `float(p)`
 for specifying inexact numeric types. Here, `p` specifies the minimum
-acceptable precision in *binary* digits. However, the `p` argument is
-currently ignored and all `float(p)` types are simply mapped to
-`double precision`. `float` with no precision specified is also mapped
+acceptable precision in *binary* digits.  Here, `p` specifies the minimum
+acceptable precision in binary digits. The types `float(1)` to `float(24)`
+are mapped to the `real` type. The types `float(25)` to `float(53)` map
+to `double precision`. `float` with no precision specified also maps
 to `double precision`.

website/yarn.lock

@@ -2716,21 +2716,21 @@ binary-extensions@^2.0.0:
   resolved "https://registry.yarnpkg.com/binary-extensions/-/binary-extensions-2.2.0.tgz#75f502eeaf9ffde42fc98829645be4ea76bd9e2d"
   integrity sha512-jDctJ/IVQbZoJykoeHbhXpOlNBqGNcwXJKJog42E5HDPUwQTSdjCHdihjj0DlnheQ7blbT6dHOafNAiS8ooQKA==
 
-[email protected].1:
-  version "1.20.1"
-  resolved "https://registry.yarnpkg.com/body-parser/-/body-parser-1.20.1.tgz#b1812a8912c195cd371a3ee5e66faa2338a5c668"
-  integrity sha512-jWi7abTbYwajOytWCQc37VulmWiRae5RyTpaCyDcS5/lMdtwSz5lOpDE67srw/HYe35f1z3fDQw+3txg7gNtWw==
+[email protected].2:
+  version "1.20.2"
+  resolved "https://registry.yarnpkg.com/body-parser/-/body-parser-1.20.2.tgz#6feb0e21c4724d06de7ff38da36dad4f57a747fd"
+  integrity sha512-ml9pReCu3M61kGlqoTm2umSXTlRTuGTx0bfYj+uIUKKYycG5NtSbeetV3faSU6R7ajOPw0g/J1PvK4qNy7s5bA==
   dependencies:
     bytes "3.1.2"
-    content-type "~1.0.4"
+    content-type "~1.0.5"
     debug "2.6.9"
     depd "2.0.0"
     destroy "1.2.0"
     http-errors "2.0.0"
     iconv-lite "0.4.24"
     on-finished "2.4.1"
     qs "6.11.0"
-    raw-body "2.5.1"
+    raw-body "2.5.2"
     type-is "~1.6.18"
     unpipe "1.0.0"
 
@@ -3168,7 +3168,7 @@ [email protected]:
   dependencies:
     safe-buffer "5.2.1"
 
-content-type@~1.0.4:
+content-type@~1.0.4, content-type@~1.0.5:
   version "1.0.5"
   resolved "https://registry.yarnpkg.com/content-type/-/content-type-1.0.5.tgz#8b773162656d1d1086784c8f23a54ce6d73d7918"
   integrity sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==
@@ -3188,10 +3188,10 @@ [email protected]:
   resolved "https://registry.yarnpkg.com/cookie-signature/-/cookie-signature-1.0.6.tgz#e303a882b342cc3ee8ca513a79999734dab3ae2c"
   integrity sha512-QADzlaHc8icV8I7vbaJXJwod9HWYp8uCqf1xa4OfNu1T7JVxQIrUgOWtHdNDtPiywmFbiS12VjotIXLrKM3orQ==
 
-cookie@0.5.0:
-  version "0.5.0"
-  resolved "https://registry.yarnpkg.com/cookie/-/cookie-0.5.0.tgz#d1f5d71adec6558c58f389987c366aa47e994f8b"
-  integrity sha512-YZ3GUyn/o8gfKJlnlX7g7xq4gyO6OSuhGPKaaGssGB2qgDUS0gPgtTvoyZLTt9Ab6dC4hfc9dV5arkvc/OCmrw==
+cookie@0.6.0:
+  version "0.6.0"
+  resolved "https://registry.yarnpkg.com/cookie/-/cookie-0.6.0.tgz#2798b04b071b0ecbff0dbb62a505a8efa4e19051"
+  integrity sha512-U71cyTamuh1CRNCfpGY6to28lxvNwPG4Guz/EVjgf3Jmzv0vlDp1atT9eS5dDjMYHucpHbWns6Lwf3BKz6svdw==
 
 copy-text-to-clipboard@^3.0.1:
   version "3.2.0"
@@ -3866,16 +3866,16 @@ execa@^5.0.0:
     strip-final-newline "^2.0.0"
 
 express@^4.17.3:
-  version "4.18.2"
-  resolved "https://registry.yarnpkg.com/express/-/express-4.18.2.tgz#3fabe08296e930c796c19e3c516979386ba9fd59"
-  integrity sha512-5/PsL6iGPdfQ/lKM1UuielYgv3BUoJfz1aUwU9vHZ+J7gyvwdQXFEBIEIaxeGf0GIcreATNyBExtalisDbuMqQ==
+  version "4.19.2"
+  resolved "https://registry.yarnpkg.com/express/-/express-4.19.2.tgz#e25437827a3aa7f2a827bc8171bbbb664a356465"
+  integrity sha512-5T6nhjsT+EOMzuck8JjBHARTHfMht0POzlA60WV2pMD3gyXw2LZnZ+ueGdNxG+0calOJcWKbpFcuzLZ91YWq9Q==
   dependencies:
     accepts "~1.3.8"
     array-flatten "1.1.1"
-    body-parser "1.20.1"
+    body-parser "1.20.2"
     content-disposition "0.5.4"
     content-type "~1.0.4"
-    cookie "0.5.0"
+    cookie "0.6.0"
     cookie-signature "1.0.6"
     debug "2.6.9"
     depd "2.0.0"
@@ -4062,9 +4062,9 @@ flux@^4.0.1:
     fbjs "^3.0.1"
 
 follow-redirects@^1.0.0, follow-redirects@^1.14.7:
-  version "1.15.4"
-  resolved "https://registry.yarnpkg.com/follow-redirects/-/follow-redirects-1.15.4.tgz#cdc7d308bf6493126b17ea2191ea0ccf3e535adf"
-  integrity sha512-Cr4D/5wlrb0z9dgERpUL3LrmPKVDsETIJhaCMeDfuFYcqa5bldGV6wBsAN6X/vxlXQtFBMrXdXxdL8CbDTGniw==
+  version "1.15.6"
+  resolved "https://registry.yarnpkg.com/follow-redirects/-/follow-redirects-1.15.6.tgz#7f815c0cda4249c74ff09e95ef97c23b5fd0399b"
+  integrity sha512-wWN62YITEaOpSK584EZXJafH1AGpO8RVgElfkuXbTOrPX4fIfOyEpW/CsiNd8JdYrAoOvafRTOEnvsO++qCqFA==
 
 fork-ts-checker-webpack-plugin@^6.5.0:
   version "6.5.3"
@@ -6439,10 +6439,10 @@ range-parser@^1.2.1, range-parser@~1.2.1:
   resolved "https://registry.yarnpkg.com/range-parser/-/range-parser-1.2.1.tgz#3cf37023d199e1c24d1a55b84800c2f3e6468031"
   integrity sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==
 
-[email protected].1:
-  version "2.5.1"
-  resolved "https://registry.yarnpkg.com/raw-body/-/raw-body-2.5.1.tgz#fe1b1628b181b700215e5fd42389f98b71392857"
-  integrity sha512-qqJBtEyVgS0ZmPGdCFPWJ3FreoqvG4MVQln/kCgF7Olq95IbOp0/BWyMwbdtn4VTvkM8Y7khCQ2Xgk/tcrCXig==
+[email protected].2:
+  version "2.5.2"
+  resolved "https://registry.yarnpkg.com/raw-body/-/raw-body-2.5.2.tgz#99febd83b90e08975087e8f1f9419a149366b68a"
+  integrity sha512-8zGqypfENjCIqGhgXToC8aB2r7YrBX+AQAfIPs/Mlk+BtPTztOvTS01NRW/3Eh60J+a48lt8qsCzirQ6loCVfA==
   dependencies:
     bytes "3.1.2"
     http-errors "2.0.0"
@@ -7935,9 +7935,9 @@ webpack-bundle-analyzer@^4.5.0:
     ws "^7.3.1"
 
 webpack-dev-middleware@^5.3.1:
-  version "5.3.3"
-  resolved "https://registry.yarnpkg.com/webpack-dev-middleware/-/webpack-dev-middleware-5.3.3.tgz#efae67c2793908e7311f1d9b06f2a08dcc97e51f"
-  integrity sha512-hj5CYrY0bZLB+eTO+x/j67Pkrquiy7kWepMHmUMoPsmcUaeEnQJqFzHJOyxgWlq746/wUuA64p9ta34Kyb01pA==
+  version "5.3.4"
+  resolved "https://registry.yarnpkg.com/webpack-dev-middleware/-/webpack-dev-middleware-5.3.4.tgz#eb7b39281cbce10e104eb2b8bf2b63fce49a3517"
+  integrity sha512-BVdTqhhs+0IfoeAf7EoH5WE+exCmqGerHfDM0IL096Px60Tq2Mn9MAbnaGUe6HiMa41KMCYF19gyzZmBcq/o4Q==
   dependencies:
     colorette "^2.0.10"
     memfs "^3.4.3"