Release 2022-01-18 #2055

akshaymankar · 2022-01-18T08:40:30Z

Release notes

This release introduces a mandatory federationDomain configuration setting to cargohold. Please update your values/wire-server/values.yaml to set cargohold.settings.federationDomain to the same value as the corresponding option in galley (and brig). (Servantify cargohold and add federationDomain option #1990)
The brig server config option setDefaultLocale has been replaced by setDefaultUserLocale and setDefaultTemplateLocale (see docs/reference/config-options.md for details) (Set Default User Locale #2028)
From this release onwards, the images for haskell components (brig, galley,
cargohold, etc.) will be using Ubuntu 20.04 as the base. The images are about
30-35 MB larger than the previous alpine based images. (Upgrade to GHC 8.10 and use Ubuntu 20.04 as base image for haskell docker images #1852)
Wire cloud operators: Make sure #35 is applied to all SFT servers before deploying. ([FS-266] Replace IPv4-HTTP-only Approach to SFT Server Lookup with /sft_servers_all.json #2030)

API changes

The deprecated endpoint GET /teams now ignores query parameters ids, start (SQSERVICES 1011 Servantify Gally Team API (GET /teams) #2027)
Add qualified v4 endpoints for downloading and deleting assets. The upload API is still on the same path, but the asset object it returns now contains a domain field. Note that federated behaviour is still not implemented. (Federated Cargohold API #2002)
Enable downloading assets from a remote (federated) cargohold instance via the v4 API.
The content of remote assets is returned as stream with content type
application/octet-stream.
Please refer to the Swagger API documentation for more details. (Cargohold: remote assets #2004)
Remove resumable upload API (Remove resumable upload API #1998)

Features

Allow configuring setDefaultLocale in brig using helm chart (charts/brig: Make setDefaultLocale configurable #2025)
If the guest links team feature is disabled guest links will be revoked. (SQSERVICES-547 Enforce disabling guest links when conversation is joined #1976)
Revoke guest links if feature is disabled. If the guest links team feature is disabled get /conversations/join, post /conversations/:cnv/code, and get /conversations/:cnv/code will return an error. (SQSERVICES-547 post/get conversation/:cnv/code returns error if feature disabled #1980)
Specialize setDefaultLocale to distinguish between default user locale and default template locale if the user's locale is n/a. (Set Default User Locale #2028)

Bug fixes and other updates

Fix an issue with remote asset streaming (Fix Remote Asset Downloading via Better Resource Handling #2037, Followup to remote asset download fix #2038)

Documentation

Annotate a first batch of integration and unit tests to map them to externally-facing documentation (add first mapping #1869)
Add the description to several test cases (Improve Test Documentation for BSI (SQSERVICES-1127) #1991)
Improve documentation for stern tool and helm chart (Stern/backoffice docs. #2032)

Internal changes

Replace servant-generic in Galley with a custom Named combinator (Remove uses of servant-generic in public Galley API #2022)
The Swagger documentation module is not regenerated anymore if its content is unchanged (Faster brig setup #2018)
cabal-run-integration.sh - remove Makefile indirection (cabal-run-integration.sh - remove Makefile indirection #2044)
Fix test runner for global cabal make target (Fix make c test=1 #1987)
The cabal-install-artefacts.sh script now creates the dist directory if it does not exist (Fix cabal install script #2007)
Set purge: false in fake-s3 chart (Do not purge bucket in fake-s3 #1981)
Add missing backendTwo.carghold in integration.yaml (integration.yaml add missing backendTwo.carghold #2039)
Use GHC 8.10.7 and stack 2.7.3 for builds (Upgrade to GHC 8.10 and use Ubuntu 20.04 as base image for haskell docker images #1852)
Fix non-controversial HLint issues in federator to improve code quality (Fix doubtless HLint issues in Federator #2011)
Added laws for DefaultSsoCode, Now, IdP and ScimExternalIdStore (Polysemy Spar: Laws for DefaultSsoCode, Now, IdP and ScimExternalIdStore #1940)
Moved specifications for Spar effects out of the test suite and into the library (Move Spar Polysemy Specs into spar:lib #2005)
Tag integration tests for security audit. (Documentation for Federation Test Cases #2000)
Upgrade nixpkgs pin used to provision developement dependencies (Upgrade to GHC 8.10 and use Ubuntu 20.04 as base image for haskell docker images #1852)
Servantify Galley Teams API. (SQSERVICES 1011 Servantify Galley/Team API #2008, SQSERVICES 1011 servantify gally team api (PUT /teams/:tid) #2010, SQSERVICES 1011 Servantify Gally Team API (GET /teams) #2027)
When sending an activation code, the blocked domains are checked before the whitelist. This only affects the wire SaaS staging environment (there is no whitelist configuration in prod, and blocked domains are not applicable to on-prem installations). (Return specific error when a domain is blocked #2023)
Add a helm chart that deploys restund (Add helm chart for restund #2003)
Publish restund helm chart (Publish restund helm chart #2036)
Improve optional field API in schema-profunctor (schema-profunctor: better optional field API #1988)
Migrate the public API of Cannon to Servant. (There is an internal API that is not yet migrated.) (Servantify Cannon Public endpoint #2024)
sftd chart: Add multiSFT option, remove additionalArgs option (SFT chart: Add multiSFT option, remove additionalArgs option #1992)
sftd chart: Fix quoted args for multiSFT option (Fix bug in SFT chart: multi sft args are quoted #1999)
rangedSchema does not need to be passed singletons explicitly anymore (Simplify rangedSchema function #2017)
Split cannon benchmarks and tests (Split cannon benchmarks and tests #1986)
Tag integration tests for certification. (SQSERVICES-1126 Missing Clients (BSI Docs) #1985)
Tag integration tests for certification. (Add end-tags for test tag parser (BSI audit). #2001)
New internal endpoint to configure the guest links team feature. (SQSERVICES-1169 New internal endpoint to configure the guest links team feature #1993)

Federation changes

Make federator capable of streaming responses (Streaming federator #1966)
Use Named routes for the federation API (Use named routes in Federation API #2033)
Fix Brig's configmap for SFT lookups (Fix Brig's configmap for SFT Lookup Templating #2015)
SFTD chart: provide a /sft_servers_all.json url that can be used by brig to populate /calls/config/v2 (Sftd helm chart: provide a list of all sftd urls over http(s) using sftd_disco sidecar process. #2019)
Allow making HTTP-only requests to SFTs via an IPv4 address ([FS-266] Fix SFT URL fecthing based on plain HTTP via IPv4 #2026)
Replace IPv4-HTTP-only Approach to SFT Server Lookup with /sft_servers_all.json ([FS-266] Replace IPv4-HTTP-only Approach to SFT Server Lookup with /sft_servers_all.json #2030)
Extend GET /calls/config/v2 to include all SFT servers in federation (FS-266: Add sft_servers_all to calls/config/v2 #2012)
Improve Brig's configuration for SFTs and fix a call to SFT servers (Enable Insecure Requests for SFT Lookups in a Test Environment #2014)

* Do not purge bucket in fake-s3 With `purge: true` the reaper script will delete the bucket's content when it runs, causing integration tests to fail occasionally. This fixes a very frequent flake in CI integration tests (e.g. https://bridge-ie-concourse.zinfra.io/builds/21651258).

* add first mapping * further discovered tests. * ormolu * changelog entry * fixup; add two tests * Add another test * fixup * comments * Add annotation for access token tests * Add subsystem tags in ZAuth.hs * Add subsystem tags * add testmapping to test from documentation * Update services/brig/test/integration/API/User/Auth.hs * Update services/brig/test/integration/API/User/Auth.hs * Update services/brig/test/integration/API/User/Auth.hs * Update services/brig/test/integration/API/User/Client.hs * Update services/brig/test/integration/API/User/Client.hs * Update services/brig/test/integration/API/User/Handles.hs * Update services/brig/test/integration/API/User/Auth.hs * Update services/brig/test/integration/API/User/Client.hs * Update services/brig/test/integration/API/User/Client.hs * Update services/spar/test-integration/Test/Spar/Scim/AuthSpec.hs * Update services/spar/test-integration/Test/Spar/Scim/UserSpec.hs * hi ci Co-authored-by: jschaul <[email protected]> Co-authored-by: VictorWissink <[email protected]> Co-authored-by: fisx <[email protected]> Co-authored-by: Leif Battermann <[email protected]>

* added stand alone test for checking that sending messages with missing clients should fail * naming Co-authored-by: fisx <[email protected]> * naming Co-authored-by: fisx <[email protected]> * description and more tags * changelog * wip * test for point 1. Co-authored-by: fisx <[email protected]>

* Add body argument to Service effect Also remove use of polysemy-mock from ExternalServer tests. * Remove dependency on polysemy-mocks * Federator: initial impl of service streaming * Remove unused function interpretService * Forward Content-Type header of service calls Co-authored-by: Sven Tennie <[email protected]> * Use Codensity IO as a base monad for Service This implements the bracketing pattern needed for closing a streaming response at the level of the base monad, instead of introducing CPS for the main polysemy actions. * Formatting fixes * Add basic response streaming primitive for HTTP2 Also use the same return type for the response as the one in Servant.Client, namely `ResponseF`. * RunStreamingClient instance for FederatorClient * Create test for withHTTP2Request * Fix CPS to Codensity in federator client * Test federator client streaming * Enable streaming in outward service Co-authored-by: Sven Tennie <[email protected]>

…section Add a paragraph about using VSCode in a direnv env

* Split cannon benchmarks and tests

* Moved integration test BSI tags to top * More descriptive test names

* Fix bug in SFT chart: multi sft args are quoted * Add changelog entry

* Improve documentation for tests per SQSERVICES-1127 Co-authored-by: fisx <[email protected]> * Update the changelog * Add a test: attempt to delete client via an incorrect password * Fixup Co-authored-by: fisx <[email protected]>

* added integration test and tag for BSI * clean up * More spar tests. Co-authored-by: fisx <[email protected]>

* Add end-tags for test tag parser (BSI audit). * Changelog. * Fixup * Make test identifiers unique-ish. * Fixup * Fixup

…ore (#1940)

* schema-profunctor: better optional field API * Add nullable combinator and tests

…am feature (#1993) Internal endpoints for: * set guest link team feature status * get guest link team feature status * set guest link team feature lock status

…ned (#1976)

@isovector

* Schema instances for Asset and related types * Initial stub of cargohold servant API * Servantify asset upload endpoint * Add federation domain configuration to Cargohold * Add examples to Asset swagger schema * Formatting fixes * Document postAsset errors in Swagger * Add federation domain conf to kube-ci * Migrate v3 dowload assets endpoint to Servant * Migrate v3 delete asset endpoint to Servant * Simplify implementation of ZLocalUser * Simplify implementation of MultiVerb headers * Migrate v3 token endpoints to Servant * Add bot and provider asset APIs * Specialise asset API for bots and providers * Check Z-Type header for bot and provider requests * Migrate legacy asset API to Servant * Migrate internal status endpoints to Servant * Use dependent kinds to define `PrincipalPrefix` Thanks @isovector for suggesting this approach. * Add servant prometheus metrics and remove wai * Document new federationDomain option * Add CHANGELOG entry about cargohold domain * Improve federationDomain docs Co-authored-by: jschaul <[email protected]> * Formatting fixes * Fix and simplify API paths by principal * Support asset_token query parameter Co-authored-by: Sven Tennie <[email protected]> Co-authored-by: jschaul <[email protected]>

)

This reverts commit b3730cf.

Co-authored-by: Leif Battermann <[email protected]> Co-authored-by: Matthias Fischmann <[email protected]>

* Move NowSpec into spar:lb * Move DefaultSsoCodeSpec into spar:lib * Move IdPRawMetadataStoreSpec to spar:lib * Move ScimExternalIdStoreSpec to spar:lib * Move IdPSpec into spar:lib * Pend the flakey test * CI * make format * Changelog

* Fix cabal install script The script should create the `dist` directory, if it does not exist.

Note that streaming is not yet supported for the client. Therefore, when a client for an endpoint that includes a streaming response is invoked, and a streaming response is received, the client will just fail.

- This is for making an HTTP IPv4-based request to an SFT server to get a URL that clients can use

[FS-266] Fix SFT URL fecthing based on plain HTTP via IPv4

Co-authored-by: Matthias Fischmann <[email protected]>

…ft_servers_all.json (#2030) * Generalize getRandomSFTServers function * Use SFTs provided at GET /sft_servers_all.json * Polysemyize newConfig * Fix how SFT environments are constructed - This fixes the logic of how values around the sft_servers_all feature are computed. Now it only depends on either the SFT static URL or the SFT environment (the "sft" object in Brig's configmap.yaml). - Also promote an SFT static URL integration into several unit tests that provide better coverage * Clean up helm charts and integration test configuration

Servantify Cannon Public endpoint * Clean up * Remove servant-websockets by inlining it and fixing a subtle bug regarding failing requests * Better swagger doc inspired by MultiVerb * Add changelog entry * Update comment regarding the origin of WebSocketPending * Fix review issues Co-authored-by: Paolo Capriotti <[email protected]>

* Initial implementation of remote asset download * Add RPC to check if an asset exists * Download local asset for streaming * Implement federation test for get-asset Co-authored-by: Sven Tennie <[email protected]> * Added get-asset test * Test get-asset with public and private assets * WIP: federation: test large asset download * Test stream-asset * Improve large asset test * Test streaming invalid assets * Introduce TestM in Cargohold integration tests This ReaderT gives access to the HTTP manager and the Cargohold endpoint. The wrapped monad is HTTP (to make requests). * Reintroduce tasty resource for configuration file This avoids having to read the configuration file before every test. * Add more API client utilities in tests * Use servant client for federation tests * Set up federator mocking in cargohold * Fix formatting * Assert the content of a remote asset download * Add changelog.d entry * Test 404 when fetching remote asset * Check federated requests in remote asset test * Add config to cargohold integration chart * Simple end2end remote asset download test * Remove unused `uploadRandom` function * fixup! Simple end2end remote asset download test * Enable federation in cargohold helm chart * Make asset public in end-to-end test * Set hostname for instance with mocked federator * fixup! Set hostname for instance with mocked federator * Fix local domain in fed request assertions * Fix cargohold test in federator * Use `isJust` Co-authored-by: Marko Dimjašević <[email protected]> * Simplify implementation of `getAsset` Co-authored-by: Marko Dimjašević <[email protected]> * Remove redundant `$` Co-authored-by: Marko Dimjašević <[email protected]> Co-authored-by: Sven Tennie <[email protected]> Co-authored-by: Marko Dimjašević <[email protected]>

The generic mechanism in Servant for defining API types is cumbersome and slow to compile. This commit replaces the API record for the public Galley API with a tree explicitly built using the `:<|>` operator. To make up for the lack of record labels, it also introduces a `Named` combinator for servant API types, which can be used to tag an endpoint with a name (usually a symbol). The name must match the one with which the handler is wrapped.

Co-authored-by: fisx <[email protected]>

* Add failing test * Initial fix for Cargohold streaming bug

* integration.yaml add missing backendTwo.carghold * changelog

* Add federator failure test in cargohold Also turn the exception being thrown by streaming clients into a proper Wai error.

* Add restund to CHARTS_INTEGRATION * add changelog * Add restund to CHARTS_RELEASE * update changelog

* Replace servant-generic with Named in fed API * Fix usage of fed API in brig * Add domain to all federation endpoints * Fix usage of fed API in galley * Fix usage of fed API in cargohold * Fix usage of fed API in federator

- The test makes sure that the server certificate has to have an extension marking its purpose for server authentication

This reverts commit 1cb0a65.

* cabal-run-integration.sh - remove indirection * add changelog entry * remove pattern argument * change wording of comment

…cker images (#1852) Also upgrade nixpkgs pin and stack Co-authored-by: Akshay Mankar <[email protected]>

jschaul · 2022-01-18T16:10:55Z

CHANGELOG.md

+* Add qualified v4 endpoints for downloading and deleting assets. The upload API is still on the same path, but the asset object it returns now contains a `domain` field. Note that federated behaviour is still not implemented. (#2002)
+* Enable downloading assets from a remote (federated) cargohold instance via the v4 API.
+  The content of remote assets is returned as stream with content type
+  `application/octet-stream`.
+  Please refer to the Swagger API documentation for more details. (#2004)


These sound like they could be merged into one line (together with the asset bugfix 2037 and 2038). The problem with the automated changelog is than the resulting changelog is of less good quality as multiple PRs together implementing one thing are not grouped into one changelog item, but many individual ones.

Not a blocker though. Some other things in this changelog could also be combined. (SFT stuff, a range of internal items)

It's not really a problem with the automated changelog, just a mistake. Whoever wrote the second entry should have updated the first one as well. This is going to be true for any system where the changelog is updated incrementally at every commit. But honestly, I'm not sure why we're not just using the PR description here.

pcapriotti and others added 30 commits December 13, 2021 11:57

Fix make c test=1 (#1987)

fb912ae

SFT chart: Add multiSFT option, remove additionalArgs option (#1992)

1e5cb90

Add a paragraph about using VSCode in a direnv env

7329d41

Merge pull request #1995 from wireapp/sventennie/update_vscode_setup_…

c4e6625

…section Add a paragraph about using VSCode in a direnv env

Split cannon benchmarks and tests (#1986)

70316ac

* Split cannon benchmarks and tests

Move Integration Test BSI Tags to Top (#1996)

215d34d

* Moved integration test BSI tags to top * More descriptive test names

Remove resumable upload API (#1998)

f15892e

Fix bug in SFT chart: multi sft args are quoted (#1999)

3c376f3

* Fix bug in SFT chart: multi sft args are quoted * Add changelog entry

SQSERVICES-1128 Tag Integration Tests (PROVISIONING/CHANNEL) (#1997)

0f967e0

* added integration test and tag for BSI * clean up * More spar tests. Co-authored-by: fisx <[email protected]>

Add end-tags for test tag parser (BSI audit). (#2001)

b50e906

* Add end-tags for test tag parser (BSI audit). * Changelog. * Fixup * Make test identifiers unique-ish. * Fixup * Fixup

Polysemy Spar: Laws for DefaultSsoCode, Now, IdP and ScimExternalIdSt…

4e1b8fb

…ore (#1940)

schema-profunctor: better optional field API (#1988)

e4503ff

* schema-profunctor: better optional field API * Add nullable combinator and tests

SQSERVICES-1169 New internal endpoint to configure the guest links te…

6cff0a9

…am feature (#1993) Internal endpoints for: * set guest link team feature status * get guest link team feature status * set guest link team feature lock status

SQSERVICES-547 Enforce disabling guest links when conversation is joi…

f99f1e4

…ned (#1976)

Update polysemy-check version in cabal.freeze (#2006)

a176340

post/get conversation/:cnv/code returns error if feature disabled (#1980

dd36838

)

wip

b3730cf

Revert "wip"

dcd754e

This reverts commit b3730cf.

Documentation for Federation Test Cases (#2000)

04fde5a

Co-authored-by: Leif Battermann <[email protected]> Co-authored-by: Matthias Fischmann <[email protected]>

Move Spar Polysemy Specs into spar:lib (#2005)

9285b74

* Move NowSpec into spar:lb * Move DefaultSsoCodeSpec into spar:lib * Move IdPRawMetadataStoreSpec to spar:lib * Move ScimExternalIdStoreSpec to spar:lib * Move IdPSpec into spar:lib * Pend the flakey test * CI * make format * Changelog

Fix cabal install script (#2007)

1e6843a

* Fix cabal install script The script should create the `dist` directory, if it does not exist.

Add streaming support to MultiVerb

b631191

Note that streaming is not yet supported for the client. Therefore, when a client for an endpoint that includes a streaming response is invoked, and a streaming response is received, the client will just fail.

Implement qualified asset download endpoint

3e4d728

Add qualified asset delete endpoint

254ec8f

mdimjasevic and others added 25 commits January 5, 2022 10:19

Use plain HTTP instead of HTTPS

401c5a3

- This is for making an HTTP IPv4-based request to an SFT server to get a URL that clients can use

Add a proper integration test for sft_servers_all

424e5fd

Add a changelog

5bcdddb

charts/brig: Make setDefaultLocale configurable (#2025)

425ee23

Merge pull request #2026 from wireapp/fs-266/fix-ipv4-plain-http

f8bc6a3

[FS-266] Fix SFT URL fecthing based on plain HTTP via IPv4

SQSERVICES 1011 Servantify Gally Team API (GET /teams) (#2027)

be24d66

Co-authored-by: Matthias Fischmann <[email protected]>

Add helm chart for restund (#2003)

312afa5

Set Default User Locale (#2028)

8057636

Co-authored-by: fisx <[email protected]>

Stern/backoffice docs. (#2032)

067dd56

Fix Remote Asset Downloading via Better Resource Handling (#2037)

82436c4

* Add failing test * Initial fix for Cargohold streaming bug

integration.yaml add missing backendTwo.carghold (#2039)

b9844bb

* integration.yaml add missing backendTwo.carghold * changelog

Followup to remote asset download fix (#2038)

3c683ed

* Add federator failure test in cargohold Also turn the exception being thrown by streaming clients into a proper Wai error.

Publish restund helm chart (#2036)

8015566

* Add restund to CHARTS_INTEGRATION * add changelog * Add restund to CHARTS_RELEASE * update changelog

Use named routes in Federation API (#2033)

c63de7e

* Replace servant-generic with Named in fed API * Fix usage of fed API in brig * Add domain to all federation endpoints * Fix usage of fed API in galley * Fix usage of fed API in cargohold * Fix usage of fed API in federator

Tag a negative test case for X509v3 Extended Key Usage

1cb0a65

- The test makes sure that the server certificate has to have an extension marking its purpose for server authentication

Revert "Tag a negative test case for X509v3 Extended Key Usage"

5590c8c

This reverts commit 1cb0a65.

cabal-run-integration.sh - remove Makefile indirection (#2044)

6fea757

* cabal-run-integration.sh - remove indirection * add changelog entry * remove pattern argument * change wording of comment

Upgrade to GHC 8.10 and use Ubuntu 20.04 as base image for haskell do…

194f9da

…cker images (#1852) Also upgrade nixpkgs pin and stack Co-authored-by: Akshay Mankar <[email protected]>

nix/cryptobox: Fix cargoSha256 (#2050)

b02e8a6

Add release notes for #2030 (#2049)

096c48c

Changelog: Add entry for [2020-01-18]

653adbb

akshaymankar changed the title ~~Release 2022 01 18~~ Release 2022-01-18 Jan 18, 2022

jschaul approved these changes Jan 18, 2022

View reviewed changes

Update CHANGELOG

2fd80aa

akshaymankar merged commit b97b966 into master Jan 19, 2022

akshaymankar deleted the release_2022_01_18 branch January 19, 2022 09:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release 2022-01-18 #2055

Release 2022-01-18 #2055

akshaymankar commented Jan 18, 2022

jschaul Jan 18, 2022

pcapriotti Jan 19, 2022

Release 2022-01-18 #2055

Release 2022-01-18 #2055

Conversation

akshaymankar commented Jan 18, 2022

Release notes

API changes

Features

Bug fixes and other updates

Documentation

Internal changes

Federation changes

jschaul Jan 18, 2022

Choose a reason for hiding this comment

pcapriotti Jan 19, 2022

Choose a reason for hiding this comment