Merge pull request #1534 from wireapp/release_2021-05-26

Release 2021-05-26
wireapp · May 26, 2021 · 101f0fa · 101f0fa
2 parents c43d804 + d65ec3c
commit 101f0fa
Show file tree

Hide file tree

Showing 5,002 changed files with 154,823 additions and 5,624 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -14,10 +14,79 @@
 
 -->
 
+# [2021-05-26]
+
+## Release Notes
+
+**Legalhold:** This release introduces a notion of "consent" to
+legalhold (LH).  If you are using LH on your site, follow the
+instructions in
+https://github.com/wireapp/wire-server/blob/814f3ebc251965ab4492f5df4d9195f3b2e0256f/docs/reference/team/legalhold.md#whitelisting-and-implicit-consent
+after the upgrade.  **Legalhold will not work as expected until you
+change `galley.conf` as described!**
+
+**SAML/SCIM:** This release introduces changes to the way `NameID` is
+processed: all identifiers are stored in lower-case and qualifiers are
+ignored.  No manual upgrade steps are necessary, but consult
+https://docs.wire.com/how-to/single-sign-on/trouble-shooting.html#theoretical-name-clashes-in-saml-nameids
+on whether you need to re-calibrate your SAML IdP / SCIM setup.
+(Reason / technical details: this change is motivated by two facts:
+(1) email casing is complicated, and industry best practice appears to
+be to ignore case information even though that is in conflict with the
+official standard documents; and (2) SCIM user provisioning does not
+allow to provide SAML NameID qualifiers, and guessing them has proven
+to be infeasible.  See
+https://github.com/wireapp/wire-server/pull/1495 for the code
+changes.)
+
+## Features
+ - [SAML/SCIM] More lenient matching of user ids (#1495)
+ - [Legalhold] Block and kick users in case of LH no_consent conflict (1:1 convs). (#1507, #1530)
+ - [Legalhold] Add legalhold status to user profile (#1522)
+ - [Legalhold] Client-supported capabilities end-point (#1503)
+ - [Legalhold] Whitelisting Teams for LH with implicit consent (#1502)
+ - [Federation] Remove OptionallyQualified data type from types-common (#1517)
+ - [Federation] Add RPC getConversations (#1493)
+ - [Federation] Prepare remote conversations: Remove Opaque/Mapped Ids, delete remote identifiers from member/user tables. (#1478)
+ - [Federation] Add schema migration for new tables (#1485)
+ - [SAML/SCIM] Normalize SAML identifiers and fix issues with duplicate account creation (#1495)
+ - Internal end-point for ejpd request processing. (#1484)
+
+## Bug fixes and other updates
+ - Fix: NewTeamMember vs. UserLegalHoldStatus (increase robustness against rogue clients) (#1496)
+
+## Documentation
+ - Fixes a typo in the wire-api documentation (#1513)
+ - Unify Swagger 2.0 docs for brig, galley and spar (#1508)
+
+## Internal changes
+ - Cleanup (no change in behavior) (#1494, #1501)
+ - wire-api: Add golden test for FromJSON instance of NewOtrMessage (#1531)
+ - Swagger/JSON cleanup (#1521, #1525)
+ - Work around a locale issue in Ormolu (#1520)
+ - Expose mock federator in wire-api-federation (#1524)
+ - Prettier looking golden tests (#1527)
+ - Refactorings, bug fixes (in tests only) (#1523)
+ - Use sed instead of yq to read yaml files (#1518)
+ - Remove zauth dependency from wire-api (#1516)
+ - Improve naming conventions federation RPC calls (#1511)
+ - Event refactoring and schema instances (#1506)
+ - Fix: regenerate cabal files. (#1510)
+ - Make DerivingVia a package default. (#1505)
+ - Port instances to schemas library (#1482)
+ - wire-api-federator: Make client tests more reliable (#1491)
+ - Remove duplicated roundtrip test (#1498)
+ - schema-profunctor: Add combinator for nonEmptyArray (#1497)
+ - Golden tests for JSON instances (#1486)
+ - galley: Convert conversation endpoints to servant (#1444, #1499)
+ - Fix Arbitrary instances and enable corresponding roundtrip tests (#1492)
+ - wire-api-fed: Mark flaky tests as pending
+ - RFC: Schemas for documented bidirectional JSON encoding (#1474)
+
 # [2021-05-04]
 
 ## Features
- - [brig] New option to use a random prekey selection strategy to remove DynamoDB dependency (#1416, #1476) 
+ - [brig] New option to use a random prekey selection strategy to remove DynamoDB dependency (#1416, #1476)
  - [brig] Ensure servant APIs are recorded by the metrics middleware (#1441)
  - [brig] Add exact handle matches from all teams in /search/contacts (#1431, #1455)
  - [brig] CSV endpoint: Add columns to output (#1452)
@@ -34,9 +103,9 @@
  - [brig] Fix FromJSON instance of ListUsersQuery (#1456)
  - [galley] Lower the limit for URL lengths for galley -> brig RPC calls (#1469)
  - [chores] Remove unused dependencies (#1424) …
- - [compilation] Stop re-compiling nginz when running integration test for unrelated changes 
- - [tooling] Use jq magic instead of bash (#1432), Add wget (#1443) 
- - [chores] Refactor Dockerfile apk installation tasks (#1448) 
+ - [compilation] Stop re-compiling nginz when running integration test for unrelated changes
+ - [tooling] Use jq magic instead of bash (#1432), Add wget (#1443)
+ - [chores] Refactor Dockerfile apk installation tasks (#1448)
  - [tooling] Script to generate token for SCIM endpoints (#1457)
  - [tooling] Ormolu script improvements (#1458)
  - [tooling] Add script to colourise test failure output (#1459)

diff --git a/Makefile b/Makefile
@@ -19,6 +19,18 @@ BUILDAH_PUSH          ?= 0
 KIND_CLUSTER_NAME     := wire-server
 BUILDAH_KIND_LOAD     ?= 1
 
+# This ensures that focused unit tests written in hspec fail. This is supposed
+# to help us avoid merging PRs with focused tests. This will not catch focused
+# integration tests as they are run in kubernetes where this Makefile doesn't
+# get executed. This is set here as the CI uses this Makefile, this could live
+# in several Makefiles we have in this repository, but there is little point of
+# doing so.
+#
+# Additionally, if stack is being used with nix, environment variables do not
+# make it into the shell where hspec is run, to tackle that this variable is
+# also exported in stack-deps.nix.
+export HSPEC_OPTIONS = --fail-on-focused
+
 default: fast
 
 init:

diff --git a/charts/brig/templates/tests/configmap.yaml b/charts/brig/templates/tests/configmap.yaml
@@ -22,6 +22,10 @@ data:
       host: cargohold
       port: 8080
 
+    gundeck:
+      host: gundeck
+      port: 8080
+
     spar:
       host: spar
       port: 8080

diff --git a/charts/nginz/values.yaml b/charts/nginz/values.yaml
@@ -338,6 +338,10 @@ nginx_conf:
       - staging
       disable_zauth: true
       basic_auth: true
+    - path: /galley-api/swagger-ui
+      disable_zauth: true
+      envs:
+      - all
     gundeck:
     - path: /push
       envs:

diff --git a/deploy/services-demo/conf/nginz/nginx.conf b/deploy/services-demo/conf/nginz/nginx.conf
@@ -361,6 +361,16 @@ http {
       proxy_pass http://galley;
     }
 
+    location /galley-api/swagger-ui {
+        include common_response_no_zauth.conf;
+        proxy_pass http://galley;
+    }
+
+    location /galley-api/swagger.json {
+        include common_response_no_zauth.conf;
+        proxy_pass http://galley;
+    }
+
     # Gundeck Endpoints
 
     rewrite ^/api-docs/push  /push/api-docs?base_url=http://127.0.0.1:8080/ break;

diff --git a/docs/reference/cassandra-schema.cql b/docs/reference/cassandra-schema.cql
@@ -101,16 +101,37 @@ CREATE TABLE galley_test.data_migration (
     AND read_repair_chance = 0.0
     AND speculative_retry = '99PERCENTILE';
 
-CREATE TABLE galley_test.team_features (
-    team_id uuid PRIMARY KEY,
-    app_lock_enforce int,
-    app_lock_inactivity_timeout_secs int,
-    app_lock_status int,
-    digital_signatures int,
-    legalhold_status int,
-    search_visibility_status int,
-    sso_status int,
-    validate_saml_emails int
+CREATE TABLE galley_test.user_remote_conv (
+    user uuid,
+    conv_remote_domain text,
+    conv_remote_id uuid,
+    PRIMARY KEY (user, conv_remote_domain, conv_remote_id)
+) WITH CLUSTERING ORDER BY (conv_remote_domain ASC, conv_remote_id ASC)
+    AND bloom_filter_fp_chance = 0.1
+    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
+    AND comment = ''
+    AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
+    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
+    AND crc_check_chance = 1.0
+    AND dclocal_read_repair_chance = 0.1
+    AND default_time_to_live = 0
+    AND gc_grace_seconds = 864000
+    AND max_index_interval = 2048
+    AND memtable_flush_period_in_ms = 0
+    AND min_index_interval = 128
+    AND read_repair_chance = 0.0
+    AND speculative_retry = '99PERCENTILE';
+
+CREATE TABLE galley_test.team (
+    team uuid PRIMARY KEY,
+    binding boolean,
+    creator uuid,
+    deleted boolean,
+    icon text,
+    icon_key text,
+    name text,
+    search_visibility int,
+    status int
 ) WITH bloom_filter_fp_chance = 0.1
     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
     AND comment = ''
@@ -179,8 +200,6 @@ CREATE TABLE galley_test.member (
     provider uuid,
     service uuid,
     status int,
-    user_remote_domain text,
-    user_remote_id uuid,
     PRIMARY KEY (conv, user)
 ) WITH CLUSTERING ORDER BY (user ASC)
     AND bloom_filter_fp_chance = 0.1
@@ -268,8 +287,6 @@ CREATE TABLE galley_test.meta (
 CREATE TABLE galley_test.user (
     user uuid,
     conv uuid,
-    conv_remote_domain text,
-    conv_remote_id uuid,
     PRIMARY KEY (user, conv)
 ) WITH CLUSTERING ORDER BY (conv ASC)
     AND bloom_filter_fp_chance = 0.1
@@ -287,17 +304,14 @@ CREATE TABLE galley_test.user (
     AND read_repair_chance = 0.0
     AND speculative_retry = '99PERCENTILE';
 
-CREATE TABLE galley_test.team (
-    team uuid PRIMARY KEY,
-    binding boolean,
-    creator uuid,
-    deleted boolean,
-    icon text,
-    icon_key text,
-    name text,
-    search_visibility int,
-    status int
-) WITH bloom_filter_fp_chance = 0.1
+CREATE TABLE galley_test.member_remote_user (
+    conv uuid,
+    user_remote_domain text,
+    user_remote_id uuid,
+    conversation_role text,
+    PRIMARY KEY (conv, user_remote_domain, user_remote_id)
+) WITH CLUSTERING ORDER BY (user_remote_domain ASC, user_remote_id ASC)
+    AND bloom_filter_fp_chance = 0.1
     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
     AND comment = ''
     AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
@@ -375,6 +389,31 @@ CREATE TABLE galley_test.conversation_codes (
     AND read_repair_chance = 0.0
     AND speculative_retry = '99PERCENTILE';
 
+CREATE TABLE galley_test.team_features (
+    team_id uuid PRIMARY KEY,
+    app_lock_enforce int,
+    app_lock_inactivity_timeout_secs int,
+    app_lock_status int,
+    digital_signatures int,
+    legalhold_status int,
+    search_visibility_status int,
+    sso_status int,
+    validate_saml_emails int
+) WITH bloom_filter_fp_chance = 0.1
+    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
+    AND comment = ''
+    AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
+    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
+    AND crc_check_chance = 1.0
+    AND dclocal_read_repair_chance = 0.1
+    AND default_time_to_live = 0
+    AND gc_grace_seconds = 864000
+    AND max_index_interval = 2048
+    AND memtable_flush_period_in_ms = 0
+    AND min_index_interval = 128
+    AND read_repair_chance = 0.0
+    AND speculative_retry = '99PERCENTILE';
+
 CREATE TABLE galley_test.user_team (
     user uuid,
     team uuid,
@@ -1112,6 +1151,7 @@ CREATE TABLE brig_test.password_reset (
 CREATE TABLE brig_test.clients (
     user uuid,
     client text,
+    capabilities set<int>,
     class int,
     cookie text,
     ip inet,
@@ -1354,6 +1394,28 @@ CREATE TABLE spar_test.bind_cookie (
     AND read_repair_chance = 0.0
     AND speculative_retry = '99PERCENTILE';
 
+CREATE TABLE spar_test.user_v2 (
+    issuer text,
+    normalized_uname_id text,
+    sso_id text,
+    uid uuid,
+    PRIMARY KEY (issuer, normalized_uname_id)
+) WITH CLUSTERING ORDER BY (normalized_uname_id ASC)
+    AND bloom_filter_fp_chance = 0.1
+    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
+    AND comment = ''
+    AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
+    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
+    AND crc_check_chance = 1.0
+    AND dclocal_read_repair_chance = 0.1
+    AND default_time_to_live = 0
+    AND gc_grace_seconds = 864000
+    AND max_index_interval = 2048
+    AND memtable_flush_period_in_ms = 0
+    AND min_index_interval = 128
+    AND read_repair_chance = 0.0
+    AND speculative_retry = '99PERCENTILE';
+
 CREATE TABLE spar_test.data_migration (
     id int,
     version int,

diff --git a/docs/reference/team/legalhold.md b/docs/reference/team/legalhold.md
@@ -125,3 +125,44 @@ New legalhold disabled event:
 ```
 
 These events are sent to the user, all team members (including admins) and connections.
+
+## Whitelisting and implicit consent
+
+This release introduces a notion of intial "consent" to legalhold
+(LH): In addition to the popup before getting exposed to LH devices
+(either by getting assigned one or by entering a conversation or
+connection with one present), users needs to grant their consent to
+even have the option of being exposed.  Until they do, they may be
+blocked from using wire by their team admin (if they are a team user),
+but they cannot be assigned a LH device, and they cannot enter
+conversations with LH devices present.
+
+For now, there is on way in the UI for the user to grant consent.
+Instead, "implict consent" can be given by the site operator for any
+team in the server configuration file `galley.yaml`:
+
+```yaml
+  featureFlags:
+    # [...]
+    legalhold: whitelist-teams-and-implicit-consent
+  legalHoldTeamsWhitelist:
+  - 14172c08-b3c8-11eb-a763-6fe8c2ea993d
+  - 162d7894-b3c8-11eb-b137-074ff453399d
+```
+
+Since consent is required for LH to work, users in teams that are not
+whitelisted cannot be assigned LH devices (pull request #1502), and
+they are blocked or removed from conversations that are exposed to LH
+devices (TODO: name the PRs where this happens).
+
+### Implementation status and future work
+
+The notion of consent is introduced to make it explicit, ie. users
+would have UI components to grant consent themselves, and there would
+be clear feedback in situations where communication is blocked for
+lack of consent, so that these situations can be resolved offline.
+
+Whitelisting and implicit consent is a short cut.  The server side
+already implements granting explicit consent, but until the UI is
+ready, site operators have the option of allowing LH to function on a
+fixed set of teams.
diff --git a/libs/api-bot/api-bot.cabal b/libs/api-bot/api-bot.cabal
@@ -4,7 +4,7 @@ cabal-version: 1.12
 --
 -- see: https://github.com/sol/hpack
 --
--- hash: 335a962994f5a2cba6af9d33696e6cf4a5a0f8e599d0e5a302e96cc15e7f1124
+-- hash: 75d171e4af4336949458672c5c82f2cc7fb00e0e5ef4823cb10692f903396df1
 
 name:           api-bot
 version:        0.4.2
@@ -36,7 +36,7 @@ library
       Paths_api_bot
   hs-source-dirs:
       src
-  default-extensions: AllowAmbiguousTypes BangPatterns ConstraintKinds DataKinds DefaultSignatures DerivingStrategies DeriveFunctor DeriveGeneric DeriveLift DeriveTraversable EmptyCase FlexibleContexts FlexibleInstances FunctionalDependencies GADTs InstanceSigs KindSignatures LambdaCase MultiParamTypeClasses MultiWayIf NamedFieldPuns NoImplicitPrelude OverloadedStrings PackageImports PatternSynonyms PolyKinds QuasiQuotes RankNTypes ScopedTypeVariables StandaloneDeriving TemplateHaskell TupleSections TypeApplications TypeFamilies TypeFamilyDependencies TypeOperators UndecidableInstances ViewPatterns
+  default-extensions: AllowAmbiguousTypes BangPatterns ConstraintKinds DataKinds DefaultSignatures DerivingStrategies DerivingVia DeriveFunctor DeriveGeneric DeriveLift DeriveTraversable EmptyCase FlexibleContexts FlexibleInstances FunctionalDependencies GADTs InstanceSigs KindSignatures LambdaCase MultiParamTypeClasses MultiWayIf NamedFieldPuns NoImplicitPrelude OverloadedStrings PackageImports PatternSynonyms PolyKinds QuasiQuotes RankNTypes ScopedTypeVariables StandaloneDeriving TemplateHaskell TupleSections TypeApplications TypeFamilies TypeFamilyDependencies TypeOperators UndecidableInstances ViewPatterns
   ghc-options: -O2 -Wall -Wincomplete-uni-patterns -Wincomplete-record-updates -Wpartial-fields -fwarn-tabs -optP-Wno-nonportable-include-path
   build-depends:
       HaskellNet >=0.5