Merge pull request #1744 from wireapp/release_2021_09_08

CHANGELOG-draft.md

@@ -1,12 +1,7 @@
 THIS FILE ACCUMULATES THE RELEASE NOTES FOR THE UPCOMING RELEASE.
 
-<!--
-
 # [2021-xx-xx]
 
-[please put all changes that only affect federation into this section to unclutter the rest of the release notes.]
-[if something is both an API change and a feature, please mention it twice (you can abbreviate the second mention and add "see above").]
-
 ## Release Notes
 
 ## API Changes
@@ -19,4 +14,4 @@ THIS FILE ACCUMULATES THE RELEASE NOTES FOR THE UPCOMING RELEASE.
 
 ## Internal changes
 
--->
+## Federation changes

CHANGELOG.md

@@ -1,6 +1,49 @@
 
 <!-- if you're not the release manager, do your edits to changelog in CHANGELOG-draft.md -->
 
+# [2021-09-08]
+
+## Release Notes
+
+## API Changes
+
+* Add `POST /conversations/list/v2` (#1703)
+* Deprecate `POST /list-conversations` (#1703)
+
+## Features
+
+* Bump SFTD to 2.0.127 (#1745)
+
+## Bug fixes and other updates
+
+* Remove support for managed conversations in member removal (#1718)
+* Update the webapp to correct labeling on CBR calling (#1743)
+
+## Documentation
+
+* Document backend internals for user connections (#1717)
+* Open Update spar braindump and explain idp deletion (#1728)
+
+## Internal changes
+
+* Integration test script now displays output interactively (#1700)
+* Fixed a few issues with error response documentation in Swagger (#1707)
+* Make mapping between (team) permissions and roles more lenient (#1711)
+* The `DELETE /conversations/:cnv/members/:usr` endpoint rewritten to Servant (#1697)
+* Remove leftover auto-connect internal endpoint and code (#1716)
+* Clean up JSON golden tests (#1729, #1732, #1733)
+* Make regenerated golden tests' JSON output deterministic (#1734)
+* Import fix for snappy linker issue (#1736)
+
+## Federation changes
+
+* Added client certificate support for server to server authentication (#1682)
+* Implemented full server-to-server authentication (#1687)
+* Add an endpoint for removing a qualified user from a local conversation (#1697)
+* Refactored remote error handling in federator (#1681)
+* The update conversation membership federation endpoint takes OriginDomainHeader (#1719)
+* Added new endpoint to allow fetching conversation metadata by qualified ids (#1703)
+
 # [2021-08-27]
 
 ## Release Notes

charts/federator/values.yaml

@@ -14,7 +14,8 @@ tls:
   useSharedFederatorSecret: false
 
 resources:
-  # FUTUREWORK: come up with numbers which didn't appear out of thin air
+  # FUTUREWORK: come up with numbers which didn't appear out of thin air (See
+  # https://wearezeta.atlassian.net/browse/BE-518)
   requests:
     memory: "256Mi"
     cpu: "100m"

charts/sftd/Chart.yaml

@@ -11,4 +11,4 @@ version: 0.0.42
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.
-appVersion: 1.0.88
+appVersion: 2.0.127

charts/webapp/values.yaml

@@ -9,7 +9,7 @@ resources:
     cpu: "1"
 image:
   repository: quay.io/wire/webapp
-  tag: 2021-08-25-v0.28.24-master
+  tag: 2021-09-06-staging.3-v0.28.24-e6e306b
 service:
   https:
     externalPort: 443

docs/reference/spar-braindump.md

@@ -113,7 +113,12 @@ export IDP_ID=...
 
 Copy the new metadata file to one of your spar instances.
 
-Ssh into it.
+Ssh into it.  If you can't, [the scim
+docs](provisioning/scim-via-curl.md) explain how you can create a
+bearer token if you have the admin's login credentials.  If you follow
+that approach, you need to replace all mentions of `-H'Z-User ...'`
+with `-H'Authorization: Bearer ...'` in the following, and you won't need
+`$ADMIN_ID`, but something like `$BEARER`.
 
 There are two ways to update an IDP, described below, each with their own tradeoffs that affect users.
 
@@ -132,7 +137,7 @@ Effects:
   created for them, or they are blocked (both not what you want).
 
 ```shell
-curl -v
+curl -v \
      -XPUT http://localhost:8080/identity-providers/${IDP_ID} \
      -H"Z-User: ${ADMIN_ID}" \
      -H'Content-type: application/xml' \
@@ -165,13 +170,41 @@ Effects:
   https://github.com/wireapp/wire-team-settings/issues/3465).
 
 ```shell
-curl -v
+curl -v \
      -XPOST http://localhost:8080/identity-providers'?replaces='${IDP_ID} \
      -H"Z-User: ${ADMIN_ID}" \
      -H'Content-type: application/xml' \
      -d@"${METADATA_FILE}"
 ```
 
+
+### deleting an idp via curl
+
+Read the beginning of the last section up to "Option 1".  You need
+`ADMIN_ID` (or `BEARER`) and `IDP_ID`, but not `METADATA_FILE`.
+
+```shell
+curl -v
+     -XDELETE http://localhost:8080/identity-providers/${IDP_ID} \
+     -H"Z-User: ${ADMIN_ID}" \
+     -H'Content-type: application/json
+```
+
+If there are still users in your team with SAML credentials associated
+with this IdP, you will get an error.  You can either move these users
+elsewhere, delete them manually, or purge them implicitly during
+deletion of the IdP:
+
+```shell
+curl -v
+     -XDELETE http://localhost:8080/identity-providers/${IDP_ID}?purge=true \
+     -H"Z-User: ${ADMIN_ID}" \
+     -H'Content-type: application/json
+```
+
+Haskell code: https://github.com/wireapp/wire-server/blob/d231550f67c117b7d100c7c8c6c01b5ad13b5a7e/services/spar/src/Spar/API.hs#L217-L271
+
+
 ### setting a default SSO code
 
 To avoid having to give users the login code, a backend can also provide a default code on the endpoint `/sso/settings`.
@@ -277,13 +310,7 @@ clients; and does currently not affect deletability of users.
 
 #### delete via deleting idp
 
-[Currently](https://github.com/wireapp/wire-server/blob/010ca7e460d13160b465de24dd3982a397f94c16/services/spar/src/Spar/API.hs#L172-L187),
-deleting an IdP does not delete any user data.  In particular:
-
-- cookies of users that have authenticated via an IdP will remain valid if the IdP gets deleted.
-- if a user authenticates via an IdP that has been deleted to obtain a new cookie, the login code will not work, and the user will never be able to login again.
-- the user will still show in the team settings, and can be manually deleted from there.
-- if a new idp is registered, and a user authenticates via that idp, the old user is unreachable.  (spar will look up the wire `UserId` under the saml user id that consists partly of the id of the new IdP, come up empty, and [create a fresh user on brig](https://github.com/wireapp/wire-server/blob/010ca7e460d13160b465de24dd3982a397f94c16/services/spar/src/Spar/App.hs#L306).)
+[Currently](https://github.com/wireapp/wire-server/blob/d231550f67c117b7d100c7c8c6c01b5ad13b5a7e/services/spar/src/Spar/API.hs#L217-L271), we only have the rest API for this.  Team settings will follow with a button.
 
 
 #### user deletes herself

docs/reference/user/connection.md

@@ -59,3 +59,49 @@ For two users to be considered "connected", both A->B and B->A connections have
 Users belonging to the same team are always implicitly treated as connected, to make it easier for team members to see each other's profiles, create conversations, etc.
 
 Since there is no explicit connection state between two team members, changing the connection status (e.g. blocking a fellow team member) is impossible.
+
+# Connection backend internals
+
+In the regular case of a single backend (no federation involved), and in the easiest case of two users Alice and Adham which want to start talking, the simplified internals involving the services brig and galley and cassandra can be seen as follows: (as of 2021-08)
+
+![Connection backend internal flow](connections-flow-1-backend.png)
+
+<details>
+<summary>(To edit this diagram, copy the code in this details block to https://swimlanes.io )</summary>
+
+```
+title: Connections: (no federation)
+
+note: this is a simplified view of what happens internall inside the backend in the simple case for connection requests. For the full details refer to the code.
+
+note: Alice sends a connection request to Adham (all on backend A)
+
+order: Alice, Adham, brig, galley, cassandra
+
+Alice -> brig: POST /connections
+
+brig -> cassandra:  write in 'connections': Alice-Adham-sent
+brig -> cassandra:  write in 'connections': Adham-Alice-pending
+note brig, galley: when a connection request is sent, that also creates a conversation of type 'connection' containing only the sender:
+brig -> galley: /i/conversations/connect
+galley -> cassandra: write in conversations: ID-A-A: connection/[Alice]
+brig -> Adham: Event: new connection request from Alice
+
+...: {fas-spinner}
+
+note Alice, cassandra: Adham reacts and sends a request to accept the connection request
+
+Adham -> brig: *PUT /connections/<id>*
+brig -> cassandra: read 'connections' for Alice-Adham
+brig -> cassandra: read 'connections' for Adham-Alice
+brig -> cassandra:  write in 'connections': Alice-Adham-accept
+brig -> cassandra:  write in 'connections': Adham-Alice-accept
+
+note brig, galley: Accepting a connection also leads to the upgrade of the 'connect' conversation to a 'one2one' conversation and adds Adham to the member list
+brig -> galley: /i/conversations/:convId/accept/v2
+galley -> cassandra: write in conversations: ID-A-A: one2one/[Alice,Adham]
+brig -> Alice: Event: connection request accepted
+```
+</details>
+
+The connection / one2one conversation ID is deterministically determined using a combination of the two involved user's UUIDs, using the [addv4](https://github.com/wireapp/wire-server/blob/3b1d0c5acee58bb65d8d72e71baf68dd4c0096ae/libs/types-common/src/Data/UUID/Tagged.hs#L67-L83) function.

hack/bin/integration-test.sh

@@ -5,7 +5,5 @@ NAMESPACE=${NAMESPACE:-test-integration}
 
 echo "Running integration tests on wire-server"
 
-DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 CHART=wire-server
-helm test -n "${NAMESPACE}" "${NAMESPACE}-${CHART}" --timeout 600s |
-  "$DIR/integration-test-logs.sh"
+helm test --logs -n "${NAMESPACE}" "${NAMESPACE}-${CHART}" --timeout 600s

libs/wire-api-federation/src/Wire/API/Federation/API/Galley.hs

@@ -66,6 +66,7 @@ data Api routes = Api
       routes
         :- "federation"
         :> "update-conversation-memberships"
+        :> OriginDomainHeader
         :> ReqBody '[JSON] ConversationMemberUpdate
         :> Post '[JSON] (),
     leaveConversation ::
@@ -121,6 +122,7 @@ data RegisterConversation = MkRegisterConversation
     -- | The user that created the conversation
     rcOrigUserId :: Qualified UserId,
     -- | The qualified conversation ID
+    -- FUTUREWORK: Make this unqualified, assume the conversation is being hosted by OriginDomain
     rcCnvId :: Qualified ConvId,
     -- | The conversation type
     rcCnvType :: ConvType,
@@ -148,7 +150,10 @@ data ConversationMembersAction
 data ConversationMemberUpdate = ConversationMemberUpdate
   { cmuTime :: UTCTime,
     cmuOrigUserId :: Qualified UserId,
-    cmuConvId :: Qualified ConvId,
+    -- | The unqualified ID of the conversation where the update is happening.
+    -- The ID is local to prevent putting arbitrary domain that is different
+    -- than that of the backend making a conversation membership update request.
+    cmuConvId :: ConvId,
     -- | A list of users from a remote backend that need to be sent
     -- notifications about this change. This is required as we do not expect a
     -- non-conversation owning backend to have an indexed mapping of

libs/wire-api-federation/src/Wire/API/Federation/Client.hs

@@ -123,6 +123,7 @@ data FederationError
   | FederationNotImplemented
   | FederationNotConfigured
   | FederationCallFailure FederationClientFailure
+  deriving (Show, Eq)
 
 data FederationClientFailure = FederationClientFailure
   { fedFailDomain :: Domain,

libs/wire-api-federation/src/Wire/API/Federation/GRPC/Types.hs

@@ -56,7 +56,6 @@ data InwardResponse
   deriving (Typeable, Show, Eq, Generic)
   deriving (Arbitrary) via (GenericUniform InwardResponse)
 
--- FUTUREWORK: add roundtrip tests for these to/from schema definitions.
 instance ToSchema Router "InwardResponse" InwardResponse where
   toSchema r =
     let protoChoice = case r of

libs/wire-api-federation/test/Test/Wire/API/Federation/Golden/ConversationMemberUpdate.hs

@@ -15,7 +15,11 @@
 -- You should have received a copy of the GNU Affero General Public License along
 -- with this program. If not, see <https://www.gnu.org/licenses/>.
 
-module Test.Wire.API.Federation.Golden.ConversationMemberUpdate where
+module Test.Wire.API.Federation.Golden.ConversationMemberUpdate
+  ( testObject_ConversationMemberUpdate1,
+    testObject_ConversationMemberUpdate2,
+  )
+where
 
 import Data.Domain (Domain (Domain))
 import Data.Id (Id (Id), UserId)
@@ -49,9 +53,7 @@ testObject_ConversationMemberUpdate1 =
           (Id (fromJust (UUID.fromString "00000000-0000-0000-0000-000100000007")))
           (Domain "golden.example.com"),
       cmuConvId =
-        Qualified
-          (Id (fromJust (UUID.fromString "00000000-0000-0000-0000-000100000006")))
-          (Domain "golden2.example.com"),
+        Id (fromJust (UUID.fromString "00000000-0000-0000-0000-000100000006")),
       cmuAlreadyPresentUsers = [],
       cmuAction = ConversationMembersActionAdd ((qAlice, roleNameWireMember) :| [(qBob, roleNameWireAdmin)])
     }
@@ -65,9 +67,7 @@ testObject_ConversationMemberUpdate2 =
           (Id (fromJust (UUID.fromString "00000000-0000-0000-0000-000100000007")))
           (Domain "golden.example.com"),
       cmuConvId =
-        Qualified
-          (Id (fromJust (UUID.fromString "00000000-0000-0000-0000-000100000006")))
-          (Domain "golden2.example.com"),
+        Id (fromJust (UUID.fromString "00000000-0000-0000-0000-000100000006")),
       cmuAlreadyPresentUsers = [chad, dee],
       cmuAction = ConversationMembersActionRemove (qAlice :| [qBob])
     }

libs/wire-api-federation/test/Test/Wire/API/Federation/Golden/GoldenSpec.hs

@@ -48,6 +48,5 @@ spec =
         (LeaveConversationResponse.testObject_LeaveConversationResponse5, "testObject_LeaveConversationResponse5.json"),
         (LeaveConversationResponse.testObject_LeaveConversationResponse6, "testObject_LeaveConversationResponse6.json"),
         (LeaveConversationResponse.testObject_LeaveConversationResponse7, "testObject_LeaveConversationResponse7.json"),
-        (LeaveConversationResponse.testObject_LeaveConversationResponse8, "testObject_LeaveConversationResponse8.json"),
-        (LeaveConversationResponse.testObject_LeaveConversationResponse9, "testObject_LeaveConversationResponse9.json")
+        (LeaveConversationResponse.testObject_LeaveConversationResponse8, "testObject_LeaveConversationResponse8.json")
       ]

libs/wire-api-federation/test/Test/Wire/API/Federation/Golden/LeaveConversationResponse.hs

@@ -11,22 +11,19 @@ testObject_LeaveConversationResponse2 :: LeaveConversationResponse
 testObject_LeaveConversationResponse2 = LeaveConversationResponse $ Left RemoveFromConversationErrorRemovalNotAllowed
 
 testObject_LeaveConversationResponse3 :: LeaveConversationResponse
-testObject_LeaveConversationResponse3 = LeaveConversationResponse $ Left RemoveFromConversationErrorManagedConvNotAllowed
+testObject_LeaveConversationResponse3 = LeaveConversationResponse $ Left RemoveFromConversationErrorNotFound
 
 testObject_LeaveConversationResponse4 :: LeaveConversationResponse
-testObject_LeaveConversationResponse4 = LeaveConversationResponse $ Left RemoveFromConversationErrorNotFound
+testObject_LeaveConversationResponse4 = LeaveConversationResponse $ Left RemoveFromConversationErrorCustomRolesNotSupported
 
 testObject_LeaveConversationResponse5 :: LeaveConversationResponse
-testObject_LeaveConversationResponse5 = LeaveConversationResponse $ Left RemoveFromConversationErrorCustomRolesNotSupported
+testObject_LeaveConversationResponse5 = LeaveConversationResponse $ Left RemoveFromConversationErrorSelfConv
 
 testObject_LeaveConversationResponse6 :: LeaveConversationResponse
-testObject_LeaveConversationResponse6 = LeaveConversationResponse $ Left RemoveFromConversationErrorSelfConv
+testObject_LeaveConversationResponse6 = LeaveConversationResponse $ Left RemoveFromConversationErrorOne2OneConv
 
 testObject_LeaveConversationResponse7 :: LeaveConversationResponse
-testObject_LeaveConversationResponse7 = LeaveConversationResponse $ Left RemoveFromConversationErrorOne2OneConv
+testObject_LeaveConversationResponse7 = LeaveConversationResponse $ Left RemoveFromConversationErrorConnectConv
 
 testObject_LeaveConversationResponse8 :: LeaveConversationResponse
-testObject_LeaveConversationResponse8 = LeaveConversationResponse $ Left RemoveFromConversationErrorConnectConv
-
-testObject_LeaveConversationResponse9 :: LeaveConversationResponse
-testObject_LeaveConversationResponse9 = LeaveConversationResponse $ Left RemoveFromConversationErrorUnchanged
+testObject_LeaveConversationResponse8 = LeaveConversationResponse $ Left RemoveFromConversationErrorUnchanged

libs/wire-api-federation/test/golden/testObject_ConversationMemberUpdate1.json

@@ -24,8 +24,5 @@
             ]
         ]
     },
-    "conv_id": {
-        "domain": "golden2.example.com",
-        "id": "00000000-0000-0000-0000-000100000006"
-    }
-}
+    "conv_id": "00000000-0000-0000-0000-000100000006"
+}

libs/wire-api-federation/test/golden/testObject_ConversationMemberUpdate2.json

@@ -21,8 +21,5 @@
             }
         ]
     },
-    "conv_id": {
-        "domain": "golden2.example.com",
-        "id": "00000000-0000-0000-0000-000100000006"
-    }
-}
+    "conv_id": "00000000-0000-0000-0000-000100000006"
+}

libs/wire-api-federation/test/golden/testObject_LeaveConversationResponse3.json

@@ -1,3 +1,3 @@
 {
-    "Left": "RemoveFromConversationErrorManagedConvNotAllowed"
+    "Left": "RemoveFromConversationErrorNotFound"
 }

libs/wire-api-federation/test/golden/testObject_LeaveConversationResponse4.json

@@ -1,3 +1,3 @@
 {
-    "Left": "RemoveFromConversationErrorNotFound"
+    "Left": "RemoveFromConversationErrorCustomRolesNotSupported"
 }

libs/wire-api-federation/test/golden/testObject_LeaveConversationResponse5.json

@@ -1,3 +1,3 @@
 {
-    "Left": "RemoveFromConversationErrorCustomRolesNotSupported"
+    "Left": "RemoveFromConversationErrorSelfConv"
 }

libs/wire-api-federation/test/golden/testObject_LeaveConversationResponse6.json

@@ -1,3 +1,3 @@
 {
-    "Left": "RemoveFromConversationErrorSelfConv"
+    "Left": "RemoveFromConversationErrorOne2OneConv"
 }