Merge pull request #1832 from wireapp/release-2021-10-01

.github/pull_request_template.md

@@ -2,11 +2,13 @@
 
  - [ ] The **PR Title** explains the impact of the change.
  - [ ] The **PR description** provides context as to why the change should occur and what the code contributes to that effect. This could also be a link to a JIRA ticket or a Github issue, if there is one.
- - [ ] If end-points have been added or changed: the **endpoint / config-flag checklist** (see Wire-employee only backend [wiki page](https://github.com/zinfra/backend-wiki/wiki/Checklists)) has been followed.
- - [ ] If a schema migration has been added, I ran **`make git-add-cassandra-schema`** to update the cassandra schema documentation.
+ - [ ] If HTTP endpoint paths have been added or renamed, the **endpoint / config-flag checklist** (see Wire-employee only backend [wiki page](https://github.com/zinfra/backend-wiki/wiki/Checklists)) has been followed.
+ - [ ] If a cassandra schema migration has been added, I ran **`make git-add-cassandra-schema`** to update the cassandra schema documentation.
  - [ ] **changelog.d** contains the following bits of information:
    - [ ] A file with the changelog entry in one or more suitable sub-sections. The sub-sections are marked by directories inside `changelog.d`.
-   - [ ] If /a: measures to be taken by instance operators.
-   - [ ] If /a: list of cassandra migrations.
-   - [ ] If public end-points have been changed or added: does nginz need upgrade?
+   - [ ] If new config options introduced: added usage description under docs/reference/config-options.md
+   - [ ] If new config options introduced: recommended measures to be taken by on-premise instance operators.
+   - [ ] If a cassandra schema migration is backwards incompatible (see also [these docs](https://github.com/wireapp/wire-server/blob/develop/docs/developer/cassandra-interaction.md#cassandra-schema-migrations)), measures to be taken by on-premise instance operators are explained.
+   - [ ] If a data migration (not schema migration) introduced: measures to be taken by on-premise instance operators.
+   - [ ] If public end-points have been changed or added: does nginz need un upgrade?
    - [ ] If internal end-points have been added or changed: which services have to be deployed in a specific order?

CHANGELOG.md

@@ -1,5 +1,98 @@
 <!-- if you're not the release manager, do your edits to changelog under CHANGELOG.d/ -->
 
+# [2021-10-01]
+
+## Release notes
+
+* Deploy brig before galley (#1811, #1818)
+* The conference call initiation feature can now be configured for personal accounts in `brig.yaml`.  `enabled` is the default and the previous behavior.  If you want to change that, read [/docs/reference/config-options.md#conference-calling-1](https://github.com/wireapp/wire-server/blob/develop/docs/reference/config-options.md#conference-calling-1) (#1811, #1818)
+* Only if you are an early adopter of multi-team IdP issuers on release [2021-09-14](https://github.com/wireapp/wire-server/releases/tag/v2021-09-14): note that the [query parameter for IdP creation has changed](https://github.com/wireapp/wire-server/pull/1763/files#diff-bd66bf2f3a2445e08650535a431fc33cc1f6a9e0763c7afd9c9d3f2d67fac196).  This only affects future calls to this one end-point. (#1763)
+* For wire.com cloud operators: reminder to also deploy nginz. (No special action needed for on-premise operators) (#1773)
+
+## API changes
+
+* Add endpoint `POST /connections/:domain/:userId` to create a connection (#1773)
+* Deprecate `PUT /conversations/:cnv/access` endpoint (#1807)
+* Deprecate `PUT /conversations/:cnv/message-timer` endpoint (#1780)
+* Deprecate `PUT /conversations/:cnv/members/:usr` endpoint (#1784)
+* Deprecate `PUT /conversations/:cnv/receipt-mode` endpoint (#1797)
+* Add endpoint `GET /connections/:domain/:userId` to get a single connection (#1773)
+* Add `POST /list-connections` endpoint to get connections (#1773)
+* Add qualified endpoint for updating conversation access (#1807)
+* Add qualified endpoint for updating message timer (#1780)
+* Add qualified endpoint for updating conversation members (#1784)
+* Add qualified endpoint for updating receipt mode (#1797)
+* Add endpoint `PUT /connections/:domain/:userId` to update a connection (#1773)
+
+## Features
+
+* Helm charts to deploy [ldap-scim-bridge](https://github.com/wireapp/ldap-scim-bridge) (#1709)
+* Per-account configuration of conference call initiation (details: /docs/reference/config-options.md#conference-calling-1) (#1811, #1818)
+
+## Bug fixes and other updates
+
+* An attempt to create a 3rd IdP with the same issuer was triggering an exception. (#1763)
+* When a user was auto-provisioned into two teams under the same pair of `Issuer` and `NameID`, they where directed into the wrong team, and not rejected. (#1763)
+
+## Documentation
+
+* Expand documentation of `conversations/list-ids` endpoint (#1779)
+* Add documentation of the multi-table paging abstraction (#1803)
+* Document how to use IdP issuers for multiple teams (#1763)
+* All named Swagger schemas are now displayed in the Swagger UI (#1802)
+
+## Internal changes
+
+* Abstract out multi-table-pagination used in list conversation-ids endpoint (#1788)
+* Testing: rewrite monadic to applicative style generators (#1782)
+* Add a test checking that creating conversations of exactly the size limit is allowed (#1820)
+* Rewrite the DELETE /self endpoint to Servant (#1771)
+* Fix conversation generator in mapping test (#1778)
+* Polysemize spar (#1806, #1787, #1793, #1814, #1792, #1781, #1786, #1810, #1816, #1815)
+* Refactored a few functions dealing with conversation updates, in an attempt to
+  make the conversation update code paths more uniform, and also reduce special
+  cases for local and remote objects. (#1801)
+* Merged http2-client fixes as mentioned in the comments of #1703 (#1809)
+* Some executables now have a runtime dependency on ncurses (#1791)
+* Minor changes around SAML and multi-team Issuers.
+  - Change query param to not contain `-`, but `_`.  (This is considered an internal change because the feature has been release in the last release, but only been documented in this one.)
+  - Haddocks.
+  - Simplify code.
+  - Remove unnecessary calls to cassandra. (#1763)
+* Clean up JSON Golden Tests (Part 6) (#1769)
+* Remove explicit instantiations of ErrorDescription (#1794)
+* Remove one flaky integration test about ordering of search results (#1798)
+* Report all failures in JSON golden tests in a group at once (#1746)
+* Convert the `PUT /conversations/:cnv/access` endpoint to Servant (#1807)
+* Move /connections/* endpoints to Servant (#1770)
+* Servantify Galley's DELETE /i/user endpoint (#1772)
+* Convert the `PUT /conversations/:cnv/message-timer` endpoint to Servant (#1780)
+* Convert the `PUT /conversations/:cnv/members/:usr` endpoint to Servant (#1796)
+* Convert the `PUT /conversations/:cnv/receipt-mode` endpoint to Servant (#1797)
+* Expose wire.com internal EJDP process to backoffice/stern. (#1831)
+* Update configurable boolean team feature list in backoffice/stern. (#1829)
+* Handle upper/lower case more consistently in scim and rich-info data. (#1754)
+
+## Federation changes
+
+* Add value for verification depth of client certificates in federator ingress (#1812)
+* Document federation API conventions and align already existing APIs (#1765)
+* Notify remote users when a conversation access settings are updated (#1808)
+* Notify remote users when a conversation member role is updated (#1785)
+* Notify remote users when a conversation message timer is updated (#1783)
+* Notify remote users when a conversation is renamed (#1767)
+* Make sure that only users that are actually part of a conversation get notified about updates in the conversation metadata (#1767)
+* Notify remote users when a conversation receipt mode is updated (#1801)
+* Implement updates to remote members (#1785)
+* Make conversation ID of the on-conversation-created RPC unqualified (#1766)
+* 4 endpoints for create/update/get/list connections designed for remote users in mind. So far, the implementation only works for local users (actual implementation will come as a follow-up) (#1773)
+* The returned `connection` object now has a `qualified_to` field with the domain of the (potentially remote) user. (#1773)
+* Add migration for remote connection table (#1789)
+* Remove a user from remote conversations upon deleting their account (#1790)
+* Remove elasticsearch specific details from the search endpoint (#1768)
+* Added support for updating self member status of remote conversations (#1753)
+
+
 # [2021-09-14]
 
 ## API changes
@@ -16,7 +109,7 @@
 ## Features
 
 * Disallow changing phone number to a black listed phone number (#1758)
-* Support using a single IDP with a single EntityID (aka issuer ID) to set up two teams. Required to support multiple teams in environments where the IDP software cannot present anything but one EntityID (E.G.: DualShield). (#1755)
+* Support using a single IDP with a single EntityID (aka issuer ID) to set up two teams. Sets up a migration, and makes teamID + EntityID unique, rather than relying on EntityID to be unique. Required to support multiple teams in environments where the IDP software cannot present anything but one EntityID (E.G.: DualShield). (#1755)
 
 ## Documentation
 

build/alpine/Dockerfile.deps

@@ -27,4 +27,5 @@ RUN apk add --no-cache \
             llvm-libunwind \
             ca-certificates \
             dumb-init \
-            libxml2
+            libxml2 \
+            ncurses

changelog.d/mk-changelog.sh

@@ -16,7 +16,7 @@ for d in "$DIR"/*; do
     echo -n "## "
     sed '$ a\' "$d/.title"
     echo ""
-    for f in "$d"/*; do
+    for f in "$d"/*[^~]; do
         pr=$(getPRNumber $f)
         sed -r '
           # create a bullet point on the first line

charts/ldap-scim-bridge/.helmignore

@@ -0,0 +1,21 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj

charts/ldap-scim-bridge/Chart.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+description: ldap-scim-bridge - Sync LDAP via Wire Server SCIM API
+name: ldap-scim-bridge
+version: 0.0.1

charts/ldap-scim-bridge/README.md

@@ -0,0 +1,17 @@
+# ldap-scim-bridge
+
+To do a test deployment on a existing cluster from a machine able to deploy helm charts…
+```bash
+git clone wire-server
+cd wire-server
+# deploy test instance of openldap with preloaded users
+helm upgrade --install -n wire openldap charts/openldap/
+# deploy ldap-scim-bridge with default chart values
+helm upgrade --install -n wire ldap-scim-bridge charts/ldap-scim-bridge -f charts/ldap-scim-bridge/values.yaml
+```
+
+The kubernetes cronjob resource will spawn a new `ldap-scim-bridge-XXXXXX` pod every minute. Logs for the pod can be gathered with `kubectl log`.
+```
+kubectl get pods -n wire
+kubectl logs ldap-scim-bridge-XXXXXX -n wire
+```

charts/ldap-scim-bridge/templates/cronjob.yaml

@@ -0,0 +1,39 @@
+apiVersion: batch/v1beta1
+kind: CronJob
+metadata:
+  name: ldap-scim-bridge
+  labels:
+    wireService: ldap-scim-bridge
+    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
+    release: {{ .Release.Name }}
+    heritage: {{ .Release.Service }}
+spec:
+  concurrencyPolicy: Forbid
+  schedule: {{ .Values.schedule | quote }}
+  jobTemplate:
+    metadata:
+      labels:
+        wireService: ldap-scim-bridge
+        release: {{ .Release.Name }}
+      annotations:
+        # An annotation of the configmap checksum ensures changes to the configmap cause a redeployment upon `helm upgrade`
+        checksum/secret: {{ include (print .Template.BasePath "/secret.yaml") . | sha256sum }}
+    spec:
+      backoffLimit: 0
+      template:
+        spec:
+          restartPolicy: Never
+          volumes:
+            - name: "ldap-scim-bridge-config"
+              secret:
+                secretName: "ldap-scim-bridge"
+          containers:
+            - name: ldap-scim-bridge
+              image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+              imagePullPolicy: {{ default "" .Values.imagePullPolicy | quote }}
+              args: ["ldap-scim-bridge", "/etc/ldap-scim-bridge/config.yaml"]
+              volumeMounts:
+              - name: "ldap-scim-bridge-config"
+                mountPath: "/etc/ldap-scim-bridge/"
+              resources:
+{{ toYaml .Values.resources | indent 16 }}

charts/ldap-scim-bridge/templates/secret.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ldap-scim-bridge
+  labels:
+    wireService: ldap-scim-bridge
+    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+    release: "{{ .Release.Name }}"
+    heritage: "{{ .Release.Service }}"
+type: Opaque
+stringData:
+  config.yaml: |
+{{ toYaml .Values.config | indent 4 }}

charts/ldap-scim-bridge/values.yaml

@@ -0,0 +1,42 @@
+image:
+  repository: quay.io/wire/ldap-scim-bridge
+  tag: 0.2
+resources:
+  requests:
+    memory: "256Mi"
+    cpu: "100m"
+  limits:
+    memory: "512Mi"
+    cpu: "500m"
+# https://v1-19.docs.kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#schedule
+# schedule: "@hourly"
+schedule: "*/1 * * * *"
+# https://github.com/wireapp/ldap-scim-bridge
+config:
+  logLevel: "Debug"  # one of Trace,Debug,Info,Warn,Error,Fatal; `Fatal` is least noisy, `Trace` most.
+  ldapSource:
+    tls: false
+    host: "openldap"
+    port: 389
+    dn: "cn=admin,dc=nodomain"
+    password: "admin"
+    search:
+      base: "ou=People,dc=nodomain"
+      objectClass: "account"
+    codec: "utf8"
+    deleteOnAttribute:  # optional, related to `delete-from-directory`.
+      key: "deleted"
+      value: "true"
+    deleteFromDirectory:  # optional; ok to use together with `delete-on-attribute` if you use both.
+      base: "ou=DeletedPeople,dc=nodomain"
+      objectClass: "account"
+  scimTarget:
+    tls: false
+    host: "spar"
+    port: 8080
+    path: "/scim/v2"
+    token: "Bearer U6DRfAcwsvCg9eBStJWtiHu/XqTB1iVDcvNcsuftvqk="
+  mapping:
+    userName: "uidNumber"
+    externalId: "uid"
+    email: "email"

charts/nginx-ingress-services/templates/ingress_federator.yaml

@@ -13,6 +13,7 @@ metadata:
     nginx.ingress.kubernetes.io/ssl-redirect: "true"
     nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
     nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
+    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "{{ .Values.tls.verify_depth }}"
     nginx.ingress.kubernetes.io/auth-tls-secret: "{{ .Release.Namespace }}/federator-ca-secret"
     nginx.ingress.kubernetes.io/configuration-snippet: |
       grpc_set_header "X-SSL-Certificate" $ssl_client_escaped_cert;

charts/nginx-ingress-services/values.yaml

@@ -29,6 +29,8 @@ tls:
   #            `helm upgrade --install -n cert-manager-ns --set 'installCRDs=true' cert-manager jetstack/cert-manager`
   #
   useCertManager: false
+  # the validation depth between a federator client certificate and tlsClientCA
+  verify_depth: 1
 
 certManager:
   # Indicates whether Letsencrypt's staging API server is used and therefore certificates are NOT trusted

charts/nginz/values.yaml

@@ -97,6 +97,9 @@ nginx_conf:
     - path: /connections
       envs:
       - all
+    - path: ~* ^/list-connections$
+      envs:
+      - all
     - path: /invitations
       envs:
       - all

charts/openldap/.helmignore

@@ -0,0 +1,21 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj

charts/openldap/Chart.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+description: openldap - Test instance for LDAP sync with ldap-scim-bridge
+name: openldap
+version: 0.0.1