Skip to content

Commit

Permalink
Merge pull request #1962 from wireapp/release_2021_12_02
Browse files Browse the repository at this point in the history
Release 2021-12-02
  • Loading branch information
julialongtin authored Dec 2, 2021
2 parents 793b764 + 2c93482 commit f3321ea
Show file tree
Hide file tree
Showing 223 changed files with 7,633 additions and 6,869 deletions.
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
- [ ] The **PR description** provides context as to why the change should occur and what the code contributes to that effect. This could also be a link to a JIRA ticket or a Github issue, if there is one.
- [ ] If HTTP endpoint paths have been added or renamed, the **endpoint / config-flag checklist** (see Wire-employee only backend [wiki page](https://github.com/zinfra/backend-wiki/wiki/Checklists)) has been followed.
- [ ] If a cassandra schema migration has been added, I ran **`make git-add-cassandra-schema`** to update the cassandra schema documentation.
- [ ] **changelog.d** contains the following bits of information:
- [ ] **changelog.d** contains the following bits of information ([details](https://github.com/wireapp/wire-server/blob/develop/docs/developer/changelog.md)):
- [ ] A file with the changelog entry in one or more suitable sub-sections. The sub-sections are marked by directories inside `changelog.d`.
- [ ] If new config options introduced: added usage description under docs/reference/config-options.md
- [ ] If new config options introduced: recommended measures to be taken by on-premise instance operators.
Expand Down
47 changes: 45 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,48 @@
<!-- if you're not the release manager, do your edits to changelog under CHANGELOG.d/ -->

# [2021-12-02]

## Release notes

* Breaking change to the `fake-aws-s3` (part of `fake-aws`) helm chart. We now use minio helm chart from https://charts.min.io. The options are documented [here](https://github.com/minio/minio/tree/master/helm/minio) (#1944)

Before running the upgrade, the operators must use `kubectl edit deployment fake-aws-s3` and explicitly set `spec.template.spec.containers[0].serviceAccount` and `spec.template.spec.containers[0].serviceAccountName` to null. (#1944)
* Upgrade team-settings version to 4.3.0-v0.28.28-a2f11cf (#1856)
* Upgrade webapp version to 2021-12-02-production.0-v0.28.29-0-ec2fa00 (#1954)

## Features

* By default install elasticsearch version 6.8.18 when using the elasticsearch-ephemeral chart (#1952)
* Use fluent-bit chart from fluent.github.io instead of deprecated charts.helm.sh. Previous fluent-bit values are not compatible with the new chart, the documentation for the new chart can be found [here](https://github.com/fluent/helm-charts/tree/main/charts/fluent-bit) (#1952)
* Use kibana chart from helm.elastic.co instead of deprecated charts.helm.sh. Previous kibana values are not compatible with the new chart, the documentation for the new chart can be found [here](https://github.com/elastic/helm-charts/tree/main/kibana). This also upgrades kibana to version 6.8.18. (#1952)
* Use kube-prometheus-stack instead of prometheus-operator and update grafana dashboards for compatibility and add federation endpoints to relevant queries. (#1915)
* Add log format called 'StructuredJSON' for easier log aggregation (#1951)

## Bug fixes and other updates

* elasticsearch-ephemeral: Disable automatic creation of indices (#1949)

## Documentation

* Document the wire-server PR process better. (#1934)
* Remove documentation of unsupported scim end-point use case. (#1941)
* Document servant setup and combinators (#1933)

## Internal changes

* Add in-memory interpreters for most Spar effects (#1920)
* Use minio helm chart in fake-aws-s3 from charts.min.io instead of helm.min.io, the latter seems to be down (#1944)
* Upgrade to polysemy-1.7.0.0
(#1932)
* Replace Galley monad with polysemy's Sem throughout Galley (#1917)
* Separate VerdictFormatStore effect from AReqIdStore effect (#1925)

## Federation changes

* The server-to-server API now uses HTTP2 directly instead of gRPC (#1930)
* Errors when leaving a conversation are now correctly handled instead of resulting in a generic federation error. (#1928)


# [2021-11-15]

## Release notes
Expand Down Expand Up @@ -32,7 +75,7 @@
* Turn placeholder access effects into actual Polysemy effects. (#1904)
* Fix a bug in the IdP.Mem interpreter, and added law tests for IdP (#1863)
* Introduce fine-grained error types and polysemy error effects in Galley. (#1907)
* Add polysemy store effects and split off Cassandra specific functionality from the Galley.Data module hierarchy (#1890, #1906). (#1890)
* Add polysemy store effects and split off Cassandra specific functionality from the Galley.Data module hierarchy (#1890, #1906)
* Make golden-tests in wire-api package a separate test suite (for faster feedback loop during development). (#1926)
* Separate IdPRawMetadataStore effect from IdP effect (#1924)
* Test sending message to multiple remote domains (#1899)
Expand All @@ -50,7 +93,7 @@
## Release notes

* Upgrade SFT to 2.1.15 (#1849)
* Upgrade team settings to Release: [v4.2.0](https://github.com/wireapp/wire-team-settings/releases/tag/v4.2.0) and image tag: 4.2.0-v0.28.28-1e2ef7 (#1856)
* Upgrade team settings to Release: [v4.3.0](https://github.com/wireapp/wire-team-settings/releases/tag/v4.3.0) and image tag: 4.3.0-v0.28.28-a2f11cf (#1950)
* Upgrade Webapp to image tag: 20021-10-28-federation-m1 (#1856)

## API changes
Expand Down
10 changes: 7 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ DOCKER_TAG ?= $(USER)
# default helm chart version must be 0.0.42 for local development (because 42 is the answer to the universe and everything)
HELM_SEMVER ?= 0.0.42
# The list of helm charts needed for integration tests on kubernetes
CHARTS_INTEGRATION := wire-server databases-ephemeral fake-aws nginx-ingress-controller nginx-ingress-services
CHARTS_INTEGRATION := wire-server databases-ephemeral fake-aws nginx-ingress-controller nginx-ingress-services wire-server-metrics fluent-bit kibana
# The list of helm charts to publish on S3
# FUTUREWORK: after we "inline local subcharts",
# (e.g. move charts/brig to charts/wire-server/brig)
# this list could be generated from the folder names under ./charts/ like so:
# CHARTS_RELEASE := $(shell find charts/ -maxdepth 1 -type d | xargs -n 1 basename | grep -v charts)
CHARTS_RELEASE := wire-server redis-ephemeral databases-ephemeral fake-aws fake-aws-s3 fake-aws-sqs aws-ingress fluent-bit kibana backoffice calling-test demo-smtp elasticsearch-curator elasticsearch-external elasticsearch-ephemeral fluent-bit minio-external cassandra-external nginx-ingress-controller nginx-ingress-services reaper wire-server-metrics sftd
CHARTS_RELEASE := wire-server redis-ephemeral databases-ephemeral fake-aws fake-aws-s3 fake-aws-sqs aws-ingress fluent-bit kibana backoffice calling-test demo-smtp elasticsearch-curator elasticsearch-external elasticsearch-ephemeral minio-external cassandra-external nginx-ingress-controller nginx-ingress-services reaper wire-server-metrics sftd
BUILDAH_PUSH ?= 0
KIND_CLUSTER_NAME := wire-server
BUILDAH_KIND_LOAD ?= 1
Expand Down Expand Up @@ -61,15 +61,19 @@ endif
c:
cabal build $(WIRE_CABAL_BUILD_OPTIONS) $(package)
ifeq ($(test), 1)
./hack/bin/cabal-run-tests.sh $(package)
./hack/bin/cabal-run-tests.sh $(package) $(testargs)
endif
./hack/bin/cabal-install-artefacts.sh $(package)

# ci here doesn't refer to continuous integration, but to cabal-integration
# Usage: make ci package=brig test=1
.PHONY: ci
ci: c
ifeq ("$(pattern)", "")
make -C services/$(package) i
else
make -C services/$(package) i-$(pattern)
endif

# Build everything (Haskell services and nginz)
.PHONY: services
Expand Down
13 changes: 1 addition & 12 deletions cabal.project
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,6 @@ source-repository-package
location: https://github.com/kim/snappy-framing
tag: d99f702c0086729efd6848dea8a01e5266c3a61c

source-repository-package
type: git
location: https://github.com/lucasdicioccio/http2-client
tag: 73f5975e18eda9d071aa5548fcea6b5a51e61769

source-repository-package
type: git
location: https://github.com/vincenthz/hs-certificate
Expand Down Expand Up @@ -140,13 +135,7 @@ source-repository-package
source-repository-package
type: git
location: https://github.com/wireapp/http2
tag: 7c465be1201e0945b106f7cc6176ac1b1193be13

source-repository-package
type: git
location: https://github.com/wireapp/http2-grpc-haskell
tag: eea98418672626eafbace3181ca34bf44bee91c0
subdir: http2-client-grpc
tag: 1ee1ce432d923839dab6782410e91dc17df2a880

source-repository-package
type: git
Expand Down
24 changes: 6 additions & 18 deletions cabal.project.freeze
Original file line number Diff line number Diff line change
Expand Up @@ -487,7 +487,6 @@ constraints: any.AC-Angle ==1.0,
any.commutative ==0.0.2,
any.comonad ==5.0.6,
any.compactmap ==0.1.4.2.1,
any.compendium-client ==0.2.1.1,
any.compensated ==0.8.1,
any.compiler-warnings ==0.1.0,
any.composable-associations ==0.1.0.0,
Expand Down Expand Up @@ -1138,8 +1137,6 @@ constraints: any.AC-Angle ==1.0,
any.http-reverse-proxy ==0.6.0,
any.http-streams ==0.8.7.2,
any.http-types ==0.12.3,
any.http2-grpc-proto3-wire ==0.1.0.0,
any.http2-grpc-types ==0.5.0.0,
any.httpd-shed ==0.4.1.1,
any.human-readable-duration ==0.2.1.4,
any.hunit-dejafu ==2.0.0.4,
Expand Down Expand Up @@ -1489,14 +1486,6 @@ constraints: any.AC-Angle ==1.0,
any.mpi-hs-cereal ==0.1.0.0,
any.mtl-compat ==0.2.2,
any.mtl-prelude ==2.0.3.1,
any.mu-avro ==0.4.0.4,
any.mu-grpc-client ==0.4.0.1,
any.mu-grpc-common ==0.4.0.0,
any.mu-grpc-server ==0.4.0.0,
any.mu-optics ==0.3.0.1,
any.mu-protobuf ==0.4.2.0,
any.mu-rpc ==0.4.0.1,
any.mu-schema ==0.3.1.2,
any.multi-containers ==0.1.1,
any.multiarg ==0.30.0.10,
any.multimap ==1.2.1,
Expand Down Expand Up @@ -1619,7 +1608,6 @@ constraints: any.AC-Angle ==1.0,
any.papillon ==0.1.1.1,
any.parallel ==3.2.2.0,
any.parallel-io ==0.3.3,
any.parameterized ==0.5.0.0,
any.paripari ==0.6.0.1,
any.parseargs ==0.2.0.9,
any.parsec-class ==1.0.0.0,
Expand Down Expand Up @@ -1701,9 +1689,10 @@ constraints: any.AC-Angle ==1.0,
any.poly-arity ==0.1.0,
any.polynomials-bernstein ==1.1.2,
any.polyparse ==1.13,
any.polysemy ==1.6.0.0,
any.polysemy-mocks ==0.1.0.0,
any.polysemy-plugin ==0.2.5.2,
any.polysemy ==1.7.0.0,
any.polysemy-check ==0.8.1.0,
any.polysemy-mocks ==0.2.0.0,
any.polysemy-plugin ==0.4.2.0,
any.pooled-io ==0.0.2.2,
any.port-utils ==0.2.1.0,
any.posix-paths ==0.2.1.6,
Expand Down Expand Up @@ -1761,7 +1750,7 @@ constraints: any.AC-Angle ==1.0,
any.proto-lens-protoc ==0.7.0.0,
any.proto-lens-runtime ==0.7.0.0,
any.proto-lens-setup ==0.4.0.4,
any.proto3-wire ==1.2.0,
any.proto3-wire ==1.1.0,
any.protobuf ==0.2.1.3,
any.protobuf-simple ==0.1.1.0,
any.protocol-radius ==0.0.1.1,
Expand Down Expand Up @@ -2415,8 +2404,7 @@ constraints: any.AC-Angle ==1.0,
any.wai-session ==0.3.3,
any.wai-slack-middleware ==0.2.0,
any.wai-websockets ==3.0.1.2,
any.warp ==3.3.13,
any.warp-grpc ==0.4.0.1,
any.warp ==3.3.17,
any.warp-tls ==3.2.12,
any.warp-tls-uid ==0.2.0.6,
any.wave ==0.2.0,
Expand Down
2 changes: 2 additions & 0 deletions charts/elasticsearch-ephemeral/templates/es.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ spec:
value: "false"
- name: "discovery.type"
value: "single-node"
- name: "action.auto_create_index"
value: ".watches,.triggered_watches,.watcher-history-*,pod-*,node-*"
ports:
- containerPort: 9200
name: http
Expand Down
2 changes: 1 addition & 1 deletion charts/elasticsearch-ephemeral/values.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
image:
repository: elasticsearch
tag: 6.7.1
tag: 6.8.18

service:
httpPort: 9200
Expand Down
4 changes: 2 additions & 2 deletions charts/fake-aws-s3/requirements.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
dependencies:
- name: minio
version: 8.0.3
repository: https://helm.min.io/
version: 3.2.0
repository: https://charts.min.io/
27 changes: 23 additions & 4 deletions charts/fake-aws-s3/templates/reaper.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,36 @@ spec:
labels:
app: {{ template "fullname" . }}-reaper
spec:
volumes:
- name: minio-configuration
projected:
# These are created by the minio chart and used for create buckets and
# users after deployment.
sources:
- configMap:
name: {{ .Values.minio.fullnameOverride }}
- secret:
name: {{ .Values.minio.fullnameOverride }}
containers:
- name: initiate-fake-aws-s3
image: mesosphere/aws-cli:1.14.5
image: "{{ .Values.minio.mcImage.repository }}:{{ .Values.minio.mcImage.tag }}"
imagePullPolicy: {{ .Values.minio.mcImage.pullPolicy }}
command: [/bin/sh]
args:
- -c
- |
echo 'Creating AWS resources'
echo 'Creating MinIO Users and Buckets'
while true
do
AWS_SECRET_ACCESS_KEY={{ .Values.minio.secretKey }} AWS_ACCESS_KEY_ID={{ .Values.minio.accessKey }} aws s3 --endpoint http://{{ .Values.minio.fullnameOverride }}:9000 mb s3://{{ .Values.minio.defaultBucket.name }} | grep -ev "BucketAlreadyOwnedByYou"
/bin/sh /config/initialize
/bin/sh /config/add-user
sleep 10
done
env:
- name: MINIO_ENDPOINT
value: {{ .Values.minio.fullnameOverride | quote }}
- name: MINIO_PORT
value: {{ .Values.minio.service.port | quote }}
volumeMounts:
- name: minio-configuration
mountPath: /config
18 changes: 14 additions & 4 deletions charts/fake-aws-s3/values.yaml
Original file line number Diff line number Diff line change
@@ -1,14 +1,24 @@
# See defaults in https://github.com/helm/charts/tree/master/stable/minio
minio:
mcImage:
repository: quay.io/minio/mc
tag: RELEASE.2021-10-07T04-19-58Z
pullPolicy: IfNotPresent
fullnameOverride: fake-aws-s3
accessKey: dummykey
secretKey: dummysecret
service:
port: "9000"
mode: standalone
users:
- accessKey: dummykey
secretKey: dummysecret
policy: consoleAdmin
persistence:
enabled: false
environment:
MINIO_BROWSER: "off"
defaultBucket:
name: dummy-bucket
resources:
requests:
memory: 200Mi
buckets:
- name: dummy-bucket
purge: true
Expand Down
2 changes: 1 addition & 1 deletion charts/fake-aws/templates/NOTES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ SQS : http://fake-aws-sqs:{{ index .Values "fake-aws-sqs" "service" "httpPo
{{- end }}
{{- if index .Values "fake-aws-s3" "enabled" }}
S3 : http://fake-aws-s3:9000
bucket: {{ index .Values "fake-aws-s3" "minio" "defaultBucket" "name" }}
bucket: {{ index .Values "fake-aws-s3" "minio" "buckets" 0 "name" }}
{{- end }}
{{- if index .Values "fake-aws-ses" "enabled" }}
SES : http://fake-aws-ses:{{ index .Values "fake-aws-ses" "service" "externalPort" }}
Expand Down
2 changes: 1 addition & 1 deletion charts/federator/templates/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ data:
{{- with .optSettings }}
optSettings:
# Filepath to one or more PEM-encoded server certificates to use as a trust
# store when making grpc requests to remote backends
# store when making requests to remote backends
{{- if $.Values.remoteCAContents }}
remoteCAStore: "/etc/wire/federator/ca/ca.crt"
{{- end }}
Expand Down
4 changes: 2 additions & 2 deletions charts/fluent-bit/requirements.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
dependencies:
- name: fluent-bit
version: 2.7.0
repository: https://charts.helm.sh/stable
version: 0.19.6
repository: https://fluent.github.io/helm-charts
48 changes: 38 additions & 10 deletions charts/fluent-bit/values.yaml
Original file line number Diff line number Diff line change
@@ -1,11 +1,39 @@
# See defaults in https://github.com/helm/charts/tree/master/stable/fluent-bit
# See defaults in https://github.com/fluent/helm-charts/tree/main/charts/fluent-bit
fluent-bit:
backend:
type: es
es:
host: elasticsearch-ephemeral
parsers:
enabled: true
regex:
- name: nginz
regex: '^(?<remote_addr>[^ ]*) (?<remote_user>[^ ]*) "(?<thetime>[0-9\/a-zA-Z:]* [+][0-9]*)" "(?<path>.*)" (?<status>[0-9]*) (?<body_bytes_sent>[0-9]*) "(?<http_referer>[^ ])" "(?<http_user_agent>.*)" (?<http_x_forwarded_for>[^ ]*) (?<connection>[^ ]*) (?<request_time>[^ ]*) (?<upstream_response_time>[^ ]*) (?<upstream_cache_status>[^ ]*) (?<zauth_user>[^ ]*) (?<zauth_connection>[^ ]*) (?<request>[a-z0-9]*)'
config:
outputs: |
[OUTPUT]
Name es
Match kube.*
Host elasticsearch-ephemeral
Generate_ID On
Logstash_Format On
Logstash_Prefix pod
Retry_Limit False
Trace_Error On
Replace_Dots On
[OUTPUT]
Name es
Match host.*
Host elasticsearch-ephemeral
Generate_ID On
Logstash_Format On
Logstash_Prefix node
Retry_Limit False
Trace_Error On
Replace_Dots On
## https://docs.fluentbit.io/manual/pipeline/parsers
customParsers: |
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
[PARSER]
Name nginz
Format regex
Regex ^(?<remote_addr>[^ ]*) (?<remote_user>[^ ]*) "(?<thetime>[0-9\/a-zA-Z:]* [+][0-9]*)" "(?<method>[^ ]*) (?<path>.*) (?<http_version>.*)" (?<status>[0-9]*) (?<body_bytes_sent>[0-9]*) "(?<http_referer>[^ ])" "(?<http_user_agent>.*)" (?<http_x_forwarded_for>[^ ]*) (?<connection>[^ ]*) (?<request_time>[^ ]*) (?<upstream_response_time>[^ ]*) (?<upstream_cache_status>[^ ]*) (?<zauth_user>[^ ]*) (?<zauth_connection>[^ ]*) (?<request>[a-z0-9]*)
Time_Key thetime
Time_Format %d/%b/%Y:%H:%M:%S %z
4 changes: 2 additions & 2 deletions charts/kibana/requirements.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
dependencies:
- name: kibana
version: 2.2.0
repository: https://charts.helm.sh/stable
version: 6.8.18
repository: https://helm.elastic.co
Loading

0 comments on commit f3321ea

Please sign in to comment.