Merge pull request #674 from wireapp/release/2019-03-25

CHANGELOG.md

@@ -1,4 +1,36 @@
-# 2019-02-28
+# 2019-03-25 #674
+
+## API changes
+
+  * SCIM delete user endpoint (#660)
+  * Require reauthentication when creating a SCIM token (#639)
+  * Disallow duplicate external ids via SCIM update user (#657)
+
+## Documentation changes
+
+  * Make an index for the docs/ (#662)
+  * Docs: using scim with curl. (#659)
+  * Add spar to the arch diagram. (#650)
+
+## Bug fixes
+
+  * ADFS-workaround for SAML2 authn response signature validation. (#670)
+  * Fix: empty objects `{}` are valid TeamMemberDeleteData. (#652)
+  * Better logo rendering in emails (#649)
+
+## Internal changes
+
+  * Remove some unused instances (#671)
+  * Reusable wai middleware for prometheus (for Galley only for now) (#669)
+  * Bump cql-io dep from merge request to latest release. (#661)
+  * docker image building for all of the docker images our integration tests require. (#622, #668)
+  * Checking for 404 is flaky; depends on deletion succeeding (#667)
+  * Refactor Galley Tests to use Reader Pattern (#666)
+  * Switch Cargohold to YAML-only config (#653)
+  * Filter newlines in log output.  (#642)
+
+
+# 2019-02-28 #648
 
 ## API changes
 

README.md

@@ -136,6 +136,8 @@ Integration tests require all of the haskell services (brig, galley, cannon, gun
     - SNS
     - S3
     - DynamoDB
+- Required additional software:
+    - netcat (in order to allow the services being tested to talk to the dependencies above)
 
 Setting up these real, but in-memory internal and "fake" external dependencies is done easiest using [`docker-compose`](https://docs.docker.com/compose/install/). Run the following in a separate terminal (it will block that terminal, C-c to shut all these docker images down again):
 

deploy/docker-ephemeral/build/README.md

@@ -0,0 +1,54 @@
+A makefile that uses docker.io, and qemu-user-static to build dependencies for our integration tests.
+
+Builds and uploadsdocker images for multiple architectures. Allows for '-j' to build multiple images at once. Uploads assume the hub.docker.com docker registry.
+
+# Setup
+
+## Docker
+
+Follow the instructions in [our dependencies file](doc/Dependencies.md) to ensure you have docker installed, and logged in.
+
+## qemu
+
+### Debian
+
+
+```bash
+apt-get install qemu-user-static
+sudo service binfmt-support start
+```
+
+### Fedora
+
+'sudo dnf install -y qemu-user-static'
+
+# Using
+
+Assuming you have docker, and have followed the above instructions, "make build-all" should work. This builds all of the images, and places them in docker on the local machine.
+to build an individual image (and it's dependent images), run "make-<imagename>". to see a list of images that are buildable, run "make names".
+
+## Using with Dockerhub
+
+If you want to upload images to dockerhub, you must go to dockerhub, and create repositories under your user with the names of the images you want to upload. use `make names` to get the list of buildable images.
+
+If you don't want to change the Makefile, add the DOCKER_USERNAME, DOCKER_EMAIL, and DOCKER_REALNAME environment variables.
+
+For instance, when I want to build all debian images, and upload them to dockerhub I use:
+```bash
+make DIST=DEBIAN DOCKER_USERNAME=julialongtin [email protected] DOCKER_REALNAME='Julia Longtin' push-all
+```
+
+You can also push a single image (and it's dependent images) with "make push-<imagename>".
+
+If you want your builds to go faster, and are okay with having interleaved output from multiple builds, use the '-j' argument to make, to parallize the builds. '-j' can take an integer argument for the number of threads you want it to run at once, or no argument for 'all of the things you can figure out how to do at once'.
+
+By default this makefile builds and uploads the debian based images. Use the 'DIST=ALPINE' environment variable to build the alpine based images instead.
+
+# Troubleshooting:
+## binfmt support:
+
+examine the following file, and ensure the 'flags:' line has an "F" flag on it:
+cat /proc/sys/fs/binfmt_misc/qemu-arm | grep flags
+
+if it doesn't, try re-starting binfmt-support on debian.
+

deploy/docker-ephemeral/docker-compose.yaml

@@ -6,23 +6,26 @@ networks:
 services:
   fake_dynamodb:
     container_name: demo_wire_dynamodb
-    image: cnadiminti/dynamodb-local:2018-04-11
+#    image: cnadiminti/dynamodb-local:2018-04-11
+    image: julialongtin/dynamodb_local:0.0.9
     ports:
       - 127.0.0.1:4567:8000
     networks:
       - demo_wire
 
   fake_sqs:
     container_name: demo_wire_sqs
-    image: airdock/fake-sqs:0.3.1
+#    image: airdock/fake-sqs:0.3.1
+    image: julialongtin/airdock_fakesqs:0.0.9
     ports:
       - 127.0.0.1:4568:4568
     networks:
       - demo_wire
 
   fake_localstack:
     container_name: demo_wire_localstack
-    image: localstack/localstack:0.8.0  # NB: this is younger than 0.8.6!
+#    image: localstack/localstack:0.8.0  # NB: this is younger than 0.8.6!
+    image: julialongtin/localstack:0.0.9
     ports:
       - 127.0.0.1:4569:4579 # ses # needed for local integration tests
       - 127.0.0.1:4575:4575 # sns
@@ -35,16 +38,17 @@ services:
 
   basic_smtp: # needed for demo setup
     container_name: demo_wire_smtp
-    # https://github.com/namshi/docker-smtp
-    image: namshi/smtp
+#    image: namshi/smtp
+    image: julialongtin/smtp:0.0.9
     ports:
         - 127.0.0.1:2500:25
     networks:
       - demo_wire
 
   fake_s3:
     container_name: demo_wire_s3
-    image: minio/minio:RELEASE.2018-05-25T19-49-13Z
+#    image: minio/minio:RELEASE.2018-05-25T19-49-13Z
+    image: julialongtin/minio:0.0.9
     ports:
      - "127.0.0.1:4570:9000"
     environment:
@@ -59,6 +63,7 @@ services:
   #   ports:
   #     - "61613:61613"
 
+  # FIXME: replace redis image with one we build.
   redis:
     container_name: demo_wire_redis
     image: redis:3.0.7-alpine
@@ -69,20 +74,33 @@ services:
 
   elasticsearch:
     container_name: demo_wire_elasticsearch
-    image: elasticsearch:5.6
+    #image: elasticsearch:5.6
+    image: julialongtin/elasticsearch:0.0.9-amd64
     # https://hub.docker.com/_/elastic is deprecated, but 6.2.4 did not work without further changes.
     # image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
     ports:
       - "127.0.0.1:9200:9200"
       - "127.0.0.1:9300:9300"
+    environment:
+      - "bootstrap.system_call_filter=false"
+# ES_JVM_OPTIONS is reserved, so...
+# what's present in the jvm.options file by default.
+#      - "JVM_OPTIONS_ES=-Xmx2g -Xms2g"
+      - "JVM_OPTIONS_ES=-Xmx512m -Xms512m"
+      - "discovery.type=single-node"
     networks:
       - demo_wire
 
   cassandra:
     container_name: demo_wire_cassandra
-    image: cassandra:3.11.2
+    #image: cassandra:3.11.2
+    image: julialongtin/cassandra:0.0.9
     ports:
       - "127.0.0.1:9042:9042"
+    environment:
+# what's present in the jvm.options file by default.
+#      - "CS_JAVA_OPTIONS=-Xmx1024M -Xms1024M -Xmn200M"
+      - "CS_JVM_OPTIONS=-Xmx128M -Xms128M -Xmn50M"
     networks:
       - demo_wire
 
@@ -146,6 +164,7 @@ services:
     networks:
       - demo_wire
 
+  # FIXME: replace aws_cli with an image that we build.
   aws_cli:
     image: mesosphere/aws-cli:1.14.5
     depends_on:

deploy/services-demo/conf/cargohold.demo-docker.yaml

@@ -11,3 +11,6 @@ aws:
 settings:
   maxTotalBytes: 27262976
   downloadLinkTTL: 300    # Seconds
+
+logLevel: Info
+logNetStrings: false

deploy/services-demo/conf/cargohold.demo.yaml

@@ -11,3 +11,6 @@ aws:
 settings:
   maxTotalBytes: 27262976
   downloadLinkTTL: 300    # Seconds
+
+logLevel: Info
+logNetStrings: false

deploy/services-demo/demo.sh

@@ -144,7 +144,7 @@ if [ "$docker_deployment" = "false" ]; then
     run_haskell_service galley ${yellow}
     run_haskell_service gundeck ${blue}
     run_haskell_service cannon ${orange}
-    run_haskell_service cargohold ${purpleish} Info
+    run_haskell_service cargohold ${purpleish}
     run_haskell_service proxy ${redish} Info
     run_haskell_service spar ${orange}
     run_nginz ${blueish}

docs/README.md

@@ -0,0 +1,29 @@
+(incomplete)
+
+# Reference documentation
+
+What you need to know as a user of the Wire backend: concepts, features, and API.
+
+## Users
+
+We support the following functionality related to users and user profiles:
+
+* [Rich info](reference/user/rich-info.md) `{#RefRichInfo}`
+* TODO
+
+## Provisioning
+
+We have support for provisioning users via SCIM ([RFC 7664][], [RFC 7643][]). It's in the beta stage.
+
+[RFC 7664]: https://tools.ietf.org/html/rfc7664
+[RFC 7643]: https://tools.ietf.org/html/rfc7643
+
+* [Using the SCIM API with curl](reference/provisioning/scim-via-curl.md) `{#RefScimViaCurl}`
+* TODO
+
+# Developer documentation
+
+What you need to know as a Wire backend developer. All of these documents can and should be referenced in the code.
+
+* [Development setup](developer/dependencies.md) `{#DevDeps}`
+* TODO

docs/developer/architecture/wire-arch-2.xml

@@ -1 +1 @@
-<?xml version="1.0" encoding="UTF-8"?><mxfile userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" version="8.5.15" editor="www.draw.io"><diagram id="97eeb785-4a3e-bce8-b2c8-d474cc0a017c" name="Page-1">7Vxtc5s4EP41/lgPSIDhY/PS3s20ndy5M9d8upFBwbpi5AG5tu/XnwQSRginuIaQc5zMZNAiCXie3dVqWTKBt6vdxwytl59phJMJsKLdBN5NAAgCyP8Kwb4U2L7vlZI4I5GUHQRz8i+WQktKNyTCudaRUZowstaFIU1THDJNhrKMbvVuTzTRr7pGMTYE8xAlpvQvErFlKfXB7CD/DZN4qa5se0F5ZoHC73FGN6m83gTAp+KnPL1Cai75oPkSRXRbE8H7CbzNKGXl0Wp3ixOBrYKtHPfhyNnqvjOcsi4DJFE/ULLB6o6L+2J7hcV2SRier1Eo2lvO9wTeLNkq4S2bH+Yso98rlITkiSTJLU1oVoxXz8/lNGWSatvh7QjlSxxp09SGWcUPPyPvEGcM744+pV1hx3US0xVm2Z53kfrnSLS3By7tioJljUhHKSCSChRXcx1A5AcSx3ZMnbeBqWd3BNX2egDVNTDEETdZ2aQZW9KYpii5P0hvCjss0LB0fPGOsG+140fRZeqKVsrv65vefJQT5Axl7L3wLlyQ0hQr2Qcibrvog9NI9QgTlOckLIWyi7jgP5ixveQMbRjlosPtf6J0rShvUFsCIJ5aYyynmyxUIuUrURZjiT50OzOb4QQx8kOf/xzSvLNIs7uSNjIxPzOz0ylzxqNsNoSdWW/AzmwwHmn+EHb2Jkgb0dKCs0kL6UpAeNTqSpYUhYIQFW0ajB34qJNm66RJWg+MWScwdtxLnsFln7zJoQ+U8BmBJeMizwr0IaX6yF4NtqvLdlIAFWu+lAbYDRN2QNOIi8YDzgh/Cpz9VGmGN/P/q9KoS6n9b+AOpkT2ELuNBrj13Yck/fmNhnMnfs/caMgBrutoYKqtdn3fYbdsO4IefLRa1C8X3SrYbKpqHV7HNuF11Np5Fr5wkHDTcHuvMzoZ3hG5ls4u9BqrWekfDUd0+kQ9erRB8ievyeZcq2Fy3kuanGvAe7/jqz3XYANm/oisDdkGci1gooTEqbAkDomIJG4EYCREyXt5YkWiqLDmNiprbAGrsHjGrYCKce+CnvJXLjTzV07LMgJ6SF7ZnoH57+kbxBzA2cthPjMwv+VeHaVRhibASwTIi4wfxeLoTxyR3OBCC30r/2CdSA/3Hq4lfrk8QQucPNCcSGgNrj41OiwoY3TVQiYTq40IztfiZle7WLyJmdKnJxLiaYQYWqAc59XR0VXpLE/mB0BfFqAZnEG/xZPBHgj2DYLXGd3tL5nFv8PNoo3Koi2f2O6B12Cm0VrNULfbmUmrC3qgNTBoDXlsQZfiHd6V2vMDfl/fm1Z29ALcguEyHiqv8c6aWqCSNHKX3VMbnfJhryq7Ub0nr6U3VeTc+0aDI4D2tQ5rEffnhiqcFvgDM5WxyEh8Nfo+dhz+eEZv5lBilCT4ulD3QKznjUisWTwSC1cdfr8y20No7TeYfcEYrKWE5Zm90z1fBjmic4yycHnJ1A+zh3JVEnSEPRQwk0E1oq9MnuiMZyMyaaaYHunmq/BVhsHO15SRp33LCRFq3yZ0E5nnPlIac00A1me0vqBMSSieNp+uM/IDMe7ci4cfRDsCXTkc3yxKdFuSvrCHXBgwc2Fzfjd8nFWw/SGjMm1/5fS0zfRsNh6pZv5rfj8vWZ3/IQ/u9ila0bsb056/4N2KmuKvW5IQetWFX9ljuePpgpk0m3+p68KVztPjb69Bp/mWbig61Rw1OtOYpLseeWziu8pDhKcFaRzcHE9jju8W7U3KqzehTdB/XQV++uaq1Ik+jNTX32OrZp3UlhdSbh+kmumtEKWcjsuxzvF2y7Y1Gy8RAgepYtHr3IFegDvREtygaxX8gOW16juD11LIDi++jMS2YLOQpK12C5hK79huDwCbuYMLA9h3RsX3rReNqAG2r9MAHJOGoapIoLlzflvVUsdIeMnyKWjudM0F1/hMocaD5jJE4wExQWIZfFjOxHgV3PXF7HOfxBwvR+2ynAYty2nQmbaO8ey55aW+vuWFQYPtruWlHmxM5DQm6q+8FJpb5Qtbthy1JD2zbEG1uvReX+q02OYgwfAI8a7BQZvN9l7EHUA9yQibnyR1tbJmPAMCfygrczpE362VNG01PY1ynoL97gyZH2mO5S6bhmng39ld6jx6vXlL3jz874my++EffMD7/wA=</diagram></mxfile>
+<mxfile modified="2019-03-05T12:19:04.858Z" host="www.draw.io" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36" version="10.3.2" etag="wqgw6m7A8E-uy8NE5ST1" type="device"><diagram id="97eeb785-4a3e-bce8-b2c8-d474cc0a017c" name="Page-1">7Vxbc5s4FP41fqwHENfHxkm7nWky6XpnNnnqyKBgWow8ICf2/vqVQGCEZAfHYFLHyUOQEAK+79x0dMgITBbrrylczm9xgOKRoQXrEbgeGYZuWRr9w3o2RY+hGbwnTKOAj9p2TKP/EO8sh62iAGXCQIJxTKKl2OnjJEE+EfpgmuIXcdgTjsW7LmGIpI6pD2O5998oIPOi1zWcbf9fKArn5Z112yvOzKD/O0zxKuH3GxngKf8pTi9gORd/0WwOA/xS6wI3IzBJMSbF0WI9QTEDt4StuO7LjrPVc6coIW0uMIsLnmG8QuUT589FNiUWL/OIoOkS+qz9Qgkfgas5WcS0pdPDjKT4d4US63mK4niCY5zm15fvT/txQjjVuknbAczmKBCmqV2m5T/0jPxK/C2fUUrQutbFX/ErwgtE0g0dwuXPtPklL1sydbOU0nmdSZt3Qi5BYTXZFkV6wIFUgwo+BqhaS0xNvQtMvT5AbYBXBznBCXoVT/Oa/R6JJz9rmvyNOL66Z0kAA2Cr8DWPx9eS0EQBNYm8iVMyxyFOYHyz7b3K7VwOjiYijdYReagdP7IhY4u1EvpcD2LzkU+QEZiSz8x6b+HP+75E7LHzMSgJyhF+DLMs8otOPoTd8BciZMNZhSuCadf28b9jvNxB/k4KM7xK/dIxlN4IpiHiwwDHjgG2l+gUxZBEz6KPOYY0+yjS9LakDUzMbiu2T5ryxvVaaG14681U//r5w4ofbGNlZcDV73483DnPn0x7KP6dPpRW+xBKqxtDkeb2obQfhDRzKNK8o0nz8YJBuFPrCpZKChkh5dJAYmzLR500XSSN07plTDuAsd0m9yguu+ONX3qPI3rbKoSyNU+8pBAePqrBdnXbVgJQxrCnkgC9ocI0OFQ5uHuURvQtUPqq0PSv5n+q0JQ3asbdPQhRuSg420WMZZkCmGVepL5I1BVrRK8DG1269PNFl86mFlVhDa7L8HayRNRBL+GmZPbea3TStyGyNJFdYDe8WWEdJUN0+EQdWrReEojvSefKjFalcvYpVc6S4L1ZU29PJViCmb4PUSHbQE4BJoyjMGGaRFFikcQVQyfyYfyZn1hEQZBrs4rKGluGlms8oVqA2XWfvI5yjRaQE7imwo0YHWQadVvC/FvyATE3gHM6zB0J8wm16jAJUjgy7JiBPEvpUciO/kZBlElcCKFvZR+0A+mh1sPS2C/tj+EMxfc4izi0ElffGwNmmBC8UJBJmLdhwfmSPexiHbJ9szF+eop8NA4ggTOYoaw6OsQrtbdkricmmAGQgzPgKiwZ6IBgVyJ4meL15pxZ/OmvZioq8zZ/Y8WG3cG8eo5AazVDXW8dmVbL6IBWeVfGp7EFnrMN1wu1xwf8rrg2rfToBNwa/WU8yrzGJ22sGVVPI3fZPrXRKh/2zrIbVVlDLb1Zxs4dLzQoAnBTG7BkcX8micJhgb8hpzJmaRRelL6LFYc7nNLLOZQQxjG6OOoOiLXtAYmVi1JCZqr93xdmOwit3QazJ4zBFDVce9ZON9QNUkSnCKb+/Jyp72cNZZVJ0AHWUIacDKoRfWHyQGPsDMiknGJ6xKt/mK2SFHa6xCR62ihOsFB7EuNVIJ/7inFIJcHQbuHyjDIlPnvbbLxMo2dIqHHPX74X6fBE4TBdudjRUiR9QQe5MEPOhU3p09DrtJztLynmafsLp4ctph1nOFLl/Nf0ZlqwOv3BD643CVzg6ytZn+/QeoHl7n9eojjCF1l4yxrLGk4W5KTZ9K4uCxc6D4+/7Qad8i5dX3SWc9ToTMIoWXfIYxPfReZDNM5Jo+BmaBxSfF/gRqa82gltgv52EXh156qQiS6U1BX3sctmnVTFhpTVBalyesuHCaXjfLRzuNWyrjnDJUJAL1UsYtG8IRbgjoQEt9G2pL7X8tryowXhQ4bBymvB2ZeR6BpoFpKoarcMWehN3eoAYDl3cGYAu+ag+H70opHyrKmJNFTe+ARVJEBeOX+saqldJJyyfArIK13Z4UqfKdR4EEwGa9xDwkgsgg/NHElbwW03Zvd+LrazHLWdO/UU7tRr6U5bxrPHlpe64pIXeA2225aX2qAxkdmYqLvyUlMhO70Ea8PHY22F5Ugp8ICYBAPNT2baSkHT3xqe25sUtIgOlZUeqpqTRrlJzv5Oht70xeBp1Ln5ubiEf2t1Fnm0+9PmvV/PvuvIVPK9CmnZ7Y4NKfRX1ZD3FZqqUT/uK9Rz+N7/9Q/EG5r+qoUoVrGdm2zbNseix9XLGvVDdd3T9eZUXmOqHeq+u4hMfSdd13TlM2+ltpiyW2PS4r+EtPcUY02zRW8BNOcVyVGWLpb6weodC9mrlAQA3tzOxxqbWqM5W3vhrAcYe42vHLYO5dYkYylFl3+IX7PkbYFH+uS0Z/r59jv98y24z/d8mpt6Gz5q8u2WsYpQKo9R9TTE/I/NSx+0a3SkY3as8XD7gGqxkev0MvpG50PvUdsOx9LtNOg+ZZ2emu69xXvny3lfCu2Ki6Iei7poc/tv+gqnsf1niODmfw==</diagram></mxfile>

docs/developer/dependencies.md

@@ -1,4 +1,4 @@
-# Dependencies
+# Dependencies {#DevDeps}
 
 This page documents how to install necessary dependencies to work with the wire-server code base.
 
@@ -149,6 +149,13 @@ _Note_: While it is possible to use non-docker solutions to set up and configure
 sudo apt install docker.io docker-compose
 ```
 
+After installing docker-io, add your user to the docker group, and restart your shell (usually involving a restart of your graphical environment).
+
+once you've logged in again, if you would like to upload any docker images (optional):
+```bash
+docker login --username=<MY_DOCKER_USERNAME>
+````
+
 ### Generic:
 
 * [Install docker](https://docker.com)