Merge pull request #1125 from wireapp/release_2020_06_03

CHANGELOG.md

@@ -1,3 +1,34 @@
+# 2020-06-03
+
+## Release Notes
+
+* This release fixes a bug with searching. To get this fix, a new elasticsearch index must be used.
+  The steps for doing this migration can be found in [./docs/reference/elastic-search.md](./docs/reference/elastic-search.md#migrate-to-a-new-index)
+  Alternatively the same index can be recreated instead, this will cause downtime.
+  The steps for the recreation can be found in [./docs/reference/elastic-search.md](./docs/reference/elastic-search.md#recreate-an-index-requires-downtime)
+
+## New Features
+
+* Customer Extensions (not documented, disabled by default, use at your own risk, [details](https://github.com/wireapp/wire-server/blob/3a21a82a1781f0d128f503df6a705b0b5f733d7b/services/brig/src/Brig/Options.hs#L465-L504)) (#1108)
+* Upgrade emails to the latest version: small change in the footer (#1106)
+* Add new "team event queue" and send MemberJoin events on it (#1097, #1115)
+* Change maxTeamSize to Word32 to allow for larger teams (#1105)
+
+## Bug fixes
+
+* Implement better prefix search for name/handle (#1052, #1124)
+* Base64 encode error details in HTML presented by Spar. (#1120)
+* Bump schemaVersion for Brig and Galley (#1118)
+
+## Internal Changes
+
+* Copy swagger-ui bundle to nginz conf for integration tests (#1121)
+* Use wire-api types in public endpoints (galley, brig, gundeck, cargohold) (#1114, #1116, #1117)
+* wire-api: extend generic Arbitrary instances with implementation for 'shrink' (#1111)
+* api-client: depend on wire-api only (#1110)
+* Move and add wire-api JSON roundtrip tests (#1098)
+* Spar tests cleanup (#1100)
+
 # 2020-05-15
 
 ## New Features

docs/reference/elastic-search.md

@@ -67,3 +67,101 @@ docker run "quay.io/wire/brig-index:$WIRE_VERSION" reindex \
 Subcommand `reindex-if-same-or-newer` can be used instead of `reindex`, if you want to recreate the documents in elasticsearch regardless of their version.
 
 (Or, as above, you can also do the same thing without docker.)
+
+## Migrate to a new index
+
+This is needed if we want to migrate to a new index. It could be for updating
+analysis settings or to change any other settings on the index which cannot be
+done without restarting the index. Analysis settings can also be updated by
+recreating the index. Recreating the index is simpler to do, but requires
+downtime, the process is documented [below](#recreate-an-index-requires-downtime)
+
+This can be done in 4 steps:
+
+Before starting, please set these environment variables
+```bash
+ES_HOST=<YOUR_HOST>
+ES_PORT=<YOUR_PORT> # default is 9200
+ES_SRC_INDEX=<INDEX_NAME_ALREADY_IN_USE>
+ES_DEST_INDEX=<NEW_INDEX_NAME>
+WIRE_VERSION=<VERSION_YOU_ARE_DEPLOYING>
+SHARDS=<NUMBER_OF_SHARDS_FOR_THE_INDEX>
+REPLICAS=<NUMBER_OF_REPLICAS_FOR_THE_INDEX>
+REFRESH_INTERVAL=<REFRESH_INTERVAL_FOR_THE_INDEX>
+```
+
+1. Create the new index (please fill out values in `<>` as required)
+   ```bash
+   docker run "quay.io/wire/brig-index:$WIRE_VERSION" create \
+       --elasticsearch-server "http://$ES_HOST:$ES_PORT" \
+       --elasticsearch-index "$ES_DEST_INDEX" \
+       --elastcsearch-shards "$SHARDS" \
+       --elastcsearch-replicas "REPLICAS" \
+       --elastcsearch-refresh-interval "$REFRESH_INTERVAL"
+   ```
+1. Redeploy brig with `elasticsearch.additionalWriteIndex` set to the name of new index. Make sure no old brigs are running.
+1. Reindex data to the new index
+   ```bash
+   docker run "quay.io/wire/brig-index:$WIRE_VERSION" reindex-from-another-index \
+       --elasticsearch-server "http://$ES_HOST:$ES_PORT" \
+       --source-index "$ES_SRC_INDEX" \
+       --destination-index "$ES_DEST_INDEX"
+   ```
+   Optionally, `--timeout <NUMBER_OF_SECONDS>` can be added to increase/decrease from the default timeout of 10 minutes.
+1. Redeploy brig without `elasticsearch.additionalWriteIndex` and with `elasticsearch.index` set to the name of new index
+
+Now you can delete the old index.
+
+## Recreate an index (Requires downtime)
+
+When analysis settings of an index need to be changed, e.g. for changes
+introduced in [#1052](https://github.com/wireapp/wire-server/pull/1052),
+it is not possible to keep the index running while the changes are applied.
+
+To tackle this, a wire-server operator must either migrate to a new index as
+documented [above](#migrate-to-a-new-index) or allow for some downtime. One
+might want to choose downtime for simplicity. These steps are especially simple
+to do when using [wire-server-deploy](https://github.com/wireapp/wire-server-deploy/).
+
+Here are the steps:
+
+Before starting, please set these environment variables
+```bash
+ES_HOST=<YOUR_HOST>
+ES_PORT=<YOUR_PORT> # default is 9200
+ES_INDEX=<INDEX_NAME_ALREADY_IN_USE>
+```
+
+### Step 1: Delete the old index
+```bash
+curl -XDELETE http://$ES_HOST:$ES_PORT/$ES_INDEX
+curl -XDELETE http://$ES_HOST:$ES_PORT/wire_brig_migrations
+```
+
+### Step 2: Recreate the index
+
+#### When using helm charts from [wire-server-deploy](https://github.com/wireapp/wire-server-deploy)
+
+Just redeploy the helm chart, new index will be created and after the deployment
+data migrations will refill the index with users.
+
+
+#### When not using helm charts from [wire-server-deploy](https://github.com/wireapp/wire-server-deploy)
+
+Set these extra environment variables:
+```bash
+WIRE_VERSION=<VERSION_YOU_ARE_DEPLOYING>
+SHARDS=<NUMBER_OF_SHARDS_FOR_THE_INDEX>
+REPLICAS=<NUMBER_OF_REPLICAS_FOR_THE_INDEX>
+REFRESH_INTERVAL=<REFRESH_INTERVAL_FOR_THE_INDEX>
+```
+1. Create the index
+   ```bash
+   docker run "quay.io/wire/brig-index:$WIRE_VERSION" create \
+       --elasticsearch-server "http://$ES_HOST:$ES_PORT" \
+       --elasticsearch-index "$ES_INDEX" \
+       --elastcsearch-shards "$SHARDS" \
+       --elastcsearch-replicas "$REPLICAS" \
+       --elastcsearch-refresh-interval "$REFRESH_INTERVAL"
+   ```
+1. Refill the index as documented [above](#refill-es-documents-from-cassandra)

libs/api-bot/src/Network/Wire/Bot/Assert.hs

@@ -26,7 +26,7 @@ import Imports
 import Network.Wire.Bot.Monad
 import Network.Wire.Client.API.Conversation
 import Network.Wire.Client.API.Push
-import Network.Wire.Client.API.User hiding (UserIds)
+import Network.Wire.Client.API.User
 
 assertConvCreated ::
   (HasCallStack, MonadBotNet m) =>

libs/api-client/api-client.cabal

@@ -4,7 +4,7 @@ cabal-version: 1.12
 --
 -- see: https://github.com/sol/hpack
 --
--- hash: 247f19b4a4ffffe1eea1492b7c28f299df43c7203d768cda26b1f87f64142f80
+-- hash: c8e4c2db3a7d0e48459ccf411e98bd1cb10b885582f6e005eebe301f3f5390ef
 
 name:           api-client
 version:        0.4.2
@@ -43,19 +43,15 @@ library
     , base >=4.6 && <5
     , base64-bytestring >=1.0
     , bilge >=0.21
-    , brig-types >=0.84
     , bytestring >=0.10
     , bytestring-conversion >=0.2
-    , cargohold-types >=0.25.6
     , connection >=0.2
     , cookie >=0.4
     , cryptobox-haskell >=0.1.1
     , cryptonite >=0.17
     , data-default-class >=0.0
     , errors >=2.1
     , exceptions >=0.8
-    , galley-types >=0.66
-    , gundeck-types >=1.24
     , http-client >=0.4
     , http-types >=0.8
     , imports
@@ -74,4 +70,5 @@ library
     , unordered-containers >=0.2
     , uuid >=1.3
     , websockets >=0.9
+    , wire-api >=0.1
   default-language: Haskell2010

libs/api-client/package.yaml

@@ -14,19 +14,15 @@ dependencies:
 - base >=4.6 && <5
 - base64-bytestring >=1.0
 - bilge >=0.21
-- brig-types >=0.84
 - bytestring >=0.10
 - bytestring-conversion >=0.2
-- cargohold-types >=0.25.6
 - connection >=0.2
 - cookie >=0.4
 - cryptobox-haskell >=0.1.1
 - cryptonite >=0.17
 - data-default-class >=0.0
 - errors >=2.1
 - exceptions >=0.8
-- galley-types >=0.66
-- gundeck-types >=1.24
 - http-client >=0.4
 - http-types >=0.8
 - imports
@@ -45,6 +41,7 @@ dependencies:
 - unliftio
 - uuid >=1.3
 - websockets >=0.9
+- wire-api >=0.1
 library:
   source-dirs: src
 stability: experimental

libs/api-client/src/Network/Wire/Client/API/Asset.hs

@@ -35,7 +35,6 @@ module Network.Wire.Client.API.Asset
 where
 
 import Bilge
-import CargoHold.Types
 import qualified Codec.MIME.Type as MIME
 import Data.ByteString.Builder
 import Data.ByteString.Conversion
@@ -45,6 +44,7 @@ import Network.HTTP.Types.Method
 import Network.HTTP.Types.Status hiding (statusCode)
 import Network.Wire.Client.HTTP
 import Network.Wire.Client.Session
+import Wire.API.Asset
 
 type AssetData = LByteString
 

libs/api-client/src/Network/Wire/Client/API/Auth.hs

@@ -29,7 +29,6 @@ module Network.Wire.Client.API.Auth
 where
 
 import Bilge
-import Brig.Types.User.Auth as Auth hiding (Cookie, user)
 import Control.Monad.Catch (MonadMask)
 import qualified Data.ByteString.Char8 as C
 import qualified Data.ByteString.Lazy as Lazy
@@ -44,6 +43,7 @@ import Network.HTTP.Types.Status hiding (statusCode)
 import Network.Wire.Client.HTTP
 import Network.Wire.Client.Monad
 import Web.Cookie (parseSetCookie)
+import Wire.API.User.Auth as Auth (AccessToken, Login, access, tokenType)
 
 newtype AuthCookie = AuthCookie Cookie
 

libs/api-client/src/Network/Wire/Client/API/Client.hs

@@ -29,7 +29,6 @@ module Network.Wire.Client.API.Client
 where
 
 import Bilge
-import Brig.Types.Client as M
 import Data.ByteString.Conversion
 import Data.Id
 import Data.List.NonEmpty
@@ -38,6 +37,8 @@ import Network.HTTP.Types.Method
 import Network.HTTP.Types.Status hiding (statusCode)
 import Network.Wire.Client.HTTP
 import Network.Wire.Client.Session
+import Wire.API.User.Client as M
+import Wire.API.User.Client.Prekey as M
 
 registerClient :: MonadSession m => NewClient -> m M.Client
 registerClient a = sessionRequest req rsc readBody

libs/api-client/src/Network/Wire/Client/API/Conversation.hs

@@ -36,15 +36,17 @@ import Data.Id
 import Data.List.NonEmpty hiding (cons, toList)
 import Data.List1
 import Data.Text (pack)
-import Galley.Types as M hiding (Event, EventType, memberUpdate)
-import Galley.Types.Conversations.Roles (roleNameWireAdmin)
 import Imports
 import Network.HTTP.Types.Method
 import Network.HTTP.Types.Status hiding (statusCode)
-import Network.Wire.Client.API.Push (ConvEvent)
+import Network.Wire.Client.API.Push as M (ConvEvent (..), SimpleMembers (..), UserIdList (..))
 import Network.Wire.Client.HTTP
 import Network.Wire.Client.Monad (ClientException (ParseError))
 import Network.Wire.Client.Session
+import Wire.API.Conversation as M hiding (memberUpdate)
+import Wire.API.Conversation.Role (roleNameWireAdmin)
+import Wire.API.Event.Conversation as M (MemberUpdateData)
+import Wire.API.Message as M
 
 postOtrMessage :: MonadSession m => ConvId -> NewOtrMessage -> m ClientMismatch
 postOtrMessage cnv msg = sessionRequest req rsc readBody
@@ -96,6 +98,7 @@ removeMember cnv mem = do
         $ empty
     rsc = status200 :| [status204]
 
+-- FUTUREWORK: probably should be 'Wire.API.Conversation.Member.MemberUpdate'.
 memberUpdate :: MonadSession m => ConvId -> MemberUpdateData -> m ()
 memberUpdate cnv updt = sessionRequest req rsc (const $ return ())
   where

libs/api-client/src/Network/Wire/Client/API/Push.hs

@@ -28,10 +28,15 @@ module Network.Wire.Client.API.Push
     -- * Event Data
     Event (..),
     ConvEvent (..),
+    Connect (..),
+    ConversationRename (..),
+    MemberUpdate (..),
+    MemberUpdateData (..),
+    NoData,
+    OtrMessage (..),
     SimpleMembers (..),
+    SimpleMember (..),
     UserIdList (..),
-    OtrMessage (..),
-    NoData,
     UserInfo (..),
 
     -- * Event Type
@@ -43,7 +48,6 @@ module Network.Wire.Client.API.Push
 where
 
 import Bilge
-import Brig.Types
 import Control.Concurrent (myThreadId)
 import Control.Concurrent.Async
 import Control.Exception (bracket, finally, onException)
@@ -60,7 +64,6 @@ import Data.Text (pack)
 import qualified Data.Text as T
 import Data.Time.Clock
 import Data.UUID (UUID, fromString)
-import Galley.Types hiding (Event, EventType)
 import Imports hiding (fromString)
 import Network.Connection
 import Network.HTTP.Types.Method
@@ -71,8 +74,11 @@ import Network.Wire.Client.API.Auth
 import Network.Wire.Client.HTTP
 import Network.Wire.Client.Monad
 import Network.Wire.Client.Session
--- FUTUREWORK: We can probably monadunliftio the Session monad somehow?
 import qualified System.Logger as Log
+import Wire.API.Connection (UserConnection (..))
+import Wire.API.Conversation.Member (MemberUpdate (..))
+import Wire.API.Event.Conversation hiding (Event, EventType)
+import Wire.API.User (Name (..), User (..), userEmail)
 
 -------------------------------------------------------------------------------
 
@@ -174,7 +180,8 @@ data Event
   | EMemberLeave (ConvEvent UserIdList)
   | EConnect (ConvEvent Connect)
   | EConvRename (ConvEvent ConversationRename)
-  | EMemberStateUpdate (ConvEvent MemberUpdate)
+  | -- FUTUREWORK: seems like `MemberUpdate` should be 'Wire.API.Event.Conversation.MemberUpdateData' instead
+    EMemberStateUpdate (ConvEvent MemberUpdate)
   | EOtrMessage (ConvEvent OtrMessage)
 
 instance Show Event where

libs/api-client/src/Network/Wire/Client/API/Search.hs

@@ -27,7 +27,6 @@ module Network.Wire.Client.API.Search
 where
 
 import Bilge
-import Brig.Types
 import Control.Monad.Catch (MonadMask)
 import qualified Data.ByteString.Char8 as C
 import Data.Default.Class
@@ -38,6 +37,7 @@ import Network.HTTP.Types.Method
 import Network.HTTP.Types.Status (status200)
 import Network.Wire.Client.HTTP
 import Network.Wire.Client.Session
+import Wire.API.User.Search (Contact (..), SearchResult (..))
 
 data SearchParams = SearchParams
   { searchText :: !Text,

libs/api-client/src/Network/Wire/Client/API/User.hs

@@ -30,7 +30,6 @@ module Network.Wire.Client.API.User
 where
 
 import Bilge
-import Brig.Types as M
 import Control.Monad.Catch (MonadMask)
 import qualified Data.ByteString.Char8 as C
 import Data.ByteString.Conversion
@@ -43,6 +42,11 @@ import Network.HTTP.Types.Status hiding (statusCode)
 import Network.Wire.Client.HTTP
 import Network.Wire.Client.Monad
 import Network.Wire.Client.Session
+import Wire.API.Connection as M
+import Wire.API.User as M
+import Wire.API.User.Activation as M
+import Wire.API.User.Auth as M
+import Wire.API.User.Password as M
 
 -------------------------------------------------------------------------------
 -- Unauthenticated