Improving TTS Project

[kevmo314]

This is going to be a project to overhaul how TTS works in RealtimeChat. Today, it works by attempting to keep the main activity alive while the active channel is streaming. This comes with some complications, namely:

• It's hard to keep that activity alive
• It's battery intensive since the activity runs other things
• It's hard to ensure message parity since the message history fetches things that should not be read
• TTS logic is intermingled with other logic
• It's not so easy to use native APIs to customize TTS voices as we are reliant on third-party packages, so we run into issues like this: [tts] tts does not work when app is in the background on ios #388

This project will have four components:

1. The iOS TTS engine
2. The Android TTS engine
3. The TTS circular buffer
4. The TTS isolate

The Android and iOS TTS engines will have the same Dart APIs following:

class Engine:
  getLanguages(): string[]
  getVoices(languageId: string): string[]
  speak({ text: string, languageId: string, voiceId: string }): Promise<void>
  stop() // immediately stop speaking

Milestone 1: Native APIs

Beta Give feedback

1. Create a TTS native channel for Android #1080
   +0
   
2. Add a getLanguages() method to the TTS channel that does not return anything #1081
   +0
   
3. Update getLanguages() to return the available TTS languages #1083
   +0
   
4. Create a TTS native channel for iOS #1086
   +0
   
5. Add a speak method to the Android native channel #1087
   +0
   
6. Add a speak method to the iOS native channel #1088
   +0
   
7. Make speak block on the utterance on Android #1092
   +0
   
8. Add a stop method to stop speaking #1093
   +0
   
9. Ensure the Dart API for Android works #1097
   +0
   

Milestone 2: TTS circular buffer

Beta Give feedback

1. Create a new empty Dart class called TTSQueue #1098
   +0
   
2. Add a speak(string id, string text) method to TTSQueue that calls the Dart API to speak if no speak is enqueued, otherwise enqueue it for after the utterance #1099
   +0
   
3. Add a delete(string id) method that deletes an utterance by id if it is in the queue, otherwise does nothing #1100
   +0
   
4. Add a clear() method that clears the queue and stops any utterances #1101
   +0
   
5. Add unit tests for the circular buffer #1102
   +0
   

Milestone 4: Running in an isolate

Beta Give feedback

1. Create an empty isolate #1103
   +0
   
2. Make this isolate run in a background service #1104
   +0
   
3. Have the isolate listen to shared preferences on the key "tts_channel" #1105
   +0
   
4. Load firestore in the isolate #1106
   +0
   
5. When tts_channel is set, read the corresponding collection's messages #1151
   +0
   
6. For each message, add its vocalization to the speak system #1152
   +0
   

Milestone 5: Migrating the new TTS system

Beta Give feedback

1. Add a boolean flag (hardcoded constant) that indicates if we should use the new TTS, make it true in debug mode #1153
   +0
   
2. Update the TTS button to set tts_channel instead if the boolean flag is true #1154
   +0
   

Milestone 6: Testing and cleanup

Beta Give feedback

1. Test against some live channels #1189
   +0
   
2. Ship a new release with the new TTS engine
3. Remove the old TTS engine #1166
   +0
   
4. Remove shared_preferences dependency #1165
   +0
   
5. Test TTS in the background #1183
   +0
   
6. Add "TTS enabled"/"TTS disabled" messages #1184
   +0
   
7. Disable TTS when stream goes offline #1185
   +0
   
8. Add an action button to the TTS notification that disables TTS #1187
   +0
   
9. Update getVocalization() to use this function #1188
   +0
   
10. Disable TTS if the queue gets too long #1186
    +0