AllTalk and Memoir addon and Summarizing Memories

[Tum1370]

Hi,
I am having a issue whwn trying to use Memoir Extension that creates and stores memories from the chat in Oobabooga chat ui.
https://github.com/brucepro/Memoir?tab=readme-ov-file
It uses the LLM model to summarize the set amount of Short Term Memories in to a Long Term Memory with questions / answers etc

I have manage to adjust Memoir to ignore the audio files that AllTalk creates and put in the prompt when savng Short Term and Long Term Memories, but my problem now is trying to adjust AllTalk so it ignores the prompt when Memoir uses the LLM model to summarise the Short Term Memories in to Long Term Memories.

As it stands now, everytime Memoir summarizes the memories, i see it get passed to the AllTalk server before the summary gets passed. I see the summaries appearing the the AllTalk Console window. Am not sure whether it slows things down by trying to create an audio of these summaries which i never hear as it does not get past to the models repsonse.

I have been trying to adjust the script myself so that AllTalk does not see the summarizing of the memories but am not finding a solution at present.

Would you have any suggestions on how i could achieve this and stop AllTalk from seeing the Memoir addon summarize the memories ?

[erew123]

@Tum1370 The way Oobabooga sends data to/from extensions is it looks for a sub-folder called extensions and a file called "script.py" in whatever the extensions folder is. From there when it loads the script.py TGWUI looks for a function in the script called output_modifier https://github.com/erew123/alltalk_tts/blob/alltalkbeta/system/TGWUI_Extension/script.py#L708-L828 and then whatever is generated at each inference of an LLM is sent into the output_modifer of that extension.

If you have multiple extensions, the output it passed through each of the extensions in the order they are loaded by TGWUI (which you can modify in the settings.yaml file).

So for example, if your LLM is just outputting the text responses this is my text response to the question you asked me, then that would be sent to the 1st extension loaded in and it will process it in whatever way needed, before handing it onto the next extension. So if AllTalk is loaded 1st, it would only see this is my text response to the question you asked me.... If however you have another extension loaded before that adds something e.g. this is my text response to the question you asked me. And here is something else that the other extension added then this is what AllTalk will see for TTS generation.

So it may be something you can change by simply changing the load order. It depends how the memories add on works.

If the memories add on text is always in the reply of the LLM and you cannot change it with a load order change, then assumedly there would be something within the text that delimits/identifies what is the LLM response and what is the memory and that would/could be added as code to extract/ignore that text/memory, similar to how AllTalk ignores images when handed over

https://github.com/erew123/alltalk_tts/blob/alltalkbeta/system/TGWUI_Extension/script.py#L732

https://github.com/erew123/alltalk_tts/blob/alltalkbeta/system/TGWUI_Extension/script.py#L830-L844

Hope that gives a guide as to what may help.

Please see this re my general availability at the moment #377

Thanks