Timeout for huge UfedChats

[aberenguel]

I'm processing an UFDR that contais chats with ~50k messages.
That is triggering timeout in UfedChatParser.

Since we don't have bytes in ufed chat item stream, the timeout doesn't consider the size of the chat.

[aberenguel]

In an example, a msgstore.db of 2.052 MB had 1.118.533 messages. It results in 545 messages / MB (disregarding other data in database).
Considering timeoutPerMB = 2, it results 2 seconds to process 545 messages or ~4 seconds per 1000 messages.

[lfcnassif]

Hi @aberenguel, thanks for reporting this and proposing a PR! I would like to suggest an alternative fix: just putting contentHandler.characters(""); in the loop that is taking most time into the parser should reset the timeout counter, if it works, it would require much less code changes.

[aberenguel]

Great! I think handler.characters() is already being done in ParsingTask.parseEmbedded (adding chat html fragments).
Maybe the timeout is happening due to another reasons.

 [WARN]  [engine.task.AbstractTask]  Worker-32 TIMEOUT processing item04.ufdr/_DecodedData/Chat/Chat_39b3e7f2-6593-40dd-a39a-b387d5b3ab4e (null bytes)     iped.engine.io.TimeoutException

This specific chat has 3591 messages.

[aberenguel]

Is it possible to get worker thread dump when TIMEOUT occurs?

[lfcnassif]

Is it possible to get worker thread dump when TIMEOUT occurs?

I think it is possible using JMS.

[aberenguel]

Hi @aberenguel, thanks for reporting this and proposing a PR! I would like to suggest an alternative fix: just putting contentHandler.characters(""); in the loop that is taking most time into the parser should reset the timeout counter, if it works, it would require much less code changes.

@lfcnassif The suggestion of contentHandler.characters worked, with a little modification as bellow. If I pass empty string, the timeout counter is not reset.

char[] nameChars = (message.getName() + "\n").toCharArray();
handler.characters(nameChars, 0, nameChars.length);

It was caused by pre-processing messages in #2286.

[aberenguel]

I think this issue can be closed.

[lfcnassif]

Thanks @aberenguel. So it doesn't happen in last release or master branch, right?

[lfcnassif]

Closing, but I would appreciate the answer for question above, thank you!

[aberenguel]

Thanks @aberenguel. So it doesn't happen in last release or master branch, right?

I haven't seen that in master. Only in the branch I'm working. The contentHandler tip mentioned solved the problem