ArrayIndexOutOfBoundsException: Index -65536 out of bounds for length 71428 at org.apache.lucene.index.TermsHashPerField.writeByte

[luizedum]

The following error was presented by IPED 4.1.6 during the processing of evidence in E01 format (parts of the file names was replaced as they contained personal information):

2024-09-11 11:44:34 [INFO] [task.index.IndexTask$DocumentsIterable$1] Worker-15 Splitting text of /ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/[email protected]/Apple iCloud (Backup)/[email protected]>>report.xml>>report.xml_0
2024-09-11 11:44:34 [INFO] [engine.core.Worker] Worker-15 interrupted on /ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/[email protected]/Apple iCloud (Backup)/[email protected]>>report.xml>>report.xml_0 (1950114112 bytes)
2024-09-11 11:44:35 [ERROR] [app.processing.Main] Processing Error:
java.lang.Exception: Worker-15 Error while processing /ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/[email protected]/Apple iCloud (Backup)/[email protected]>>report.xml>>report.xml_0 (1950114112bytes)
at iped.engine.core.Worker.process(Worker.java:186) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Worker.run(Worker.java:265) ~[iped-engine-4.1.6.jar:?]
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -65536 out of bounds for length 71428
at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:219) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:241) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:86) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:127) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.TermsHashPerField.initStreamSlices(TermsHashPerField.java:175) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:198) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1224) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:729) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:241) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1532) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1503) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13]
at iped.engine.task.index.IndexTask.process(IndexTask.java:149) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processMonitorTimeout(AbstractTask.java:277) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:192) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.MinIOTask.sendToNextTask(MinIOTask.java:416) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.PythonTask.sendToNextTask(PythonTask.java:278) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.PythonTask.sendToNextTask(PythonTask.java:278) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.PythonTask.sendToNextTask(PythonTask.java:278) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.EmbeddedDiskProcessTask.sendToNextTask(EmbeddedDiskProcessTask.java:116) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?]
at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?]
at iped.engine.core.Worker.process(Worker.java:177) ~[iped-engine-4.1.6.jar:?]
... 1 more

[lfcnassif]

Thanks for reporting. Are you able to share the file:
/ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/[email protected]/Apple iCloud (Backup)/[email protected]>>report.xml

which seems to have caused the exception, so we can try to reproduce the error and validate a possible fix?

[luizedum]

Do you prefer that I share the UFDR file or the 'report.xml' file inside it?
What is the best way to share it while ensuring data confidentiality?

Is it possible to process an evidence in IPED ignoring a specific file so that the processing of the other files can be completed?

[lfcnassif]

Do you prefer that I share the UFDR file or the 'report.xml' file inside it?

At first, I think just the report.xml file could be enough.

What is the best way to share it while ensuring data confidentiality?

You can send it directly to my email in my profile.

Is it possible to process an evidence in IPED ignoring a specific file so that the processing of the other files can be completed?

Yes, the procedure below can be used to ignore specific files:
https://github.com/sepinf-inc/IPED/wiki/Troubleshooting#processing-frozen

[luizedum]

Thank you, the file was shared via email.