Got CatastrophicError in com.semmle.inmemory.trap.TrapScanner

[dydxh]

While compiling chromium with CodeQL, I meet lots of CatastrophicError.

31250437_0.trap for no link target, 1715: com.semmle.util.exception.CatastrophicError: Bad Token: unexpected EOF, expected: rest of string. Read so far: "/home/dydxh/chrom
com.semmle.inmemory.trap.TrapScanner.failEOF(TrapScanner.java:143)
com.semmle.inmemory.trap.TrapScanner.stringBody(TrapScanner.java:195)
com.semmle.inmemory.trap.TrapScanner.nextToken(TrapScanner.java:111)
com.semmle.inmemory.trap.TRAPReader.scanOneField(TRAPReader.java:853)
com.semmle.inmemory.trap.TRAPReader.scanTuple(TRAPReader.java:590)
com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:512)
com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:423)
com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:191)
com.semmle.inmemory.trap.ImportTasksProcessor.lambda$importTrap$1(ImportTasksProcessor.java:151)
com.semmle.util.concurrent.FutureUtils.lambda$mapAsync_$8(FutureUtils.java:136)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base/java.lang.Thread.run(Unknown Source)
37962707_0.trap for no link target, 1714: com.semmle.util.exception.CatastrophicError: Bad Token: unexpected EOF, expected: rest of string. Read so far: @"loc,{
com.semmle.inmemory.trap.TrapScanner.failEOF(TrapScanner.java:143)
com.semmle.inmemory.trap.TrapScanner.stringBody(TrapScanner.java:195)
com.semmle.inmemory.trap.TrapScanner.string(TrapScanner.java:224)
com.semmle.inmemory.trap.TrapScanner.nextToken(TrapScanner.java:119)
com.semmle.inmemory.trap.TRAPReader.scanOneField(TRAPReader.java:853)
com.semmle.inmemory.trap.TRAPReader.scanTuple(TRAPReader.java:590)
com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:512)
com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:423)
com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:191)
com.semmle.inmemory.trap.ImportTasksProcessor.lambda$importTrap$1(ImportTasksProcessor.java:151)
com.semmle.util.concurrent.FutureUtils.lambda$mapAsync_$8(FutureUtils.java:136)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base/java.lang.Thread.run(Unknown Source)

or something like

15364014_0.trap for no link target, 928: com.semmle.util.exception.CatastrophicError: Bad Token: unexpected EOF, expected: rest of string. Read so far: """../../mojo/public/cpp/bindings/lib/message_quota_checker.h"", line 59: err
or: expression must have a constant value
      static constexpr base::TimeDelta kSamplingInterval = base::Seconds(5);
                                                           ^
""../../base/numerics/safe_math_clang_gcc_impl.h"", line 106: note: cannot call non-constexpr function ""__builtin_mul_overflow"" (declared implicitly)
                 : !__builtin_mul_overflow(x, y,
com.semmle.inmemory.trap.TrapScanner.failEOF(TrapScanner.java:143)
com.semmle.inmemory.trap.TrapScanner.stringBody(TrapScanner.java:195)
com.semmle.inmemory.trap.TrapScanner.nextToken(TrapScanner.java:111)
com.semmle.inmemory.trap.TRAPReader.scanOneField(TRAPReader.java:853)
com.semmle.inmemory.trap.TRAPReader.scanTuple(TRAPReader.java:590)
com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:512)
com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:423)
com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:191)
com.semmle.inmemory.trap.ImportTasksProcessor.lambda$importTrap$1(ImportTasksProcessor.java:151)
com.semmle.util.concurrent.FutureUtils.lambda$mapAsync_$8(FutureUtils.java:136)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base/java.lang.Thread.run(Unknown Source)

I first run autoninja -C out/master-220104-android chrome_public_apk to compile chromium.

Then delete some .o/.a/.so files in obj folder.

Then run codeql database create -l cpp --max-disk-cache=56320 --ram=56320 -J="-Xmx56320m" -m light -c "autoninja -C out/master-220104-android chrome_public_apk" out/ql to create CodeQL database.

The error log contains the error message in chromium but the chromium build process seems successful even with these error logs.

The codeql-cli version is 2.7.3

I also open an issue at here

More log is here

[MathiasVP]

Thanks for raising this issue @dydxh,

Ideally, you shouldn't get any CatastrophicErrors. We're looking into this problem, but it's difficult to tell when this will be fixed.

The log does end with:

Successfully created database at /home/dydxh/chromium/src/out/ql.

So I hope that a database is produced at the end. That database might be incomplete due to the CatastrophicErrors, but you should still be able to run queries on it.

[dydxh]

Yeah, I can query on the generated databases indeed. But some info may lose cause of the CatastrophicError :(

[MathiasVP]

Indeed, that's annoying. I'll forward this to the appropriate team. I think we're already planning to look at Chromium failures not totally unlike this soon.

In the meantime, can you please let us know what operating system you're on, what SHA of Chromium you're trying to build?

[dydxh]

Sorry for the delay. I build these things on Ubuntu 18.04 and the Chromium commit id is fd8a8914ca0183f0add65ae55f04e287543c7d4a.

You may meet some brotli error while building the CodeQL database, but I lose the log error. I solve this problem by patching the trap_compression_kind value in cpp/tools/linux64/extractor from 2 to 0.

Hope this can help you.

[KeuntaeShin]

@dydxh , May i know how did you patch the binary file? I am also suffering from a similar issue mentioned in #7582

$ file extractor
extractor: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=cd807e77ab4fa5ab9259545e5dd4f75389f27a7c, for GNU/Linux 3.2.0, with debug_info, not stripped
babo@babo-400B4C-400B5C-200B4C-200B5C:/home/codeql_home/codeql/cpp/tools/linux64$

$ strings extractor | grep trap_compression
trap_compression_kind
cl_unknown_trap_compression_kind
ec_cl_unknown_trap_compression_kind
optk_trap_compression
trap_compression_kind
$

[dydxh]

@dydxh , May i know how did you patch the binary file? I am also suffering from a similar issue mentioned in #7582

$ file extractor extractor: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=cd807e77ab4fa5ab9259545e5dd4f75389f27a7c, for GNU/Linux 3.2.0, with debug_info, not stripped babo@babo-400B4C-400B5C-200B4C-200B5C:/home/codeql_home/codeql/cpp/tools/linux64$

$ strings extractor | grep trap_compression trap_compression_kind cl_unknown_trap_compression_kind ec_cl_unknown_trap_compression_kind optk_trap_compression trap_compression_kind $

Some tools like IDA Pro or BinaryNinja may allow you to search the symbol and do the patch.
Or you may get the address of trap_compression_kind by running objdump -t extractor | grep 'trap_compression_kind', then use hex editor to patch the value by yourself.

[KeuntaeShin]

@dydxh , May i know how did you patch the binary file? I am also suffering from a similar issue mentioned in #7582
$ file extractor extractor: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=cd807e77ab4fa5ab9259545e5dd4f75389f27a7c, for GNU/Linux 3.2.0, with debug_info, not stripped babo@babo-400B4C-400B5C-200B4C-200B5C:/home/codeql_home/codeql/cpp/tools/linux64$
$ strings extractor | grep trap_compression trap_compression_kind cl_unknown_trap_compression_kind ec_cl_unknown_trap_compression_kind optk_trap_compression trap_compression_kind $

Some tools like IDA Pro or BinaryNinja may allow you to search the symbol and do the patch. Or you may get the address of trap_compression_kind by running objdump -t extractor | grep 'trap_compression_kind', then use hex editor to patch the value by yourself.

I successfully got the crafted version, thanks!

[sashabu]

@dydxh Are you able to reproduce this with the latest Chromium and the latest CodeQL CLI (v2.8.4)? I haven't been able to reproduce this locally (even with v2.7.3), but we have fixed some CatastrophicError issues elsewhere. If you're still impacted, we can of course dig deeper.