Reported coverage results do not match corpus #12986
Comments
|
What fuzzer and job is thsi? |
|
libass_fuzzer and e.g. this recent coverage report: https://storage.googleapis.com/oss-fuzz-coverage/libass/reports/20250129/linux/src/libass/libass/ass_parse.c.html#L373 |
|
And for comparison; this is a locally generated coverage report using the public libass corpus: coverage_local_nolimit.html.gz. Here all tags in |
|
I'm looking into this same problem for some research we are doing. Still investigating things, but I think the reason that coverage is so low is that the coverage measurement is aborted due to a timeout, see report below [1]. As I understand the code on the oss-fuzz side, a timeout will only cause a larger report but not actually an "error". So for the report @TheOneric linked, the execution count of this line corresponds to the number of inputs added to the coverage report. So only 40 inputs are considered. Similarly for this fuzzer harness of sudoers, the max rss memory usage is exceeded causing only a fraction of the corpus to be executed. Though, I'm not sure why this issue does not show up for you locally @TheOneric. Does this explanation still sound plausible? [1] |
|
Yes, only a small fraction of the corpus actually contributes to the reported coverage perfectly explains the result, thanks! Let’s hope this will then resolve itself once the input size limit is actually enabled in OSS Fuzz configuration. |
|
Another easy performance improvement should be to silence the fontconfig warning as printing to stdout can be quite expensive. If the fontconfig file needs to be read for every execution, would it be possible to have that in memory instead of accessing the filesystem? Maybe it would also be a suitable fuzzer input? |
|
Yes, I’d like to silence the warning by making the config load succeed since otherwise we’re lacking fonts to render later. Preferably I’d like to get fontconfig working, since this will allow us to include fontprovider logic in fuzzing, which ime is one of the most likely areas to gain new bugs which stay undetected for a bit due to depending on particular fontname and system details. |
|
Sounds like a good plan. As it sounds like the fontconfig setup has little influence on the behavior, other than that it needs to exist, maybe do the setup in the Dockerfile? Or if you want to set it per fuzz target, do that in a LLVMFuzzerInitialize function, to avoid wasting time and having stateful LLVMFuzzerTestOneInput function. |
Ideally I’d like to, yes. However, it needs to exist in the runtime container and to be able to use the default config path the file needs to be placed on a known absolute path. (Ideally |
|
Hey, you are completely right, only the files in $OUT are available in the runtime container. Indeed the files under #elif ASS_FUZZMODE == FUZZMODE_LIBFUZZER
#include <libgen.h>
int LLVMFuzzerInitialize(int *argc, char ***argv) {
char *exe_path = strdup((*argv)[0]);
char *dir = dirname(exe_path);
char fontpath[100];
sprintf(fontpath, "%s/fonts.conf", dir);
setenv("FONTCONFIG_FILE", fontpath, true);
return 0;
}
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
char *fontpath = getenv("FONTCONFIG_FILE");
printf("%s\n", fontpath);
fflush(stdout);
// OSS Fuzz docs recommend just returning 0 on too large input
// libFuzzer docs tell us to use -1 to prevent addition to corpus
// BUT HongFuzz' LLVMFuzzerTestOneInput hook can't handle it and
// neither does OSS Fuzz convert this in their wrapper, see:
// https://github.com/google/oss-fuzz/issues/11983
if (!LEN_IN_RANGE(size))
return 0;
ASS_Track *track = NULL;
// All return values but zero and -1 are reserved
if (!init())
return 0;
track = ass_read_memory(ass_library, (char *)data, size, NULL);
if (track) {
consume_track(ass_renderer, track);
ass_free_track(track);
}
ass_renderer_done(ass_renderer);
ass_library_done(ass_library);
ass_renderer = NULL;
ass_library = NULL;
return 0;
}
#else |
#12687 brought up abysmal tag coverage and initially set out to fix it by adding sample files with the missing tags. However this approach was abandoned in favour of improving fuzzer performance, since all tags are already part of the fuzzing dictionary.
We since implemented a sample size limit in libass, though it’s not enabled yet in OSS-Fuzz’ build setup.
However, while investigating a good sample size cutoff point, it turned out all tags are already part of the corpus and measuring coverage locally yields much better figures. Yet results reported in the web claim some tags aren’t tested at all, which seems like an OSS-Fuzz bug.
We also noticed the fuzzer produces a bunch of Fontconfig warnings when run in OSS Fuzz environment, due to missing any configuration file which must be located at an absolute path. While the absence of any fonts likely affects coverage of later rendering stages, this has no impact on earlier parsing and thus cannot explain this.
As a tangent: any recommendations how to best include such a configuration file into the runtime environment? I didn’t find anything seeming helpful in the docs.
Btw, the previous, unreproducible memleak also still remains a complete mystery to us: #8598(appears to be fixed now)The text was updated successfully, but these errors were encountered: