Support report directory functionality for Grimoire

[renatahodovan]

Report funtionality is need to make Grimoire compatible with FuzzBench.

[renatahodovan]

This change should work together with google/fuzzbench#2048

[renatahodovan]

I saw that the report functionality existed before, but it was removed in a35b347 (but due to the lack of commit message, I don't understand the reason). However, without this change the FuzzBench setup fails either because of the missing -r option or - if it's omitted - due to saving internal test representation to the output corpus used for coverage measurement.

[tokatoka]

I deleted it because iirc we don't need the dumpstage.

However, without this change the FuzzBench setup fails either because of the missing -r option or - if it's omitted - due to saving internal test representation to the output corpus used for coverage measurement.

I saw your PR on fuzzbench.
Yes, but this is the problem of this file: https://github.com/google/fuzzbench/blob/master/fuzzers/grimoire/fuzzer.py
here we don't need -r option from the beginning. all the inputs should be already written to the corpus.

    let fuzzbench = libafl::stages::DumpToDiskStage::new(
        |input: &BytesInput, state: &StdState<_, _, _, _>| {
            let target_bytes = input.target_bytes();
            let bytes = target_bytes.as_slice().to_vec();
            bytes
        },
        &report_dir.join("queue"),
        &report_dir.join("crashes"),
    )
    .unwrap();

Also this code is actually doing nothing, because it is transforming BytesInput (in the corpus) to BytesInput (in the disk)
I think we only need this if the Input is not BytesInput, and want to transform it to BytesInput when saving to disk. (like NautilusInput)

I'll double check this with andrea

[renatahodovan]

@tokatoka thanks for looking into it. I tried to run the newly added mruby benchmark of FuzzBench. First if failed due to the wrong parameters. When I applied this change - which removes the grammar and report parameter and adds the input corpus argument -

--- a/fuzzers/grimoire/fuzzer.py
+++ b/fuzzers/grimoire/fuzzer.py
@@ -77,6 +77,6 @@ def fuzz(input_corpus, output_corpus, target_binary):
     grammar = os.path.join(os.environ['OUT'], 'grammar.json')
     out = os.path.join(os.environ['OUT'], 'out')
     os.mkdir(out)
-    command += (['-r', output_corpus, '-o', out, '-g', grammar])
+    command += (['-i', input_corpus, '-o', output_corpus])
     print(command)
     subprocess.check_call(command, cwd=os.environ['OUT'])

then my archive corpus contained a queue subdir with *.metadata and empty *.lafl_lock files. The metadata files contained data in JSON format with such keys as metadata, map, exec_time, executions. I assumed these files encode some internal representation and then I saw that Nautilus and others have this DumpToDiskStage stage and when I've added this to Grimoire, then the corpus contained plain tests as expected. But honestly, this is just a speculative fix, which appeared to be working for me.

[tokatoka]

The metadata files contained data in JSON format with such keys as metadata, map, exec_time, executions.

yes but this is not related to grimoire. it's a very generic metadata used in libafl.

I assumed these files encode some internal representation and then I saw that Nautilus and others have this DumpToDiskStage stage

I think for nautilus we have NautilusInput thus we have dump stage, for grimoire we don't have GrimoireInput so we don't need DumpToDiskStage

[tokatoka]

i think the reason it failed that your failed is because fuzzers/grimoire is not updated.

you can just use the setup i used here #14

[renatahodovan]

@tokatoka I tried copying and pasting the mentioned Grimoire config files (build.Dockerfile, runner.Dockerfile, and runner.py) into the latest FuzzBench master. I also tried checking out your fork at the suggested revision, but the build failed in both cases. The master had Rust compatibility issues, while the old revision also had build problems.

I would really like to measure the performance of Grimoire, but I am not familiar with either Grimoire or Rust, so I have been struggling with it for days without success. Is there any chance you could update Grimoire to work on the latest master of FuzzBench?

[tokatoka]

Sorry for being late. I had other stuff to do last two weeks.

I pushed the fuzzer using this repo's grimoire on to https://github.com/AFLplusplus/fuzzbench (check the latest commit)
To test any target you want, you can run make run-libafl_grimoire-<fuzzer name>, or run debug-libafl_grimoire-<fuzzer name>.

Then if you see your fuzzer stuck again, can you attach with gdb to see and paste the backtrace?