Compare Inputs in Results HTML

[cdesch]

Is there any way to configure benchee or benchee_html to compare the inputs on one graph?

I can do this with something like the following:

Benchee.run(
  %{
    "tx_benchmark 100" => fn -> TraditionalLedger.run_benchmark_file("./data/transactions_100.csv") end,
    "tx_benchmark 1000" => fn -> TraditionalLedger.run_benchmark_file("./data/transactions_1000.csv") end,
    "tx_benchmark 10000" => fn -> TraditionalLedger.run_benchmark_file("./data/transactions_10000.csv") end,
  },
  formatters: [
    {Benchee.Formatters.Console, extended_statistics: true},
    {Benchee.Formatters.HTML, file: "benchmarks/results.html"},
  ],
  after_each: fn _input -> reset_database() end,
)

But doing it like the following "feels" better in terms of the code, but the HTML comparison charts will put these on separate pages:

Benchee.run(
  %{
    "tx_benchmark" => fn input -> TraditionalLedger.run_benchmark_file(input) end,
  },
  formatters: [
    {Benchee.Formatters.Console, extended_statistics: true},
    {Benchee.Formatters.HTML, file: "benchmarks/results.html"},
  ],
  inputs: %{
    "100" => "./data/transactions_100.csv",
    "1_000" => "./data/transactions_1000.csv",
    "10_000" => "./data/transactions_10000.csv",
  },
  after_each: fn _input -> reset_database() end,
)

And output the following:

But I think this may just be a personal preference, but I could see there a use case for comparing inputs against each other in this fashion for benchmarking performance for different applications.