Gather distributions for problematic performance comparison

[gevtushenko]

This issue can be closed with data samples in a internal repo where current nvbench_compare.py behavior is not enough / unstable.

The ideal would be two distributions from two benchmarking runs where the existing script would say they are different, but in actuality are the "same" because the algorithm itself did not change.

The intention for this data is to provide an example of distributions to test new comparison logic that should tell us that they are indeed the same distribution.

[gevtushenko]

I'll try to find problematic benchmarks, otherwise might need to generate synthetic data for now

[oleksandr-pavlyk]

One possibly interesting distribution is associated with "copy_sweep_grid_shape" benchmark for -a "BlockSize=1024" -a "NumBlocks=64".

Distribution of sample times for copy_sweep_grid_shape benchmark

n [1]: import plotext as plt

In [2]: import numpy as np

In [3]: !jq '.benchmarks[] | select(any(.. | values; . == "axes-d0-bin/14.bin")) | .name' axes-d0
"copy_sweep_grid_shape"

In [4]: !jq '.benchmarks[] | select(any(.. | values; . == "axes-d0-bin/14.bin")) | .states[] | select(an
      ⋮ y(.. | values; . == "axes-d0-bin/14.bin")) | .axis_values' axes-d0
[
  {
    "name": "BlockSize",
    "type": "int64",
    "value": "1024"
  },
  {
    "name": "NumBlocks",
    "type": "int64",
    "value": "64"
  }
]

In [5]: t = np.fromfile('axes-d0-bin/14.bin', dtype='f4')

In [6]: v, c = np.unique(t, return_counts=True)

In [7]: plt.clf(); plt.simple_bar(1000 * 1000 * v, c, width = 2*sum(c)); plt.show()
805.888   ▇▇ 1.0
806.91205 ▇▇ 1.0
807.93604 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 7.0
808.736   ▇▇ 1.0
808.83203 ▇▇ 1.0
808.96    ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 11.0
809.024   ▇▇ 1.0
809.05597 ▇▇▇▇▇▇ 3.0
809.9841  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 20.0
810.176   ▇▇ 1.0
810.208   ▇▇ 1.0
810.272   ▇▇ 1.0
810.784   ▇▇ 1.0
810.81604 ▇▇▇▇ 2.0
810.94403 ▇▇ 1.0
810.976   ▇▇ 1.0
811.008   ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 40.0
811.072   ▇▇ 1.0
811.104   ▇▇ 1.0
811.264   ▇▇ 1.0
811.808   ▇▇ 1.0
811.872   ▇▇ 1.0
811.9681  ▇▇▇▇ 2.0
812.032   ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 42.0
812.16003 ▇▇▇▇▇▇ 3.0
812.192   ▇▇ 1.0
812.768   ▇▇ 1.0
812.80005 ▇▇ 1.0
812.832   ▇▇ 1.0
812.864   ▇▇▇▇ 2.0
813.056   ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 43.0
813.12    ▇▇ 1.0
813.184   ▇▇ 1.0
813.792   ▇▇▇▇ 2.0
813.88806 ▇▇ 1.0
813.92    ▇▇ 1.0
813.98395 ▇▇ 1.0
814.048   ▇▇▇▇ 2.0
814.08    ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 46.0
814.14404 ▇▇ 1.0
814.24    ▇▇ 1.0
814.304   ▇▇ 1.0
814.36804 ▇▇ 1.0
814.848   ▇▇ 1.0
815.104   ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 34.0
815.136   ▇▇ 1.0
815.328   ▇▇ 1.0
815.84    ▇▇ 1.0
816.032   ▇▇▇▇ 2.0
816.12805 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 19.0
816.256   ▇▇ 1.0
816.4161  ▇▇ 1.0
816.89606 ▇▇ 1.0
817.15204 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 16.0
817.44    ▇▇ 1.0
818.17596 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 7.0
819.04    ▇▇ 1.0
819.19995 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 8.0
819.488   ▇▇ 1.0
820.224   ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 8.0
821.248   ▇▇▇▇▇▇ 3.0
822.24005 ▇▇ 1.0
822.272   ▇▇ 1.0
823.296   ▇▇▇▇ 2.0
824.064   ▇▇ 1.0
824.32    ▇▇ 1.0
826.368   ▇▇▇▇ 2.0
827.39197 ▇▇▇▇ 2.0
829.44    ▇▇ 1.0
831.488   ▇▇ 1.0
844.896   ▇▇ 1.0
860.16    ▇▇ 1.0
864.256   ▇▇ 1.0

The distribution of times is clearly multi-modal. Execution times appear to only weakly correlated with frequencies.

Analysis of association between kernel execution times and SM frequencies

In [32]: t = np.fromfile('axes-d0-bin/14.bin', dtype='f4')

In [33]: f = np.fromfile('axes-d0-freqs-bin/14.bin', dtype='f4')

In [36]: def summary(data, f):
    ...:     return [np.mean(data) * f, f * np.percentile(data, [5, 25, 50, 75, 95])]
    ...: 

In [37]: {p: summary(t[np.where(f >= np.percentile(f, p))], 1000 * 1000) for p in [0, 10, 20, 25, 35, 50, 65, 75, 90, 100]}
Out[37]: 
{0: [np.float32(814.12646),
  array([808.96000145, 811.05600111, 813.05601634, 815.10399468,
         820.22399874])],
 10: [np.float32(814.098),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         820.22399874])],
 20: [np.float32(814.25244),
  array([808.96000145, 811.26397708, 813.05601634, 815.10399468,
         820.22399874])],
 25: [np.float32(814.2978),
  array([808.96000145, 811.43999705, 813.05601634, 815.10399468,
         820.22399874])],
 35: [np.float32(814.4563),
  array([808.96000145, 812.03196896, 813.05601634, 815.10399468,
         821.09438954])],
 50: [np.float32(814.6035),
  array([809.98404883, 812.03196896, 814.08000551, 815.10399468,
         820.88959171])],
 65: [np.float32(814.7154),
  array([809.5664205 , 812.03196896, 814.08000551, 815.10399468,
         820.22399874])],
 75: [np.float32(814.35785),
  array([809.98404883, 812.03196896, 814.08000551, 815.10399468,
         819.55836795])],
 90: [np.float32(813.99915),
  array([810.85439015, 812.03196896, 813.56801093, 815.10399468,
         820.22399874])],
 100: [np.float32(815.104),
  array([815.10399468, 815.10399468, 815.10399468, 815.10399468,
         815.10399468])]}

In [38]: {p: summary(t[np.where(f <= np.percentile(f, p))], 1000 * 1000) for p in [0, 10, 20, 25, 35, 50, 65, 75, 90, 100]}
Out[38]: 
{0: [np.float32(813.056),
  array([813.05601634, 813.05601634, 813.05601634, 813.05601634,
         813.05601634])],
 10: [np.float32(814.3789),
  array([811.00797979, 812.03196896, 814.08000551, 816.12804206,
         820.37759712])],
 20: [np.float32(813.64044),
  array([809.75203309, 811.00797979, 813.42400517, 815.10399468,
         817.92002311])],
 25: [np.float32(813.7249),
  array([809.04638162, 811.00797979, 813.05601634, 815.10399468,
         820.5311955 ])],
 35: [np.float32(813.5168),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         819.19995137])],
 50: [np.float32(813.6494),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         820.22399874])],
 65: [np.float32(813.80786),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         820.22399874])],
 75: [np.float32(814.0494),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         821.24798791])],
 90: [np.float32(814.1407),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         820.37759712])],
 100: [np.float32(814.12646),
  array([808.96000145, 811.05600111, 813.05601634, 815.10399468,
         820.22399874])]}

In [42]: np.corrcoef(t, f)
Out[42]: 
array([[1.        , 0.05859936],
       [0.05859936, 1.        ]])

[oleksandr-pavlyk]

I have created a repo nvbench-data-compare-study in the internal GitLab with scripts, notebooks and datasets

[oleksandr-pavlyk]

The logic of nvbench_compare.py is to retrieve "nv/cold/time/gpu/mean" value (arithmetic average of CUDA Event-based isolated measurements) for reference and candidate runs. It also gets "nv/cold/time/gpu/stdev/relative" value ( ratio of unbiased standard-deviation to the mean value, called noise) for both runs.

The decision logic is as follows:

• We compute min_noise := min(ref_noise, cmp_noise) and frac_diff = (cmp_time - ref_time) / ref_time
• If abs(frac_diff) <= min_noise the performance of the candidate is considered the "🔵 SAME" as the performance of the reference.
• Otherwise, we decide between "🔴 SLOW" and "🟢 FAST" based on sign of frac_diff
• Script only reports this conclusion if abs(frac_diff) >= threshold, but default value of the threshold is 0.0, so everything is shown

The biggest issue practically encountered with this logic is high rate of SLOW/FAST claims comparing repeated runs of benchmarks compiled with the same code-base, same compilation flags and executed on the same HW with the same driver stack.

Sample output of nvbench_compare.py

$ python nvbench_compare.py copy1d_run1.json copy1d_run2.json --no-color
['copy1d_run1.json', 'copy1d_run2.json']
# copy_1d

## [0] NVIDIA RTX A6000

|  T{ct}  |  BlockSize{ct}  |  Elements{io}  |  ItemsPerThread{io}  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |      Diff |   %Diff |  Status  |
|---------|-----------------|----------------|----------------------|------------|-------------|------------|-------------|-----------|---------|----------|
|   U8    |       64        |      2^16      |          2           |   4.132 us |       5.88% |   4.130 us |       5.95% | -0.002 us |  -0.04% | 🔵 SAME  |
|   U8    |       64        |      2^17      |          2           |   4.379 us |       9.72% |   4.373 us |       9.73% | -0.006 us |  -0.13% | 🔵 SAME  |
|   U8    |       64        |      2^18      |          2           |   5.490 us |       8.89% |   5.496 us |       8.78% |  0.006 us |   0.11% | 🔵 SAME  |
|   U8    |       64        |      2^19      |          2           |   6.856 us |       6.58% |   7.168 us |       0.00% |  0.312 us |   4.55% | 🔴 SLOW  |
|   U8    |       64        |      2^20      |          2           |   9.697 us |       5.12% |   9.676 us |       5.11% | -0.021 us |  -0.22% | 🔵 SAME  |
|   U8    |       64        |      2^21      |          2           |  15.312 us |       1.87% |  15.418 us |       1.87% |  0.106 us |   0.70% | 🔵 SAME  |
|   U8    |       64        |      2^22      |          2           |  26.629 us |       1.21% |  26.882 us |       1.62% |  0.253 us |   0.95% | 🔵 SAME  |
|   U8    |       64        |      2^23      |          2           |  49.067 us |       0.77% |  48.655 us |       1.77% | -0.412 us |  -0.84% | 🟢 FAST  |
|   U8    |       64        |      2^24      |          2           |  93.966 us |       0.47% |  91.876 us |       0.50% | -2.091 us |  -2.22% | 🟢 FAST  |
|   U16   |       64        |      2^16      |          2           |   4.136 us |       5.15% |   4.001 us |       8.75% | -0.135 us |  -3.27% | 🔵 SAME  |
|   U16   |       64        |      2^17      |          2           |   4.622 us |      10.70% |   4.467 us |      10.62% | -0.155 us |  -3.35% | 🔵 SAME  |
|   U16   |       64        |      2^18      |          2           |   5.500 us |      11.49% |   5.453 us |       8.46% | -0.047 us |  -0.86% | 🔵 SAME  |
|   U16   |       64        |      2^19      |          2           |   7.110 us |       4.83% |   7.063 us |       5.21% | -0.046 us |  -0.65% | 🔵 SAME  |
|   U16   |       64        |      2^20      |          2           |  10.240 us |       3.02% |  10.215 us |       3.12% | -0.024 us |  -0.24% | 🔵 SAME  |
|   U16   |       64        |      2^21      |          2           |  16.870 us |       3.22% |  16.872 us |       3.20% |  0.002 us |   0.01% | 🔵 SAME  |
|   U16   |       64        |      2^22      |          2           |  29.939 us |       1.59% |  29.993 us |       2.16% |  0.054 us |   0.18% | 🔵 SAME  |
|   U16   |       64        |      2^23      |          2           |  56.448 us |       0.84% |  56.435 us |       0.78% | -0.013 us |  -0.02% | 🔵 SAME  |
|   U16   |       64        |      2^24      |          2           | 109.429 us |       0.55% | 109.408 us |       0.55% | -0.020 us |  -0.02% | 🔵 SAME  |
|   U32   |       64        |      2^16      |          2           |   4.335 us |       9.93% |   4.298 us |       9.39% | -0.037 us |  -0.86% | 🔵 SAME  |
|   U32   |       64        |      2^17      |          2           |   5.078 us |       6.39% |   5.120 us |       0.00% |  0.042 us |   0.83% | 🔴 SLOW  |
|   U32   |       64        |      2^18      |          2           |   6.826 us |       7.14% |   6.831 us |       7.26% |  0.005 us |   0.07% | 🔵 SAME  |
|   U32   |       64        |      2^19      |          2           |   9.622 us |       5.90% |   9.601 us |       5.72% | -0.021 us |  -0.22% | 🔵 SAME  |
|   U32   |       64        |      2^20      |          2           |  15.360 us |       0.00% |  15.398 us |       3.41% |  0.038 us |   0.25% | 🔴 SLOW  |
|   U32   |       64        |      2^21      |          2           |  27.752 us |       2.72% |  27.701 us |       2.55% | -0.052 us |  -0.19% | 🔵 SAME  |
|   U32   |       64        |      2^22      |          2           |  52.624 us |       1.47% |  52.681 us |       1.30% |  0.056 us |   0.11% | 🔵 SAME  |
|   U32   |       64        |      2^23      |          2           | 101.407 us |       1.01% | 101.297 us |       1.14% | -0.110 us |  -0.11% | 🔵 SAME  |
|   U32   |       64        |      2^24      |          2           | 198.741 us |       0.71% | 198.737 us |       0.72% | -0.004 us |  -0.00% | 🔵 SAME  |
|   U64   |       64        |      2^16      |          2           |   5.041 us |       7.13% |   5.022 us |       7.03% | -0.019 us |  -0.37% | 🔵 SAME  |
|   U64   |       64        |      2^17      |          2           |   6.607 us |       8.05% |   6.668 us |       7.81% |  0.061 us |   0.92% | 🔵 SAME  |
|   U64   |       64        |      2^18      |          2           |   9.774 us |       6.05% |   9.755 us |       5.85% | -0.019 us |  -0.20% | 🔵 SAME  |
|   U64   |       64        |      2^19      |          2           |  16.384 us |       0.00% |  16.481 us |       4.05% |  0.097 us |   0.59% | 🔴 SLOW  |
|   U64   |       64        |      2^20      |          2           |  29.341 us |       3.02% |  29.604 us |       2.99% |  0.263 us |   0.90% | 🔵 SAME  |
|   U64   |       64        |      2^21      |          2           |  52.943 us |       1.55% |  53.038 us |       1.29% |  0.095 us |   0.18% | 🔵 SAME  |
|   U64   |       64        |      2^22      |          2           | 103.042 us |       1.09% | 102.987 us |       1.00% | -0.055 us |  -0.05% | 🔵 SAME  |
|   U64   |       64        |      2^23      |          2           | 200.435 us |       0.69% | 200.257 us |       0.52% | -0.177 us |  -0.09% | 🔵 SAME  |
|   U64   |       64        |      2^24      |          2           | 395.806 us |       0.37% | 395.773 us |       0.42% | -0.033 us |  -0.01% | 🔵 SAME  |

# Summary

- Total Matches: 36
  - Pass    (diff <= min_noise): 30
  - Unknown (infinite noise):    0
  - Failure (diff > min_noise):  6

Both JSON files were generated from the same ELF benchmark run on the same machine:

./a.out -d 0 -a "BlockSize{ct}=64" -a "ItemsPerThread{io}=2" -a "Elements{io}[pow2]=[16:24]" --stopping-criterion entropy --no-batch --json copy1d_run1.json
./a.out -d 0 -a "BlockSize{ct}=64" -a "ItemsPerThread{io}=2" -a "Elements{io}[pow2]=[16:24]" --stopping-criterion entropy --no-batch --json copy1d_run2.json

The NVBench-based benchmark reports "nv/code/time/gpu/min", "nv/code/time/gpu/max" which defines an interval over which run-times varied, with mean and standard deviation providing additional information.

We could confidently order performances only if corresponding min/max intervals do not overlap. If intervals do overlap, we need either full samples arrays or some probabilistic model consistent with reported min, max, average and sample standard deviation.