[v2] Add more granular metrics to benchmarking framework: plugin-load time, static imports time, client creation time.

[aemous]

Description of changes:

• The benchmarking framework (callable from ./scripts/performance/run-benchmarks) now emits plugin-load time, static imports time, and client creation time.
• Fixed bug with performance framework on IMDS-compatible instances where the CLI would attempt to reach the IMDS endpoints to retrieve credentials but the stubbed responses did not support this. Now, we supply mock credentials in the stubbed config files in each of our stubbed JSON benchmarks.
• In vendored botocore, emit before-create-ciient and after-create-client before and after client creation, respectively.
• In create_clidriver, add a new optional event_hooks argument to allow callers to provide their own event emitter. This change was needed so the benchmark framework can register against the load_plugins event, which is emitted during create_clidriver.
• Modified --debug-dir parameter in run-benchmarks script so that it supports relative paths (previously it only supported absolute paths).

Description of tests:

• Manually ran the benchmarking framework locally and on an IMDS-compatible EC2 instance, and verified the numbers are successfully being emitted, and are in an expected range aligned with previous benchmarking efforts.
• Manually tested the --debug-dir parameter with relative and absolute paths, and verified expected behavior in both cases.
• Successfully ran an internal pre-prod build workflow.

Example plugin-load time emission:

{
      "name": "cloudwatch.getmetricdata.plugins.import.time",
      "description": "Total time spent loading all built-in plugins.",
      "unit": "Seconds",
      "dimensions": [],
      "measurements": [
        0.08212709426879883
      ]
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[hssyoo]

Looks fine but the static import measurement seems a bit dubious. I'd like to understand how we plan on using this measurement, what we expect it to catch, how we would react.

[hssyoo]

What are we measuring here exactly?

[aemous]

This is the time to import all modules used by the AWS CLI at runtime. It's similar to the granular benchmarking that you previously implemented where you measured "Python Library Imports."

It let's us measure the total amount of time spent during the AWS CLI runtime (i.e. after interpreter started) that is spent importing modules, before we start running any initialization code.

Looks fine but the static import measurement seems a bit dubious. I'd like to understand how we plan on using this measurement, what we expect it to catch, how we would react.

It is largely prompted by our previous observation that prompt_toolkit took up a significant fraction of runtime. It embeds the philosophy "if module imports previously took up too much runtime, we should monitor the metric over time so we don't lose track of it, to prevent future regressions."

During my dev builds, we're observing 119 ms of time taken during static imports. This starts off as a baseline, that we track changes in over time.

It also automatically implements the measure that we previously did ad-hoc. When we benchmarked it ad-hoc we found useful results (particularly hefty imports). So in my view, if it was useful once, then it should be useful to track in the long term.

"What we expect it to catch, how we would react"

For the most part, it should catch large regressions. If we notice this number change significantly, it would prompt us to investigate the regression, and consider refactoring code (e.g. lazy initialization). Overall, this should help us reduce the time between (A) unintentionally increasing command execution time due to a new hefty import and (B) allocating time to investigate and mitigate the regression.