diff --git a/README.md b/README.md index e69de29..0d2c5d3 100644 --- a/README.md +++ b/README.md @@ -0,0 +1,374 @@ +# Iozone Benchmark Wrapper + +## Description + +This wrapper facilitates the automated execution of the Iozone filesystem benchmark. Iozone is an industry-standard tool for measuring filesystem I/O performance across a wide range of operations including sequential reads/writes, random I/O, and memory-mapped I/O. + +The wrapper provides: +- Automated Iozone download, build, and execution. +- Multiple test modes: in-cache, out-of-cache, direct I/O, fsync, and mmap. +- Support for multiple filesystems (XFS, ext4, ext3, GFS, GFS2). +- Support for x86_64, i386, s390/s390x, and aarch64 architectures. +- Automatic memory-based problem sizing for in-cache and out-of-cache tests. +- Multi-iteration runs with automatic averaging and comparison. +- Result collection, processing, and verification. +- CSV and JSON output formats. +- System configuration metadata capture. +- Integration with test_tools framework. +- Optional Performance Co-Pilot (PCP) integration. +- Optional kernel tuned parameter comparison. +- LVM and raw block device support. + +## Command-Line Options + +``` +Test Mode Options (all enabled by default if none specified): + --incache: Run in-cache tests. + --incache_filelimit : Max file size for in-cache tests (default: 4096 MB). + --syncedincache: Run in-cache tests with fsync. + --mmapincache: Run in-cache tests with mmap. + --outofcache: Run out-of-cache tests. + --directio: Run direct I/O tests. + --dio_filelimit : Max file size for direct I/O tests (default: 4096 MB). + --all_test: Execute all predefined tests. + +Memory & Performance Tuning Options: + --eatmem: Use the eatmem program to reduce available memory for out-of-cache testing. + --eatmem_out_of_cache_start : Starting memory consumption for eatmem (default: 128). + --eatmem_out_of_cache_end : Ending memory consumption for eatmem (default: 1024). + --eatmem_free_memory : Memory to keep free when using eatmem (default: 180). + --tunecompare: Compare default vs. tuned kernel parameters. + --swap: Disable swap during testing, re-enable after. + +File & Filesystem Options: + --filesystems : Comma-separated list of filesystems to test (default: xfs). + --devices_to_use : Devices for filesystem creation (default: auto-detected via grab_disks). + --mount_location : Base directory for mount points (default: /iozone/iozone). + --file_count_list <1,2,...>: Number of files per mount point (default: 1). + --max_file_size : Total file size for testing (default: 10 GB). + --lvm_disk: Use LVM volumes instead of raw devices. + +Iozone-Specific Options: + --iozone_kit : Iozone package to download (default: iozone3_490). + --auto: Use auto-mode testing instead of throughput mode. + --page_size : Minimum file size for auto-mode (default: 1024 KB). + --quick : Divisor for in-cache memory to speed up tests (default: 1). + --test_type <0,1,2,...>: Iozone test types (0=write, 1=read, etc.; default: 0,1). + --iozone_options : Additional iozone arguments passed directly to the binary. + --iozone_umount: Remount filesystem between tests. + --outcache_multiplier : Multiplier for out-of-cache max size (default: 4). + +Output Options: + --results_dir : Output directory (default: current_dir/results). + --test_prefix : Prefix for result files (default: test_run). + +General test_tools options: + --tools_git : Git repo to retrieve the required tools from. + Default: https://github.com/redhat-performance/test_tools-wrappers + --test_tools_release : Version tag of test_tools to use. + --no_system_packages: Skip system package manager installation (dnf/yum/apt). + --no_pip_packages: Skip Python pip package installation. + --run_user : User running the test on the target system. Defaults to current user. + --sysname : System name used for determining config files. Defaults to hostname. + --run_label : Label for the test run. + --verify_skip: Skip test verification against output. + --json_skip: Skip JSON conversion of CSV results. + --debug: Enable bash -x output for debugging. + --verbose: Enable verbose shell output (set -x). + --help / --usage: Display usage information. +``` + +## What the Script Does + +The `iozone_run.sh` script performs the following workflow: + +1. **Environment Setup**: + - Clones the test_tools-wrappers repository if not present (default: ~/test_tools). + - Sources error codes and general setup utilities. + - Gathers hardware data for system characterization. + +2. **Package Installation**: + - Installs required dependencies via package_tool (gcc, make, bc, perl, wget, etc.). + - Dependencies are defined in iozone-wrapper.json for different OS variants (RHEL, Ubuntu). + +3. **System Preparation**: + - Captures CPU info, dmesg, and BIOS info via dmidecode. + - Detects CPU count, speeds, and NUMA topology. + - Calculates available disk space (uses 90% of available). + +4. **Binary Builds**: + - Compiles `create_file` utility for pre-allocating sparse test files. + - Compiles `eatmem` utility if out-of-cache memory reduction is needed. + - Downloads and builds Iozone from source (default: iozone3_490) or uses installed system version. + - Detects architecture for correct Iozone build target (linux-AMD64, linux-arm, linux-S390X, etc.). + +5. **Memory Parameter Calculation**: + - Detects total system memory from /proc/meminfo. + - Calculates in-cache file size as highest power of 2 fitting in available memory. + - Computes out-of-cache range based on memory and multiplier settings. + - Applies direct I/O file size limits. + +6. **Filesystem Setup**: + - For each configured filesystem type, obtains available block devices. + - Creates LVM volumes (if `--lvm_disk`) or formats raw devices directly. + - Mounts filesystems at configured mount points. + +7. **Test Execution**: + - Drops page caches before each test. + - For each enabled test mode (in-cache, in-cache+fsync, in-cache+mmap, direct I/O, out-of-cache): + - Optionally launches eatmem process to reduce available memory. + - Runs Iozone pinned to CPU 0 via taskset. + - Performs analysis on results via analysis-iozone.pl. + - Optionally compares default vs. tuned kernel parameters (if `--tunecompare`). + - Repeats for configured number of iterations. + +8. **Data Collection**: + - Captures system configuration (CPU, memory, NUMA topology, kernel version). + - Records test parameters and mode configurations. + - Logs timestamps for each filesystem test run. + - Optionally records PCP performance data. + +9. **Multi-Run Processing** (if iterations > 1): + - Averages results across all runs (average.sh, average_table.pl). + - Compares each individual run against the average (compare.sh). + - Flags regressions >5% and improvements >5%. + +10. **Result Processing**: + - Extracts performance metrics from analysis files. + - Generates CSV files with filesystem, test mode, and performance data. + - Creates JSON output via csv_to_json. + - Optionally sends results to PCP archive via results2pcp_multiple. + +11. **Verification**: + - Validates results against Pydantic schema (results_iozone_auto_schema.py or results_iozone_tput_schema.py). + - Ensures all required fields are present and valid. + - Uses csv_to_json and verify_results from test_tools. + +12. **Output**: + - Creates results directory with raw output, processed CSV/JSON, and system metadata. + - Saves all raw Iozone output files, analysis logs, and configuration dumps. + - Optionally saves PCP performance data. + - Archives results to configured storage location via save_results. + +## Dependencies + +Location of underlying workload: Downloaded from http://www.iozone.org/src/current/ (default: iozone3_490). + +**General packages required**: bc, gcc, git, make, wget, unzip, zip + +**Additional packages for platform-specific builds**: +- RHEL/Fedora: perf, perl-Math-BigInt, perl-Math-BigRat. +- Ubuntu/Debian: linux-tools-generic, libmath-bigint-perl. + +To run: +```bash +git clone https://github.com/redhat-performance/iozone-wrapper +cd iozone-wrapper/iozone +./iozone_run.sh +``` + +The script requires root privileges. It will automatically detect your CPU architecture and build Iozone for the correct target. + +## The Iozone Benchmark + +Iozone is a filesystem benchmark tool that generates and measures a variety of file operations. It tests I/O performance for the following operations: + +- **Initial Write**: First write to a new file. +- **Rewrite**: Overwriting an existing file. +- **Read**: Sequential read of a file. +- **Re-read**: Second sequential read (may benefit from caching). +- **Random Read**: Reading random positions within a file. +- **Random Write**: Writing to random positions within a file. +- **Backward Read**: Reading a file in reverse. +- **Record Rewrite**: Rewriting specific records within a file. +- **Stride Read**: Reading with a stride pattern. +- **Fwrite/Frewrite**: Write/rewrite using fwrite() library calls. +- **Fread/Freread**: Read/re-read using fread() library calls. + +### Test Modes + +1. **In-Cache**: File sizes fit within system memory. Tests filesystem and cache performance. +2. **In-Cache + fsync**: Same as in-cache but forces data to disk after writes. +3. **In-Cache + mmap**: Uses memory-mapped I/O for file operations. +4. **Direct I/O**: Bypasses the OS page cache, testing raw device performance. +5. **Out-of-Cache**: File sizes exceed available memory, forcing disk I/O. + +### Performance Metrics + +- **Auto mode**: Reports throughput in KB/s for each operation across combinations of record sizes and file sizes. The wrapper computes geometric means across all sizes to produce aggregate metrics. +- **Throughput mode**: Reports aggregate throughput with 1, 2, and 4 concurrent processes for each operation. + +## Output Files + +The results directory contains: + +- **results_iozone.csv**: CSV file with filesystem, test mode, and performance metrics +- **results_iozone.json**: Validated JSON output +- **Run_N/\/**: Per-iteration raw Iozone output and analysis logs + - `iozone___default.iozone`: Raw Iozone output + - `iozone__default_analysis+rawdata.log`: Analysis with geometric means +- **Average/**: Averaged results across all iterations (if iterations > 1) +- **Compare/**: Comparison of each run against the average (if iterations > 1) +- **config/**: System configuration snapshots (cpuinfo, dmesg, dmidecode, NUMA layout) +- **build-run.log**: Compilation and test execution log +- **meta_data.yml**: System metadata (CPU, memory, kernel version) +- **test_results_report**: PASS/FAIL/WARN status marker +- **PCP data** (if --use_pcp option used): Performance Co-Pilot monitoring data + +## Examples + +### Basic run with defaults +```bash +./iozone_run.sh +``` +This runs with: +- All test modes enabled (in-cache, in-cache+fsync, in-cache+mmap, direct I/O, out-of-cache). +- XFS filesystem. +- Automatic device detection. +- 1 iteration. +- Automatic memory-based problem sizing. + +### Run only in-cache tests +```bash +./iozone_run.sh --incache +``` +Tests only in-cache performance where file sizes fit within system memory. + +### Run with direct I/O only +```bash +./iozone_run.sh --directio +``` +Bypasses the OS page cache to test raw device performance. + +### Run on specific filesystems +```bash +./iozone_run.sh --filesystems xfs,ext4 +``` +Tests both XFS and ext4 filesystems. + +### Run on specific devices +```bash +./iozone_run.sh --devices_to_use /dev/sdb,/dev/sdc +``` +Uses specified block devices instead of auto-detection. + +### Run with LVM +```bash +./iozone_run.sh --lvm_disk --devices_to_use /dev/sdb +``` +Creates an LVM volume on the specified device for testing. + +### Run multiple iterations +```bash +./iozone_run.sh --iterations 3 +``` +Runs 3 iterations and generates averaged results and per-run comparisons. + +### Run in auto mode +```bash +./iozone_run.sh --auto +``` +Uses Iozone auto mode, which tests across a matrix of file sizes and record sizes. + +### Run with reduced memory (out-of-cache with eatmem) +```bash +./iozone_run.sh --outofcache --eatmem +``` +Reduces available memory to force out-of-cache I/O patterns. + +### Run quick in-cache test +```bash +./iozone_run.sh --incache --quick 4 +``` +Divides the in-cache memory size by 4 for faster testing. + +### Compare default vs. tuned kernel parameters +```bash +./iozone_run.sh --tunecompare +``` +Runs tests with both default and tuned kernel settings, then compares results. + +### Run with PCP monitoring +```bash +./iozone_run.sh --use_pcp +``` +Collects Performance Co-Pilot data during the run. + +### Combination example +```bash +./iozone_run.sh --incache --directio --filesystems xfs,ext4 --iterations 3 --use_pcp +``` +Runs in-cache and direct I/O tests on XFS and ext4, with 3 iterations and PCP monitoring. + +## How Memory Sizing Works + +The script automatically calculates file sizes for each test mode based on system memory: + +### In-Cache File Size +1. Detects total system memory from /proc/meminfo. +2. Calculates the highest power of 2 that fits within available memory: + ``` + file_size = 2^floor(log2(available_memory)) + ``` +3. Applies the `--quick` divisor if specified (default: 1). +4. Caps at `--incache_filelimit` (default: 4096 MB). + +### Out-of-Cache File Size +1. Starts at `--eatmem_out_of_cache_start` (default: 128 MB). +2. Ends at `--eatmem_out_of_cache_end` (default: 1024 MB). +3. Multiplied by `--outcache_multiplier` (default: 4) for maximum test size. +4. If `--eatmem` is enabled, reduces available memory to force disk I/O. + +### Direct I/O File Size +- Capped at `--dio_filelimit` (default: 4096 MB). +- Uses the same base calculations as in-cache sizing. + +## Return Codes + +The script uses standardized error codes from test_tools error_codes: +- **0**: Success +- **1**: Generic error (directory creation, build failures, disk space errors). +- **101**: Git clone failure +- **E_GENERAL**: General execution errors (package installation, build failures, test execution failures, validation failures). + +Exit codes indicate specific failure points for automated testing workflows. The `test_results_report` file in the results directory contains the overall test status (PASS, FAIL, or WARN). + +## Notes + +### Architecture Support +- **x86_64**: Built as linux-AMD64. +- **i386/i486/i586/i686**: Built as linux. +- **aarch64**: Built as linux-arm. +- **s390/s390x**: Built as linux-S390 or linux-S390X. + +### Filesystem Support +- **XFS**: Default filesystem. Well-suited for large files and high throughput. +- **ext4/ext3**: Supported for comparison testing. +- **GFS/GFS2**: Supported for cluster filesystem testing (requires cluster setup). + +### Root Privileges +The script requires root privileges to create filesystems, mount devices, drop page caches, and modify kernel parameters. + +### Memory Considerations +- In-cache tests size files to fit within system memory, testing cache and filesystem performance. +- Out-of-cache tests deliberately exceed available memory, forcing actual disk I/O. +- The `--eatmem` option uses a helper program that allocates memory to further reduce what is available to the filesystem cache. +- The `--quick` option can significantly reduce in-cache test time by using smaller file sizes. + +### Multi-Run Analysis +- When running multiple iterations, the wrapper automatically computes averaged results. +- Each individual run is compared against the average, with regressions >5% and improvements >5% flagged. +- Geometric mean is used to aggregate performance across file sizes and record sizes. + +### Kernel Tuning Comparison +- The `--tunecompare` option runs each test twice: once with default kernel parameters and once with tuned parameters. +- This is useful for evaluating the impact of sysctl tuning on I/O performance. + +### Troubleshooting +- If Iozone fails to build, verify that gcc and make are installed. +- If filesystem creation fails, verify that the specified devices are available and not in use. +- The script requires root privileges; running as non-root will fail immediately. +- Use `--verbose` to enable shell tracing (set -x) for debugging. +- Check `build-run.log` in the results directory for compilation and execution details. +- If performance is unexpectedly low, check that no other I/O-intensive workloads are running. +- Use `--use_pcp` to collect detailed performance counters for analysis.