SystemController ERROR - Failed to append benchmark result

[Confuzzle]

Cannot generate test result after testing in Chinese language “Windows 11 专业版” on Intel Ultra 9 285H's NPU

I got

[08-13-2025 10:14:22.223] BenchmarkValidationLogger ERROR - Invalid UTF-8 string found in JSON object: Microsoft Windows 11 רҵ��
[08-13-2025 10:14:22.223] BenchmarkValidationLogger ERROR - Invalid UTF-8 string found in BenchmarkResult
[08-13-2025 10:14:22.223] SystemController ERROR - Failed to append benchmark result

Seems to be an issue about Chinese character, full debug log listed as below

[08-13-2025 09:40:55.948] SystemController INFO  - Configuration tested by MLCommons
[08-13-2025 09:40:55.948] SystemController WARN  - Temp path is not set in the config; using default system temp path: C:\Users\zy116\AppData\Local\Temp\MLPerf\1.0.0
[08-13-2025 09:40:57.227] main INFO  - Starting to run scenario llama3
[08-13-2025 09:40:57.229] SystemController INFO  - 
Downloading necessary files...

[08-13-2025 09:40:57.230] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/models/NativeOpenVINO/NPU/openvino_model.xml
[08-13-2025 09:40:57.233] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/models/NativeOpenVINO/NPU/openvino_model.xml
[08-13-2025 09:40:57.233] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/models/NativeOpenVINO/NPU/openvino_model.bin
[08-13-2025 09:45:53.034] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/models/NativeOpenVINO/NPU/openvino_model.bin
[08-13-2025 09:45:53.034] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/models/NativeOpenVINO/NPU/IHV-OV-tokenizers.zip
[08-13-2025 09:45:57.472] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/models/NativeOpenVINO/NPU/IHV-OV-tokenizers.zip
[08-13-2025 09:45:57.472] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/content_generation/greedy-prompt_cot.39329.json
[08-13-2025 09:45:59.463] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/content_generation/greedy-prompt_cot.39329.json
[08-13-2025 09:45:59.463] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/content_generation/greedy-prompt_t0.677207.json
[08-13-2025 09:46:01.021] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/content_generation/greedy-prompt_t0.677207.json
[08-13-2025 09:46:01.021] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/creative_writing/greedy-prompt_niv.123362.json
[08-13-2025 09:46:02.632] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/creative_writing/greedy-prompt_niv.123362.json
[08-13-2025 09:46:02.632] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/creative_writing/greedy-prompt_niv.77134.json
[08-13-2025 09:46:04.080] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/creative_writing/greedy-prompt_niv.77134.json
[08-13-2025 09:46:04.080] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/summarization/greedy-prompt_flan.1221607.json
[08-13-2025 09:46:05.618] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/summarization/greedy-prompt_flan.1221607.json
[08-13-2025 09:46:05.618] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/summarization/greedy-prompt_flan.1221694.json
[08-13-2025 09:46:07.070] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/summarization/greedy-prompt_flan.1221694.json
[08-13-2025 09:46:07.070] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/summarization2k/greedy-prompt_flan.1854150.json
[08-13-2025 09:46:08.503] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/summarization2k/greedy-prompt_flan.1854150.json
[08-13-2025 09:46:08.503] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/summarization2k/greedy-prompt_flan.997680.json
[08-13-2025 09:46:10.054] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/summarization2k/greedy-prompt_flan.997680.json
[08-13-2025 09:46:10.054] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/code_analysis/command_parser_cpp_2k.json
[08-13-2025 09:46:11.786] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/code_analysis/command_parser_cpp_2k.json
[08-13-2025 09:46:11.786] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/code_analysis/performance_counter_group_cpp_2k.json
[08-13-2025 09:46:13.306] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/data/code_analysis/performance_counter_group_cpp_2k.json
[08-13-2025 09:46:13.306] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/generation-greedy-results.json
[08-13-2025 09:46:14.678] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/generation-greedy-results.json
[08-13-2025 09:46:14.678] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/icuuc70.dll
[08-13-2025 09:46:18.587] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/icuuc70.dll
[08-13-2025 09:46:18.587] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_hetero_plugin.dll
[08-13-2025 09:46:20.737] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_hetero_plugin.dll
[08-13-2025 09:46:20.737] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/icudt70.dll
[08-13-2025 09:46:25.097] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/icudt70.dll
[08-13-2025 09:46:25.097] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_auto_plugin.dll
[08-13-2025 09:46:27.341] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_auto_plugin.dll
[08-13-2025 09:46:27.341] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_tensorflow_lite_frontend.dll
[08-13-2025 09:46:30.124] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_tensorflow_lite_frontend.dll
[08-13-2025 09:46:30.124] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino.dll
[08-13-2025 09:46:34.389] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino.dll
[08-13-2025 09:46:34.389] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_auto_batch_plugin.dll
[08-13-2025 09:46:37.225] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_auto_batch_plugin.dll
[08-13-2025 09:46:37.225] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_c.dll
[08-13-2025 09:46:39.718] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_c.dll
[08-13-2025 09:46:39.718] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_genai.dll
[08-13-2025 09:46:43.100] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_genai.dll
[08-13-2025 09:46:43.100] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_intel_cpu_plugin.dll
[08-13-2025 09:46:49.292] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_intel_cpu_plugin.dll
[08-13-2025 09:46:49.292] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_intel_gpu_plugin.dll
[08-13-2025 09:47:41.432] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_intel_gpu_plugin.dll
[08-13-2025 09:47:41.432] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_onnx_frontend.dll
[08-13-2025 09:47:45.178] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_onnx_frontend.dll
[08-13-2025 09:47:45.178] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_intel_npu_plugin.dll
[08-13-2025 09:47:48.035] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_intel_npu_plugin.dll
[08-13-2025 09:47:48.035] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_ir_frontend.dll
[08-13-2025 09:47:50.338] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_ir_frontend.dll
[08-13-2025 09:47:50.338] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_paddle_frontend.dll
[08-13-2025 09:47:54.036] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_paddle_frontend.dll
[08-13-2025 09:47:54.036] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_pytorch_frontend.dll
[08-13-2025 09:47:56.878] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_pytorch_frontend.dll
[08-13-2025 09:47:56.878] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_tokenizers.dll
[08-13-2025 09:47:59.754] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_tokenizers.dll
[08-13-2025 09:47:59.754] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_tensorflow_frontend.dll
[08-13-2025 09:48:02.620] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/openvino_tensorflow_frontend.dll
[08-13-2025 09:48:02.620] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/plugins.xml
[08-13-2025 09:48:04.023] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/plugins.xml
[08-13-2025 09:48:04.023] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/tbb12.dll
[08-13-2025 09:48:05.943] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/tbb12.dll
[08-13-2025 09:48:05.943] TaskScheduler DEBUG - Download Scheduler executing task: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/tbbmalloc.dll
[08-13-2025 09:48:07.574] TaskScheduler DEBUG - Download Scheduler task execution completed: Download https://client.mlcommons-storage.org/deps/1.0/binaries/Windows/x64/NativeOpenVINO/tbbmalloc.dll
[08-13-2025 09:48:07.574] TaskScheduler DEBUG - Download Scheduler all tasks completed.
[08-13-2025 09:48:07.806] SystemController INFO  - Unpacking downloaded files...
[08-13-2025 09:48:07.806] SystemController DEBUG - Calculating total uncompressed size of zip file: data\llama3/14656443072668365918\IHV-OV-tokenizers.zip
[08-13-2025 09:48:07.807] SystemController DEBUG - Unpacking files from zip file: data\llama3/14656443072668365918\IHV-OV-tokenizers.zip
[08-13-2025 09:48:07.950] main INFO  - Preparing for inferences execution...
[08-13-2025 09:48:10.834] main INFO  - 
Verifying files...
[08-13-2025 09:48:10.845] main INFO  - 
[08-13-2025 09:48:11.865] SystemController INFO  - Running benchmarks for EPs NativeOpenVINO{"device_id":0,"device_name":"Intel(R) AI Boost","device_type":"NPU"} on model "Llama-3.1-8B-Instruct_ov-int4-CHw" (https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/models/NativeOpenVINO/NPU/openvino_model.xml -> data\llama3\14656443072668365918\openvino_model.xml), added 1 task(s)

[08-13-2025 09:48:11.867] TaskScheduler DEBUG - Benchmark Scheduler - llama3 executing task: Benchmark NativeOpenVINO
[08-13-2025 09:48:11.867] SystemController INFO  - 
Preparing to run the benchmark for the EP NativeOpenVINO{"device_id":0,"device_name":"Intel(R) AI Boost","device_type":"NPU"}
[08-13-2025 09:48:11.867] SystemController INFO  - power_begin - start llama3 NativeOpenVINO {"device_id":0,"device_name":"Intel(R) AI Boost","device_type":"NPU"}
[08-13-2025 10:14:22.222] SystemController INFO  - Model llama3, model path Result: https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/models/NativeOpenVINO/NPU/openvino_model.xml
[08-13-2025 10:14:22.222] SystemController INFO  - power_end - end llama3 NativeOpenVINO {"device_id":0,"device_name":"Intel(R) AI Boost","device_type":"NPU"}
[08-13-2025 10:14:22.223] BenchmarkValidationLogger ERROR - Invalid UTF-8 string found in JSON object: Microsoft Windows 11 רҵ��
[08-13-2025 10:14:22.223] BenchmarkValidationLogger ERROR - Invalid UTF-8 string found in BenchmarkResult
[08-13-2025 10:14:22.223] SystemController ERROR - Failed to append benchmark result
[08-13-2025 10:14:22.223] TaskScheduler DEBUG - Benchmark Scheduler - llama3 task execution completed: Benchmark NativeOpenVINO
[08-13-2025 10:14:22.223] TaskScheduler DEBUG - Benchmark Scheduler - llama3 all tasks completed.
[08-13-2025 10:14:23.599] main INFO  - 

Finished running of scenario llama3 in 1572.753283 seconds
[08-13-2025 10:14:23.599] DisplayRecentResults INFO  - 
Benchmark Results 08-13-2025 10:14:23:

== Application Version: 1.0.0.79a71d2 MLCommons Standard Build ==
== Configuration tested by MLCommons ==

Model Variants:

1. Llama-3.1-8B-Instruct_ov-int4-CHw (https://client.mlcommons-storage.org/deps/1.0/scenario_files/llm/llama3/models/NativeOpenVINO/NPU/openvino_model.xml)

Shared Information:

    Assets:
    Input Files:
        greedy-prompt_cot.39329.json
        greedy-prompt_t0.677207.json
        greedy-prompt_niv.123362.json
        greedy-prompt_niv.77134.json
        greedy-prompt_flan.1221607.json
        greedy-prompt_flan.1221694.json
        greedy-prompt_flan.1854150.json
        greedy-prompt_flan.997680.json
        command_parser_cpp_2k.json
        performance_counter_group_cpp_2k.json
    Iterations: 3
    WarmUp Iterations: 1

Execution Summary:

Model: Llama-3.1-8B-Instruct_ov-int4-CHw (#1):
    NPU:
        NativeOpenVINO {"device_id":0,"device_name":"Intel(R) AI Boost","device_type":"NPU"}:
            Error: Invalid UTF-8 string found in output.
            Check Logs/error.log and Logs/llama3_executor.log for more details.

[swasson488]

Assigned some folks to this issue. Can we please investigate?

[ZimengLiu95]

I met the problem, too. I tried to set UTF-8 in the current session of Windows Powershell using the following codes, but it doesn't work.

$OutputEncoding = [System.Text.UTF8Encoding]::new($false)
[Console]::InputEncoding  = [System.Text.UTF8Encoding]::new($false)
[Console]::OutputEncoding = [System.Text.UTF8Encoding]::new($false)

Then I use the input command

.\mlperf-windows.exe -c .\Llama3.1\Intel_NativeOpenVINO_GPU_Default.json

And the error is

Execution Summary:

Model: Llama-3.1-8B-Instruct_ov-int4-GRw (#1):
    GPU:
        NativeOpenVINO {"device_id":0,"device_name":"Intel(R) Arc(TM) 140T GPU (16GB) (iGPU)","device_type":"GPU"}:
            Error: Invalid UTF-8 string found in output.
            Check Logs/error.log and Logs/llama3_executor.log for more details.

Check the error.log as follows:

[08-21-2025 15:36:44.658] BenchmarkValidationLogger ERROR - Invalid UTF-8 string found in JSON object: Microsoft Windows 11 专业版
[08-21-2025 15:36:44.658] BenchmarkValidationLogger ERROR - Invalid UTF-8 string found in BenchmarkResult
[08-21-2025 15:36:44.658] SystemController ERROR - Failed to append benchmark result

Maybe it happened because of the Chinese character, but I cannot change the input “Microsoft Windows 11 专业版” to English since I cannot find the entry.

[swasson488]

We believe this issue is resolved in MLPerf Client 1.5. Please test this release and let us know if it resolved the issue for you. Thanks!