Skip to content

Implement vLLM-inspired GPU VRAM allocation for LLM inference #993

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
makr-code merged 6 commits into from
Feb 2, 2026
Merged

Implement vLLM-inspired GPU VRAM allocation for LLM inference #993

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
40fb9ef
Initial plan
Copilot Feb 1, 2026
bbf250a
Add core GPU VRAM allocation classes and configuration templates
Copilot Feb 1, 2026
27ac999
Add comprehensive documentation, tests, and benchmarks for GPU VRAM a…
Copilot Feb 1, 2026
ad09a18
Add CMake build integration for GPU VRAM allocation components
Copilot Feb 1, 2026
c78ee3a
Fix Block struct atomic field copy/move semantics
Copilot Feb 1, 2026
28e9d85
Address code review feedback: fix Q4/Q3 quantization, accuracy, and b…
Copilot Feb 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 45 additions & 1 deletion benchmarks/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -997,6 +997,50 @@ if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/bench_rotary_embeddings.cpp")
message(STATUS " Rotary Embeddings: Single rotation, batch, relational, VectorIndex integration")
endif()

# ============================================================================
# GPU VRAM Allocation Benchmarks (NEW - vLLM-inspired)
# ============================================================================

if(THEMIS_ENABLE_LLM AND EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/bench_gpu_vram_allocation.cpp")
message(STATUS "Adding GPU VRAM Allocation benchmarks")

add_executable(bench_gpu_vram_allocation
bench_gpu_vram_allocation.cpp
)

target_link_libraries(bench_gpu_vram_allocation PRIVATE
${BENCHMARK_LIBS}
themis_core
spdlog::spdlog
RocksDB::rocksdb
Threads::Threads
)

target_compile_definitions(bench_gpu_vram_allocation PRIVATE
THEMIS_BENCHMARK_BUILD=1
)

if(CMAKE_BUILD_TYPE STREQUAL "Release")
if(NOT DEFINED BENCHMARK_ARCH_FLAGS)
set(BENCHMARK_ARCH_FLAGS "-march=native")
endif()
target_compile_options(bench_gpu_vram_allocation PRIVATE
-O3
${BENCHMARK_ARCH_FLAGS}
-DNDEBUG
)
endif()

install(TARGETS bench_gpu_vram_allocation
RUNTIME DESTINATION bin/benchmarks
COMPONENT benchmarks
)

message(STATUS " GPU VRAM: Allocation planning, paged KV-cache, multi-GPU, mixed precision")
else()
message(STATUS "GPU VRAM allocation benchmarks skipped (LLM disabled or file missing)")
endif()

message(STATUS "Benchmarks configured successfully")
message(STATUS " - Build LoRA benchmarks with: cmake --build . --target bench_lora_auto_binding")
message(STATUS " - Run LoRA benchmarks with: ./benchmarks/bench_lora_auto_binding")
Expand All @@ -1005,4 +1049,4 @@ message(STATUS " - Build all benchmarks with: cmake --build . --target bench_lo
message(STATUS " - Run all benchmarks with: ./benchmarks/bench_lora_framework")
message(STATUS " - Or use: make run_benchmarks")
message(STATUS " - Performance benchmarks: bench_storage_performance, bench_olap_performance,")
message(STATUS " bench_embedding_cache_performance, bench_llm_inference_performance, bench_rotary_embeddings")
message(STATUS " bench_embedding_cache_performance, bench_llm_inference_performance, bench_rotary_embeddings, bench_gpu_vram_allocation")
Loading