Skip to content

feat: add a rust API frontend#1667

Open
AlpinDale wants to merge 3 commits into
mainfrom
feat/rust-frontend
Open

feat: add a rust API frontend#1667
AlpinDale wants to merge 3 commits into
mainfrom
feat/rust-frontend

Conversation

@AlpinDale
Copy link
Copy Markdown
Collaborator

@AlpinDale AlpinDale commented May 2, 2026

Based on https://github.com/Inferact/vllm-frontend-rs, with changes to accommodate Aphrodite's API and func signatures + request-level metrics.

Rust API: E2E time: 1.73s, TTFT: 0.04s, Prefill: 1925 tokens (48891.2 tokens/s), Decode: 512 tokens (303.1 tokens/s)
Python API: E2E time: 1.69s, TTFT: 0.16s, Prefill: 1925 tokens (11845.3 tokens/s), Decode: 448 tokens (294.5 tokens/s)

Testing

Install aphrodite as normal, then run:

./build_rust.sh

APHRODITE_USE_RUST_FRONTEND=1 aphrodite run <args>

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a Rust-based frontend for the Aphrodite engine, incorporating a new Rust workspace with specialized tool parsers for DeepSeek and Qwen models, gRPC service definitions, and Jinja-based chat template rendering. The Python infrastructure is updated with environment variables and process management logic to launch and monitor the Rust binary as a subprocess. Review feedback identified several critical issues, including missing imports for the dataclasses, weakref, and multiprocessing.connection modules in utility files, as well as a formatting error in a logger warning call where a placeholder was missing its corresponding argument.

from argparse import Namespace
from http import HTTPStatus
from logging import Logger
from typing import Any
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The dataclasses module is used in the newly added _jsonify_arg_value function (lines 226-227) but is not imported in this file. This will cause a NameError at runtime.

Suggested change
from typing import Any
import dataclasses
from typing import Any

Comment thread aphrodite/v1/utils.py
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import argparse
import contextlib
import json
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Several required modules and types are missing from the imports in this file, which will lead to runtime errors:

  • weakref is used by RustFrontendProcessManager (line 283).
  • connection (from multiprocessing) is used by _SubprocessWrapper (lines 298, 303).
  • Any is used as a type hint in multiple places (lines 242, 401).
Suggested change
import json
import json
import weakref
from multiprocessing import connection
from typing import Any

args.api_server_count,
)
elif envs.APHRODITE_RUST_FRONTEND_PATH and args.api_server_count > 1:
logger.warning("Ignoring --api-server-count=%d when using rust front-end process")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The logger.warning call contains a %d placeholder but is missing the corresponding argument to format. This will result in a literal %d being logged instead of the actual server count.

Suggested change
logger.warning("Ignoring --api-server-count=%d when using rust front-end process")
logger.warning("Ignoring --api-server-count=%d when using rust front-end process", args.api_server_count)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant