GLM-4.5 Meets vLLM: Built for Intelligent Agents

Introduction

General Language Model (GLM) is a family of foundation models created by Zhipu.ai (now renamed to Z.ai). The GLM team has long-term collaboration with vLLM team, dating back to the early days of vLLM and the popular ChatGLM model series. Recently, the GLM team released the GLM-4.5 and GLM-4.5V model series, which are designed for intelligent agents. They are the top trending models in Huggingface model hub right now.

GLM-4.5 has 355 billion total parameters with 32 billion active parameters, while GLM-4.5-Air adopts a more compact design with 106 billion total parameters and 12 billion active parameters. GLM-4.5 models unify reasoning, coding, and intelligent agent capabilities to meet the complex demands of intelligent agent applications.

Both GLM-4.5 and GLM-4.5-Air are hybrid reasoning models that provide two modes: thinking mode for complex reasoning and tool usage, and non-thinking mode for immediate responses.

As demonstrated in our comprehensive evaluation across 12 industry-standard benchmarks, GLM-4.5 achieves exceptional performance with a score of 63.2, in the 3rd place among all the proprietary and open-source models. Notably, GLM-4.5-Air delivers competitive results at 59.8 while maintaining superior efficiency.

GLM-4.5V is based on GLM-4.5-Air. It continues the technical approach of GLM-4.1V-Thinking, achieving SOTA performance among models of the same scale on 42 public vision-language benchmarks.

To get more information about GLM-4.5 and GLM-4.5V, please refer to GLM-4.5 and GLM-V.

This blog will guide users on how to use vLLM to accelerate inference for the GLM-4.5V and GLM-4.5 model series on NVIDIA Blackwell and Hopper GPUs.

Installation

In the latest vLLM main branch, both the GLM-4.5V and GLM-4.5 model series are supported. You can install the nightly version and manually update transformers to enable model support.

pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
pip install transformers-v4.55.0-GLM-4.5V-preview

Usage

GLM-4.5 and GLM-4.5V both offer FP8 and BF16 precision models. In vLLM, you can use the same command to run inference for either precision.

For the GLM-4.5 model, you can start the service with the following command:

vllm serve zai-org/GLM-4.5-Air \
    --tensor-parallel-size 4 \
    --tool-call-parser glm45 \
    --reasoning-parser glm45 \
    --enable-auto-tool-choice

For the GLM-4.5V model, you can start the service with the following command:

vllm serve zai-org/GLM-4.5V \
     --tensor-parallel-size 4   \
     --tool-call-parser glm45   \
     --reasoning-parser glm45   \
     --enable-auto-tool-choice  \
     --allowed-local-media-path / \
     --media-io-kwargs '{"video": {"num_frames": -1}}'

Important Notes

• The reasoning part of the model output will be wrapped in reasoning_content. content will only contain the final answer. To disable reasoning, add the following parameter: extra_body={"chat_template_kwargs": {"enable_thinking": False}}
• If you’re using 8x H100 GPUs and encounter insufficient memory when running the GLM-4.5 model, you’ll need --cpu-offload-gb 16.
• If you encounter flash_infer issues, use VLLM_ATTENTION_BACKEND=XFORMERS as a temporary replacement. You can also specify TORCH_CUDA_ARCH_LIST='9.0+PTX' to use flash_infer, different GPUs have different TORCH_CUDA_ARCH_LIST values, please check accordingly.
• vLLM v0 is not support our model.

Grounding in GLM-4.5V

GLM-4.5V equips precise grounding capabilities. Given a prompt that requests the location of a specific object, GLM-4.5V is able to reasoning step-by-step and identify the bounding boxes of the target object. The query prompt supports complex descriptions of the target object as well as specified output formats. Example prompts are:

• Help me to locate <expr> in the image and give me its bounding boxes.
• Please pinpoint the bounding box [[x1,y1,x2,y2], …] in the image as per the given description.

Here, <expr> is the description of the target object. The output bounding box is a quadruple \([x_1,y_1,x_2,y_2]\) composed of the coordinates of the top-left and bottom-right corners, where each value is normalized by the image width (for x) or height (for y) and scaled by 1000.

In the response, the special tokens <|begin_of_box|> and <|end_of_box|> are used to mark the image bounding box in the answer. The bracket style may vary ([], [[]], (), <>, etc.), but the meaning is the same: to enclose the coordinates of the box.

Cooperation with vLLM and GLM Team

Before the release of the GLM-4.5 and GLM-4.5V models, the vLLM team worked closely with the GLM team, providing extensive support in addressing issues related to the model launch, ensuring that the vLLM main branch had full support for the open-source GLM-4.5 series before the models were released.

Acknowledgement

We would like to thank many people from the vLLM side who contributed to this effort, including: Kaichao You, Simon Mo, Zifeng Mo, Lucia Fang, Rui Qiao, Jie Li, Ce Gao, Roger Wang, Lu Fang, Wentao Ye, and Zixi Qi.