Deploying your model#

Applies to Linux and Windows


5 min read time

ROCm enables inference and deployment for various classes of models including CNN, RNN, LSTM, MLP, and transformers. This section focuses on deploying transformers-based LLM models.

ROCm supports vLLM and Hugging Face TGI as major LLM-serving frameworks.

Serving using vLLM#

vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM officially supports ROCm versions 5.7 and 6.0. AMD is actively working with the vLLM team to improve performance and support later ROCm versions.

See the GitHub repository and official vLLM documentation for more information.

For guidance on using vLLM with ROCm, refer to Installation with ROCm.

vLLM installation#

vLLM supports two ROCm-capable installation methods. Refer to the official documentation use the following links.

vLLM walkthrough#

Refer to this developer blog for guidance on serving with vLLM Inferencing and serving with vLLM on AMD GPUs — ROCm Blogs

Serving using Hugging Face TGI#

The Hugging Face Text Generation Inference (TGI) library is optimized for serving LLMs with low latency. Refer to the Quick tour of TGI for more details.

TGI installation#

The easiest way to use Hugging Face TGI with ROCm on AMD Instinct accelerators is to use the official Docker image at huggingface/text-generation-inference.

TGI walkthrough#

  1. Set up the LLM server.

    Deploy the Llama2 7B model with TGI using the official Docker image.

    docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --shm-size 1g -p 8080:80 -v $volume:/data --name tgi_amd --model-id $model
  2. Set up the client.

    1. Open another shell session and run the following command to access the server with the client URL.

    curl \\
    -X POST \\
    -d '{"inputs":"What is Deep
    Learning?","parameters":{"max_new_tokens":20}}' \\
    -H 'Content-Type: application/json'
    1. Access the server with request endpoints.

    pip install request
    PYTHONPATH=/usr/lib/python3/dist-packages python
    ```` should look like:
    .. code-block:: python
       import requests
       headers = {
         "Content-Type": "application/json",
       data = {
          'inputs': 'What is Deep Learning?',
          'parameters': { 'max_new_tokens': 20 },
       response ='', headers=headers, json=data)

vLLM and Hugging Face TGI are robust solutions for anyone looking to deploy LLMs for applications that demand high performance, low latency, and scalability.

Visit the topics in Using ROCm for AI to learn about other ROCm-aware solutions for AI development.