Skip to content

vLLM Configuration

This section explains how to configure a self-hosted vLLM model in AskSenseOps.

Prerequisites

Before configuring the model, ensure that a vLLM server is running and that you know its server URL and the model name it serves.

Adding a vLLM Model

  1. Open AskSenseOps.
  2. Navigate to the model configuration section.
  3. Click Add Model.
  4. Enter the required configuration details.
  5. Test the configuration and save the model.

Basic Info

Field Description Required
Display Name Enter a name to identify the model in AskSenseOps. Yes
Provider Select vLLM (Self-Hosted). Yes

Credentials

Field Description Required
API Key Enter the API key used when starting the vLLM server with the --api-key option. Leave this field empty if the server runs without an API key. No

Model Configuration

Field Description Required
Model Name / ID Enter the model name served by vLLM. This must match the server's --served-model-name. Yes
vLLM Server URL Enter the vLLM server URL, including the /v1 suffix. The example shown is http://localhost:8000/v1. Yes

Ensure the URL uses the correct protocol. If the vLLM server uses plain HTTP, include http:// explicitly. A URL without a protocol is treated as HTTPS and may fail when connecting to a plain-HTTP server.

Use Fetch models, when available, to retrieve the models exposed by the server.

Custom Headers

Custom headers are optional and can be used when connecting through a gateway that requires additional headers.

  1. Under Custom Headers (Optional), click + Add.
  2. Enter the required header name and value.
  3. Add additional headers if needed.

Examples shown in the interface include x-openai-token, x-fabrix-client, and x-llm-model-id.

If a custom header carries the authentication token, the API Key field can be left empty.

Testing and Saving the Model

After entering the configuration details:

  1. Review the display name, provider, API key, model identifier, and vLLM Server URL.
  2. Click Test to verify the configuration.
  3. If the test succeeds, click Save Model to save the configuration.

If the test fails, verify that the vLLM server is running, that the URL is reachable from AskSenseOps, and that the model identifier matches the name served by vLLM.