Skip to content

vllm.config.device

Classes:

  • DeviceConfig –

    Configuration for the device to use for vLLM execution.

DeviceConfig

Configuration for the device to use for vLLM execution.

Methods:

Attributes:

  • device (SkipValidation[Device | device | None]) –

    Device type for vLLM execution.

  • device_type (str) –

    Device type from the current platform. This is set in

Source code in vllm/config/device.py 
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
@config(config=ConfigDict(arbitrary_types_allowed=True))
class DeviceConfig:
    """Configuration for the device to use for vLLM execution."""

    device: SkipValidation[Device | torch.device | None] = "auto"
    """Device type for vLLM execution.
    This parameter is deprecated and will be
    removed in a future release.
    It will now be set automatically based
    on the current platform."""
    device_type: str = field(init=False)
    """Device type from the current platform. This is set in
    `__post_init__`."""

    def compile_factors(self) -> CompileFactors:
        """This config does not affect the compiled graph.

        If a field starts affecting it, return
        `get_compile_factors(self, ignored_factors)` with every other field in
        `ignored_factors`.
        """
        # Device/platform info is captured by the torch and vLLM factors.
        return {}

    def __post_init__(self):
        if self.device == "auto":
            # Automated device type detection
            from vllm.platforms import current_platform

            self.device_type = current_platform.device_type
            if not self.device_type:
                raise RuntimeError(
                    "Failed to infer device type, please set "
                    "the environment variable `VLLM_LOGGING_LEVEL=DEBUG` "
                    "to turn on verbose logging to help debug the issue."
                )
        else:
            # Device type is assigned explicitly
            if isinstance(self.device, str):
                self.device_type = self.device
            elif isinstance(self.device, torch.device):
                self.device_type = self.device.type

        # Some platforms require processing inputs on CPU.
        from vllm.platforms import current_platform

        if (
            current_platform.uses_host_device_handling()
            and self.device_type == current_platform.device_type
        ):
            self.device = None
        else:
            # Set device with device type
            self.device = torch.device(self.device_type)

device = 'auto' class-attribute instance-attribute

Device type for vLLM execution. This parameter is deprecated and will be removed in a future release. It will now be set automatically based on the current platform.

device_type = field(init=False) class-attribute instance-attribute

Device type from the current platform. This is set in __post_init__.

compile_factors()

This config does not affect the compiled graph.

If a field starts affecting it, return get_compile_factors(self, ignored_factors) with every other field in ignored_factors.

Source code in vllm/config/device.py 
29
30
31
32
33
34
35
36
37
def compile_factors(self) -> CompileFactors:
    """This config does not affect the compiled graph.

    If a field starts affecting it, return
    `get_compile_factors(self, ignored_factors)` with every other field in
    `ignored_factors`.
    """
    # Device/platform info is captured by the torch and vLLM factors.
    return {}