Skip to content

vllm.model_executor.models.lfm3_audio

Inference-only LFM3-Audio model: speech in, text out.

The speech output head (codebook embedding, depth decoder) is not served and its weights are skipped on load.

Classes:

Functions:

  • split_audio_features –

    Keeps each clip's own frames: the features are padded to the longest clip

Lfm3AudioDownsampler

Bases: Module

Three stride-2 convolutions over the (mel, time) plane: 128 mel bins and 8 frames become one vector.

Source code in vllm/model_executor/models/lfm3_audio.py 
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
class Lfm3AudioDownsampler(nn.Module):
    """Three stride-2 convolutions over the (mel, time) plane: 128 mel bins and
    8 frames become one vector."""

    def __init__(self, config: Lfm3AudioEncoderConfig) -> None:
        super().__init__()
        channels = config.conv_channels
        conv = {"kernel_size": 3, "stride": 2, "padding": 1}
        self.conv_in = nn.Conv2d(1, channels, **conv)
        self.conv_mid = nn.Conv2d(channels, channels, groups=channels, **conv)
        self.lin_mid = nn.Conv2d(channels, channels, kernel_size=1)
        self.conv_end = nn.Conv2d(channels, channels, groups=channels, **conv)
        self.lin_end = nn.Conv2d(channels, channels, kernel_size=1)
        self.lin_out = nn.Conv1d(
            config.num_mel_bins * channels // 8,
            config.text_config.hidden_size,
            kernel_size=1,
        )
        self.act = nn.SiLU()

    def forward(self, input_features: torch.Tensor) -> torch.Tensor:
        hidden_states = self.act(self.conv_in(input_features.unsqueeze(1)))
        hidden_states = self.act(self.lin_mid(self.conv_mid(hidden_states)))
        hidden_states = self.act(self.lin_end(self.conv_end(hidden_states)))
        return self.lin_out(hidden_states.flatten(1, 2)).transpose(1, 2)

Lfm3AudioEmbeddingInputs

Bases: TensorSchema

Dimensions: - na: Number of audios - nt: Number of audio tokens - hs: Hidden size of the language model

Source code in vllm/model_executor/models/lfm3_audio.py 
80
81
82
83
84
85
86
87
88
89
90
91
class Lfm3AudioEmbeddingInputs(TensorSchema):
    """Dimensions:
    - na: Number of audios
    - nt: Number of audio tokens
    - hs: Hidden size of the language model
    """

    type: Literal["audio_embeds"]
    audio_embeds: Annotated[
        torch.Tensor | list[torch.Tensor],
        TensorShape("na", "nt", "hs", dynamic_dims={"nt"}),
    ]

Lfm3AudioFeatureInputs

Bases: TensorSchema

Dimensions: - na: Number of audios - nmb: Number of mel bins - nf: Number of spectrogram frames

Source code in vllm/model_executor/models/lfm3_audio.py 
66
67
68
69
70
71
72
73
74
75
76
77
class Lfm3AudioFeatureInputs(TensorSchema):
    """Dimensions:
    - na: Number of audios
    - nmb: Number of mel bins
    - nf: Number of spectrogram frames
    """

    type: Literal["audio_features"]
    input_features: Annotated[
        torch.Tensor | list[torch.Tensor],
        TensorShape("na", "nmb", "nf", dynamic_dims={"nf"}),
    ]

split_audio_features(processed_data)

Keeps each clip's own frames: the features are padded to the longest clip of the call, so cached and freshly processed items would differ.

Source code in vllm/model_executor/models/lfm3_audio.py 
 97
 98
 99
100
101
102
103
104
105
106
107
def split_audio_features(processed_data: BatchFeature) -> BatchFeature:
    """Keeps each clip's own frames: the features are padded to the longest clip
    of the call, so cached and freshly processed items would differ."""
    if (masks := processed_data.pop("feature_attention_mask", None)) is not None:
        processed_data["input_features"] = [
            features[..., :length]
            for features, length in zip(
                processed_data["input_features"], masks.sum(-1).tolist()
            )
        ]
    return processed_data