# GLM-4.5-Air

A 106B MoE served from a GGUF checkpoint via llama.cpp on a single A100.

Source: https://v0-1.docs.modelplane.ai/recipes/glm-4.5-air/

<!-- vale write-good.Passive = NO -->
A 106B MoE served from an Unsloth GGUF checkpoint via llama.cpp instead of
vLLM, on a single A100 40 GB. Modelplane treats the engine as any
OpenAI-compatible container, so the only changes from a vLLM deployment are
the image and args: the container is still named `engine` and listens on
`:8000`. vLLM can't load this Unsloth quantization format. llama.cpp can, and
`-hf` pulls the checkpoint straight from Hugging Face at startup, so a
one-time deployment needs no `ModelCache`.

The model is bigger than one A100's VRAM, so `--n-cpu-moe` offloads the MoE
expert tensors to host RAM and the GPU runs the active path and KV cache.
That's how a 106B model fits one A100 instead of a multi-GPU node. Apply the
platform side first, then the ML side. The GKE `InferenceCluster` carries a
GCP project placeholder to edit before applying.

## Validated deployments

{{< validated-deployments >}}

## Platform

{{< manifests "recipes/glm-4.5-air/inference-class.yaml" >}}

{{< manifests path="recipes/glm-4.5-air/inference-cluster.yaml" apply="false" >}}

{{< editCode >}}
```bash
curl -fsSL {{< manifest-url "recipes/glm-4.5-air/inference-cluster.yaml" >}} \
  | sed 's/my-gcp-project/$@<your-gcp-project-id>$@/' \
  | kubectl apply -f -
```
{{< /editCode >}}

## Deployment

{{< manifests "recipes/glm-4.5-air/model-deployment.yaml" >}}

{{< manifests "recipes/glm-4.5-air/model-service.yaml" >}}
<!-- vale write-good.Passive = YES -->