[Feature request] Allow preloading models

[antolucib]

I'm currently using llama-swap as a proxy between sglang and other services using its api, such as open webui or vscode + continue, in order to allow unloading the model from VRAM after a while that the service isn't used. This works well (I had to tinker a bit to make sglang work with unloading but it worked out in the end), but sglang takes around 30s to load the model in order to perform inference, and I'd like to minimize this latency for my users.
Let's take open webui as an example: a way to accomplish this could be to make llama-swap preload a model as soon as a user first visits the webpage (if no model is loaded already?), so that the model loads while the user writes its prompt, and ideally has finished loading by the time the user sends its request.
Is there a way to do this currently? For example by running a preload command when the proxy receives a GET /v1/models or GET /get_model_info or GET /health request (or whenever the proxy detects any request or connection)?

[mostlygeek]

The easiest manual way to load a model on demand is calling /upstream/:model. This will swap to a model the same way as calling the OpenAI api endpoints.

[henfiber]

Another use case where this may be useful is when one swaps between 2 models that do not fit within the VRAM at the same time, like your aider "architect" example in the wiki.

In this case, it would be useful if there was some way to load the first model (the architect) after the second model (the coder) has finished. This avoids the waiting time for the first model to be loaded again.

[mostlygeek]

Been thinking about this a little bit on and off. I think a good place to put it is part of the Groups feature.

Here's an example configuration

groups: 
    exampleGroup:
        preload: "member1"
        members: 
            - model1
            - model2
            - model3

Thoughts about this design:

• swap: true must be set to true
• when to unload model2 and model3 so model1 is automatically preloaded?
  • should it be done by requests?
  • should it be done by ttl that already exists on a model's configuration?

[HenkieTenkie62]

Maybe it should be a preload and a "reload"?
On startup it will load the preload model.
If one of the models unloads after ttl it will default to the reload.

For my use cases i wouldn't use a ttl but i would like a model to be loaded in case there is no model loaded in the group.
Could be when the inference server crashed (OOM or otherwise).

I would love this feature. i've now glued this function in with a watcher script.

[mostlygeek]

I think I need to collect a few more use cases of how people may use a preload feature.

I’m particularly unsure about automatically swapping back to the preloaded model. What conditions would trigger the swap-back? 🤔

[henfiber]

One alternative to finding one-size-fits-all strategy would be to introduce hooks and actions, and allow users to do whatever they want with loading/unloading models.

So, instead of baking a single pre-load policy into the groups feature, we could expose a small, declarative event system (maybe also relevant to your recently introduced internal event system):

Examples of hooks:

Hook (when)	Fires	Typical uses
on_startup	once, after llama-swap boots	Pre-warm a default model
on_list_models	once, when /v1/models is first called	Pre-warm a default model
before_load	just before a model's cmd is executed	Reserve VRAM, notify a dashboard
after_load	when the health-check passes	Mark model "ready", trigger a warm-up prompt
before_unload	right before SIGTERM / cmdStop	Flush logs, snapshot KV-cache
after_unload	after the process exits	Auto-reload a standby model, free resources
after_expire	when a model is evicted by ttl	Same as above but only for TTL expiry
on_request	first request per model after idle_for seconds	Lazy-load or metrics
on_health_fail	a health-check returns non-200	Restart container, alert
on_error	llama-swap itself panics	Send Slack / e-mail

The hook handler receives a JSON payload:
{ "model":"coder", "group":"coding", "reason":"ttl" … }

Built-in actions (do)

Action	Parameters	Notes
load_model	id: model-id	Equivalent to GET /upstream/:model
unload_model	id: model-id	Graceful cmdStop + kill
reload_model	id: model-id	Convenience = unload + load
shell	cmd: string	Run arbitrary command (for maximal freedom)
http	url: string, method: GET/POST..., headers: array, body: string	Ping Grafana, webhooks, etc.
log	level, msg	Custom log entries

The action list is intentionally small since most advanced workflows can be achieved with shell or http while the first three should cover 90% of most needs. Typically the most useful http params are: url, headers, method, body.

Examples

Example 1 - Architect / Coder hot-swap

# pre-loads “architect” at startup and reloads it
# whenever the “coder” model is evicted.
hooks:
  on_startup:
    - load_model: architect

  after_unload:
    - when:
        model: coder        # fires only for this model
      do:
        - load_model: architect

Flow: user requests coder -> llama-swap swaps in coder -> coder hits its ttl -> after_unload triggers -> architect is back in RAM before the next request.

Example 2 - VRAM-tight gaming rig

groups:
  my_fav:
    swap: true
    members:
      - qwen3-0.6b
      - gemma3-27b

hooks:
  before_load:
    - when:
        group: my_fav
      do:
        - shell: nvidia-smi --persistence-mode=1  # ensure clean VRAM

  after_load:
    - shell: "curl -s -XPOST http://grafana/api/warm?model=${_CURRENT_MODEL}"

  on_request:
    - when:
        idle_for: 15          # seconds with no traffic
      do:
        - unload_model: ${_CURRENT_MODEL}
        - load_model: qwen3-0.6b   # keep the tiny model resident

Example 3 - Blue-Green production roll-out with automatic warm-up

Use case: Production roll-out - bring a new build of a model online, prime it with a dummy prompt, and then retire the old build once the new one is healthy.

models:
  gpt-green:
    cmd: "/opt/llama-server --model /models/mymodel-v1.1-green.gguf --port ${PORT}"
    proxy: "http://127.0.0.1:${PORT}"
    ttl: 0         # never auto-unload

  gpt-blue:
    cmd: "/opt/llama-server --model /models/mymodel-v1.0-blue.gguf --port ${PORT}"
    proxy: "http://127.0.0.1:${PORT}"
    ttl: 0

groups:
  prod:
    swap: true
    members: [gpt-green, gpt-blue]

hooks:
  #  on_startup, load last-known-good version (blue)
  on_startup:
    - load_model: gpt-blue

  # manual curl to /upstream/gpt-green triggers before_load / after_load for green
  after_load:
    - when:
        model: gpt-green
      do:
        # Warm-up request so users don’t pay KV-cache build time
        - http:
            url: "http://127.0.0.1:${PORT}/v1/chat/completions"
            method: POST
            body: |
              { "model":"gpt-green",
                "messages":[{"role":"system","content":"warm-up"}],
                "max_tokens":1 }

        # Retire the previous blue version once green is healthy
        - unload_model: gpt-blue

So, you ship a new model (gpt-green) and hit curl -XPOST http://llama-swap/upstream/gpt-green. When the health-check passes, after_load primes the cache and unloads the old blue build, giving you a "zero-downtime" deployment path.

Example 4 - High-availability fail-over with Slack alert

if a model’s /health endpoint fails, swap in a backup copy and page an on-call engineer.

models:
  phi-primary:
    cmd: "llama-server --model /models/phi-3-primary.gguf --port ${PORT}"
    proxy: "http://127.0.0.1:${PORT}"
    ttl: 0

  phi-backup:
    cmd: "llama-server --model /models/phi-3-backup.gguf --port ${PORT}"
    proxy: "http://127.0.0.1:${PORT}"
    ttl: 0

hooks:
  on_startup:
    - load_model: phi-primary

  on_health_fail:
    - when:
        model: phi-primary
      do:
        - unload_model: phi-primary
        - load_model:   phi-backup
        - http:
            url: "https://hooks.slack.com/services/T0000/B0000/XXXXXXXX"
            method: POST
            body: |
              { "text": "*phi-primary* failed health-check - swapped to *phi-backup*." }

The first time phi-primary returns a non-200 status on /health, llama-swap automatically unloads the problematic instance, spins up phi-backup, and sends a Slack webhook so someone can investigate.

---

Now, the structure of hooks definition with when and do (or directly calling an action) are just suggestions for thought. Thinking also what conditions should be available in the when statement (e.g. model, group, idle_for). Pre-defined runtime macros (variables) such as ${_CURRENT_MODEL} as in the examples above may be useful as well.

Some benefits of this approach:

• Users chain several simple actions instead of waiting for a one-size-fits-all preload setting.
• No breaking changes: If the hooks: section is omitted, llama-swap should behave exactly as today.
• Every hook/action invocation could be logged at debug level, keeping troubleshooting simple.
• New events (e.g. on_profile_switch) or actions can be added without touching existing configs.

If you think it is a good idea, then maybe the next steps could be:

• Agree on hook names + JSON payload structure.
• Implement some hooks such as on_startup, after_load, after_unload and the load_model / unload_model actions
• Do not publish the feature in the public docs until the yaml config and hook names are stable. Early adopters can replicate the watcher scripts many of us run today and report gaps.

[mostlygeek]

@henfiber I like the idea of hooks. Maybe instead of “hooks” name it “events” to stay consistent with the new internal event bus system.

I’m quite against having complex logic in YAML. It’s a mark up language after all. However, I think we could get pretty far starting with just the shell action. Data can be passed as env variables and macro substitution in the command’s args. This would decouple llama-swap from custom user specified logic. Also it’ll make it easier to troubleshoot.

[henfiber]

@mostlygeek "events" is ok too. I agree that you could start simple as you suggested with a minimal set of actions and event/hooks and introduce new ones only if a common scenario is not covered.

Maybe I overdid it a bit, because I am used to the home-assistant automations yaml syntax, which also defines logic declaratively (triggers, conditions, actions triplets with a lot of predefined options). Structured/constrained logic has a benefit compared to DSL languages: it can be parsed and built using a UI (some freedom is re-gained by using jinja2 templates in many cases). So, personally I am used to defining logic in YAML.

[github-actions[bot]]

This issue is stale because it has been open for 2 weeks with no activity.

[benfiola]

this is a cool idea! if this is the draft UX we want to implement, i'm more than happy to take a stab at it.

personally, i don't really need this feature - but llama-swap's been working so well for me that i'd love to help contribute towards making it better. (thank you for making such a solid piece of software!)

[mostlygeek]

Hi @benfiola,

I don't think there is a design we've settled on yet. A few key questions still need good answers before proceeding:

• who's maintaining this feature going forward? I'm not ready to make that commitment right now.
• are hooks synchronous or async? Perhaps both hooks (synchronous) and events (asynchronous)?
• where to keep logic? I'm against YAML but willing to discuss the right balance.

The discussion sort of went stale for this until you messaged so that may be a sign for this feature too. :)

[samteezy]

I'm just seeing this thread... my use case is to preload simple embedding and small "utility" models that I know other apps like Open WebUI will use.

I can't always guarantee that I can configure or modify apps to call an endpoint on launch, and I often reload llama-swap due to reconfiguration, so allowing llama-swap to load them and keep them "warm" would be beneficial for me.

[iansmathew]

@mostlygeek Bringing up this issue back to life since I would also like a preload feature. Rather than complicating the current tool with hooks or events, to address only the preload feature, wouldn't a simple solution to be something like

groups: 
    exampleGroup:
        # loops through preload members and calls /upstream/:model
        # requires swap to be false if preload.length > 1
        preload: 
            - model1
            - model2
        members: 
            - model1
            - model2
            - model3

[mostlygeek]

I like the idea of the top level hooks and I think we can start with this:

hooks: 
  on_startup:
    preload: ["model1", "model2", "model3"]

This does address the desire for this feature and my constraints above:

1. it'd be a tiny patch
2. punts the async/sync and complex design discussion to the future
3. no logic in yaml

It'll give some structure to extend the hooks in the future:

hooks: 
  on_startup:
    run_scripts: 
      - /path/to/notify-ready.sh
      - /path/to/another-script.sh
  on_shutdown:
    run_scripts: 
      - /path/to/script1.sh
      - /path/to/script2.sh
    new_action:
      param1: "value"
      param2: "value"

[mostlygeek]

I created PR #235, which is just a proof of concept. It is a really simple implementation but wanted to give anyone here a chance to test it out.

Here is the most basic config (same as above):

hooks:
  on_startup:
    preload:
      - "qwen"
      - "llama"

The current design basically automates requesting the models via the API. The benefit of this is that all current swapping behaviour stays the same. For example, I have this group:

groups:
  coding:
    swap: false
    members:
      - "qwen"
      - "llama"

Both the qwen and llama models will be loaded because of swap: false. If a request comes in for model3 it will unload both qwen and llama.

I'll be AFK for a few days but if anyone wants to test it out I would appreciate that. You will have to build it yourself from source. That should be fairly straightforward if you have go installed. Just use make all, and look in the new build/ sub-directory.

[antropiq]

I have build your PR on linux debian bookworm and it covers my use case.

Here is my configuration file:

models:
  "devstral":
    cmd: |
      /home/vgr/dev/www/llama.cpp.rocm/build/bin/llama-server
      --alias devstral
      --port ${PORT}
      -c 32768
      --device "ROCm0,ROCm1"
      --tensor-split "1,1"
      --offline
      --jinja
      -sm row
      --flash-attn
      -ctk q8_0
      -ctv q8_0
      --top-p 0.95
      --top-k 64
      --temp 0.15
      --repeat-penalty 1.05
      --min-p 0.01
      -ngl 41
      -m /home/vgr/llm/models/unsloth/devstral/Devstral-Small-2507-Q4_K_M.gguf
      --mmproj /home/vgr/llm/models/unsloth/devstral/mmproj-F16.gguf
  "llama3.2-1b":
    cmd: |
      /home/vgr/dev/www/llama.cpp.rocm/build/bin/llama-server
      --alias llama3.2-1b
      --port ${PORT}
      -c 32768
      --offline
      --jinja
      --device "ROCm3"
      -fa
      --top-p 0.95
      --top-k 40
      --temp 0.5
      --repeat-penalty 1.05
      --min-p 0.05
      -ngl 17
      -m /home/vgr/llm/models/bartowski/llama3.2/Llama-3.2-1B-Instruct-Q6_K.gguf
  "qwen3-coder-30b-a3b":
    cmd: |
      /home/vgr/dev/www/llama.cpp.rocm/build/bin/llama-server
      --alias qwen-coder
      --port ${PORT}
      -c 32768
      --offline
      --device "ROCm2,ROCm3"
      --tensor-split "1,1"
      --jinja
      -sm row
      -fa
      -ctk q8_0
      -ctv q8_0
      --top-p 0.8
      --top-k 20
      --temp 0.7
      --repeat-penalty 1.05
      --min-p 0.01
      -ngl 49
      -m /home/vgr/llm/models/unsloth/qwen3/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf

groups:
  "group_aider":
    swap: false
    exclusive: true
    members:
      - "devstral"
      - "llama3.2-1b"
      - "qwen3-coder-30b-a3b"
hooks:
  on_startup:
    preload:
      - "devstral"
      - "llama3.2-1b"
      - "qwen3-coder-30b-a3b"

The three models get loaded when i restart the systemd service file.
As you can see the configuration file repeats the 3 models name and i was wondering if it wouldn't be more accurate to use group name ? like this:

hooks:
  on_startup:
    preload models: []
    preload group: "aider_model

Anyway this PR covers my use case witch is:

When the systemd service starts, launch llama-swap and load the models group.

++

[iansmathew]

@antropiq I would argue to keep it as it is since most users would want the fine grained ability to preload models as opposed to groups.

Glad to see it’s working though. Thanks for testing it out

[mostlygeek]

Keeping the preloading at the model level instead of the group level keeps things simpler for the implementation. It may be a bit more work for users but it’s only an array of strings so not a bad tradeoff for less code and bugs.

[antropiq]

You are right it's more flexible and covers my case :)

[mostlygeek]

preloading of models has been officially released in v151 now.