Auto discover models and download them ?

[Aymendje]

Would it be possible to have llama-swap automatically discover models in .cache/huggingface/hub ?
And / Or download models automatically like open-webui does ?

[mostlygeek]

By auto-discovery, do you mean auto-config of models found in .cache/huggingface/hub?

[Aymendje]

Yes
I often need to compare different models and run different tests, and being able to just put the name of the model and have it auto download / auto configure would be great

[mostlygeek]

It would be a great feature! However, there’s so much variation in hardware and configurations that it’ll be a support nightmare for me.

Since the config just a yaml file, I would write a simple script to generate a new config when models are added. llama-swap has a —watch-config flag that reloads when the configuration changes. This would create model configs that I know would run on my specific setup.

I would automate downloading of models in a similar way. If I had a bunch to test I would script it to download what I needed and then generate the custom configs.