Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
You must log in or register to comment.
I was hosting LLM with LMStudio occasionally but can’t access it anymore due to some fuckery with CORS and http vs https in browsers.
I googled it, and it seems like you can just enable cors.
Yes you can enable cors in LMStudio. But since few months it’s blocked by all major web browsers if you aren’t using HTTPS.
Which I don’t. I had LMStudio server open to local network so I can use it on my phone or laptop via third party website.
An aside for anyone reading this:
https://sleepingrobots.com/dreams/stop-using-ollama/
And that barely scratches the surface. Please.
Use anything but Ollama. Even APIs.
Thanks for this link. Because of this article, I had claude stand up a llama.cpp container next to my already running ollama container. It ran side by side tests with the same model and parameters, and the results blew ollama out of the water. I’m in the process of moving hermes and openwebgui over to the llama.cpp instance to see how it goes day to day.
If you’re using docker anyway, and “fast” pure GPU models, you might try a vllm container while you’re at it.
It should be much faster than even llama.cpp, albeit at the cost of context length, and it supports some exotic 4-bit quantization like SPQA.
Same with TabbyAPI. It’s quantization is SOTA, though it does not support CPU offloading, and it’s speed is somewhere between vllm and llama.cpp.
