105 Commits

Author	SHA1	Message	Date
Michael Yang	12e923e158	update graph size estimate	2024-04-03 13:34:12 -07:00
Daniel Hiltgen	464d817824	Merge pull request #3464 from dhiltgen/subprocess ... Fix numgpu opt miscomparison	2024-04-02 20:10:17 -07:00
Daniel Hiltgen	6589eb8a8c	Revert options as a ref in the server	2024-04-02 16:44:10 -07:00
Michael Yang	80163ebcb5	fix metal gpu	2024-04-02 16:06:45 -07:00
Daniel Hiltgen	58d95cc9bd	Switch back to subprocessing for llama.cpp ... This should resolve a number of memory leak and stability defects by allowing us to isolate llama.cpp in a separate process and shutdown when idle, and gracefully restart if it has problems. This also serves as a first step to be able to run multiple copies to support multiple models concurrently.	2024-04-01 16:48:18 -07:00