llm-inference

Past 30d, sorted by best first

08/11	7	kubernetes-sigs/The LeaderWorkerSet API (LWS) v0.10.0 API for deploying and managing groups of pods as a single unit with leader and worker roles for multi-host inference workloads. Go 789☆ 909d old #golang #go #sig-apps #llm-inference
08/10	6	tetherto/QVAC sdk-v0.17.0 Local-first, cross-platform SDK for building peer-to-peer AI apps with local model inference, speech, translation, and RAG. TypeScript 554☆ 237d old #javascript #typescript #ai #llm #llama
08/04	6	ome-projects/OME (Open Model Engine) v1.2.2 Kubernetes operator for enterprise-grade management and serving of large language models with model lifecycle automation, runtime selection, and GPU scheduling. Go 490☆ 449d old #golang #go #k8s #llm-inference #model-as-a-service
08/10	5	thushan/Olla v0.0.29 High-performance LLM proxy and load balancer providing intelligent routing, automatic failover, and unified model discovery across local and remote inference backends. Go 273☆ 445d old #golang #ai #go #ollama #proxy
08/17	3	Avarok-Cybersecurity/Atlas Inference Engine b236 Pure Rust LLM inference engine focused on fast, hardware-optimized local model execution across GPUs and CPUs. JavaScript 650☆ 104d old #javascript #cuda #llm-inference #gb10 #mamba