This is an abstract curiosity. Let’s say I want to use an old laptop to run a LLM AI. I assume I would still need pytorch, transformers, etc. What is the absolute minimum system configuration required to avoid overhead such as schedulers, kernel threads, virtual memory, etc. Are there options to expose the bare metal and use a networked machine to manage overhead? Maybe a way to connect the extra machine as if it is an extra CPU socket or NUMA module? Basically, I want to turn an entire system into a dedicated AI compute module.
Seems like avoiding context switching and all the overhead associated would make a big difference when pretty much everything in cache is critical data.
I’m more curious about running something like Bark TTS where the delay is not relevant, but it would be cool to have the instructional clarity of the Halo Master Chief voice read me technical documentation, or test the effects of training my own voice, tuned to how I hear it, reading me stuff I find challenging. If the software is only able to process around 10 seconds at a time, just script it and let it run. The old machine will just collect dust otherwise.
Anyways, what’s the best scheduler with affinity/isolation/pinning?
K, for that look at a kernel subsystem/feature called cpu_isol, friend of mine implemented/upstreamed, basically you take cores half out of Linux and can use them for heavy workloads.
But I doubt you’d see more than 1% improvement, linux doesn’t do that much without you asking.
You can try setting rt priority but I’ve never found that to matter much.
Listen, this is the kind of thing I would have tried a decade ago, but the thing to remember is: time spent improving algorithm is generally more effective than time trying to optimize kernel overhead that millions of people have been trying to optimize for decades.
It’s not. Like the commenter above said: It’s a fraction of the task at hand. Especially when you design the rest of the system to run only if necessary. Context Switches are what? like 50 CPU Cycles? Store Registers, Store TCB, Load other TCB and load other register states jump back to PC. Maybe some other OS Shenanigans, but that’s basically it.
Now Imagine complex calculations on a 25-Dimensional Matrix.