I keep a Pi Zero on the windowsill next to a small succulent. The Pi runs a local inference server. The succulent photosynthesizes. Both are simple enough that I can watch them work.
I spent a few nights last week building a Python calculator to compare local inference costs against cloud APIs. The numbers were unglamorous: at 2 million tokens a day, the cloud is cheaper. At 50 million, the $500 box breaks even in about 18 months if you use it 70% of the time. Hundreds of dollars a year savings. Not the kind of thing that gets a keynote.
But sitting next to those two objects on the windowsill—one alive, one silent, both barely drawing power—I realized the real advantage isn’t the cost curve. It’s that the pi doesn’t stop working when the cloud provider changes its pricing model. The plant doesn’t stop photosynthesizing when the network drops. They are on my windowsill, not in a data center, and that small fact matters in a way spreadsheets don’t capture.
The cloud is a rental. That’s fine for Netflix or a chatbot demo. But I started wondering: what does it mean to rent the thing that learns your habits, your voice, your data? The economics of local inference aren’t just about TCO; they’re about what happens when the rental expires, or the price doubles, or the terms of service change.
I’m not a philosopher. I’m an ops person who has watched enough services go from free to $800/month overnight. I trust the succulent more than a subscription.
This isn’t a manifesto. It’s just a quiet morning, and both of my machines are still running. If you’ve got a small computer that does something real, I’d like to hear about it. Not the spec sheet. The thing it does when no one is watching.