I mentioned in my homelab post that one of my STILL-TO-DOs was self-hosting LLMs. Unfortunately, we run into immediate hardware constraints on that one - I don’t have a single GPU lying around! Sad.
I was ready to put this dream to rest, but a video by YouTuber NetworkChuck was super interesting and inspiring. Basically, the idea is:
In other words, I’m currently using my NAS as a pass-through for any/all LLM work. I’m using OpenWebUI (interface), Ollama (local LLMs), and LiteLLM (proxy/API connector). I’m connecting to Anthropic and OpenAI 🤮 using their APIs, and I have - for now - one local LLM installed on the NAS (Mistral 7B). In some future world, I’ll spend the big bucks to get a GPU machine - maybe… dare I dream… a bedazzled Framework desktop??
Anyway. Why do this now? Pre-GPU?
The basic idea I’ve been having is, I want my cyber<>meatspace interactions to be represented like this:
- My meatspace self (and my friends, family, strangers, employers, etc)
- …Interact with some UI…
- …To see my data/digital self… which I still own and have control over
If a UI breaks, changes, enshittifies, etc, I can just switch it out. This speaks to Cory Doctorow’s interopability dreams. And yes, it’s all about tearing down the walled gardens’, uh, walls.
I spent most of web 2.0 (2005-2015?) as a digital serf - tilling the big platforms’ fields (Facebook, Twitter, etc) with data that was precious to me. My memories! Pics of college and grad school! Old friends! Parties! Trips! Extracting that data out of the platforms has always been onerous, but I’ve been doing it for a few years now. De-Googling, as you can imagine, has been taking forever. But I have de-gmailed! And that got me thinking. If I move from ProtonMail to FastMail or wherever, I’ll have to keep telling my contacts that my e-mail address changed. What if I can keep that fixed - and just swap out the back-end? HMMMM.
So that’s the idea with the self-hosted UI for the cloud LLMs. I’m retaining a streaming, constant copy of my data locally, on my NAS. (Otherwise I’m just tilling OpenAI and Anthropic’s fields, and have to manually export my data whenever I remember.) This way, yes, they still get my data (“Khaaaaaaaan!” as Kirk would say), but at least I have it too. And I can delete it from their servers whenever I want (presumably…). And once I’m ready to buy a GPU and stand up a local, semi-powered, truly private AI, I can cut the chord completely.
Fun OpenWebUI features#
- Chatting
- Some coding
- Sometimes some more involved projects (OCR + translation + summarization)
I don’t really enjoy AI slop art, and haven’t yet gotten into text-to-speech (though I have been thinking about Whisper). I have zero interest in nightmarish video gen.
For chatting, I had a glorious prompt on the Anthropic UI for a “techbro neckbeard”. He was the “persona” I spoke with the most. Basically, I told him to be a typical Redditor, all lowercase, very snide, judgmental, etc. The “WELL ACKshually” guy. I ported that persona over to OpenWebUI, added a profile pic (you can add profile pics for your custom models!!), inserted the same Anthropic Claude-4-sonnet brain, and, voila, now I have my asshole AI assistant back!! Man, this guy makes me laugh and laugh. Also, super interestingly, while both the Claude UI and OpenWebUI neckbeard personas always assumed I was a dude (sigh…), the OpenWebUI techbro was also way more overtly sexist and rude. Wow!! Are there more guardrails on the Claude UI chat vs. the Claude API? I wonder… Anyway, I had to adjust his prompt to be a feminist techbro, jeez.
I have a few more AI personas I enjoy - man, I’m really living the Dan Simmons’s Hyperion dream here. I have a solarpunk lady and a fitness guru lady. I used to have a Thomas Bayes econometrics/stats friend - he would speak in 19th century English minister talk, very amusing. I have had dark sci-fi thoughts about making personas for my loved ones that have died 😱. But no! Must not literally recreate that one Black Mirror episode. 😬