☰
×
Login Register an account
Top Submissions Explore Upgoat Search Random Subverse Random Post Colorize! Site Rules
6

Anyone running LLMs locally?

submitted by Niggly_Puff to AI 4 monthsJan 5, 2024 16:50:24 ago (+6/-0)     (AI)

Recently got into this. I installed Silly Tavern and I am amazed at the results. I had Joe Rogan interview Jesus and it was entertaining and coherent. Having this power and knowledge 100% offline is pretty damn cool.


16 comments block

sort by: hot new top old low

[ - ] purityspiral 1 point 4 monthsJan 6, 2024 00:06:06 ago (+1/-0)*

Yea, try faraday.dev
You can also use many bindings to code the client yourself (full control of prompting etc)

I made mine talk with tts. (send prompts, get text, send text to TTS, get mp3, play it, ez)
Used this one for that: https://localai.io

Local AI (.io) is likely the best I've used so far, most open and stupid simple.
They have an AGI experiment in there too (several projects, wide open, github)

Never use online implementation (web applications, cloud based, phone apps, etc)
Find how they did it and roll your own.

I didn't care for SillyTavern, but I gave it a few tries, didn't get as good results like with faraday.dev (quick easy w/ client).
I did like how SillyTavern could paint an image of the current scene in the conversation.

[ - ] x0x7 1 point 4 monthsJan 5, 2024 20:41:47 ago (+1/-0)

I am. I'll have to try that Silly Tavern. I'm mostly done llama 7b. I have the same graphics card so I'll have to try that 13b.

[ - ] Niggly_Puff [op] 1 point 4 monthsJan 5, 2024 21:53:59 ago (+1/-0)

It just barely fits, 200-300 mb or so left to spare.

[ - ] autotic 1 point 4 monthsJan 5, 2024 17:36:48 ago (+1/-0)

Yes, ollama seems to be a decent way to deploy, use, tune quickly. Most of them are only really "expensive" when it comes to the size of the models. I've not tried to do any of my own training but that's where they are going to be really useful, and expensive, for what I do and what I plan to do.

[ - ] Niggly_Puff [op] 1 point 4 monthsJan 5, 2024 17:59:48 ago (+1/-0)

Care to share any details?

I've not tried to do any of my own training

Me neither. Imagine an LLM trained on the JQ 🤔

[ - ] israeldid9_11 1 point 4 monthsJan 5, 2024 18:28:53 ago (+1/-0)

I just want one trained as a rabbi. Full Talmud knowledge and the ability to quote it. So far everyone I have used quite the Bible, but not the Talmud.

[ - ] Niggly_Puff [op] 1 point 4 monthsJan 5, 2024 18:30:07 ago (+1/-0)

I had chat GPT admit to me the talmud says having sex with 3 year olds is ok. Made a thread about it. Let me see if I can find it.

[ - ] Niggly_Puff [op] 1 point 4 monthsJan 5, 2024 18:31:21 ago (+1/-0)

[ - ] autotic 1 point 4 monthsJan 5, 2024 21:45:49 ago (+1/-0)

ollama is just some CLI tooling which allows you to quickly start, stop, download and change models, locally.

re JQ, I have 100s of GBs of docs stored in Calibre on this and other Goat-y topics. At some point I will train a local Tay. ;)

[ - ] shitface9000 0 points 4 monthsJan 6, 2024 19:09:36 ago (+0/-0)

You can do a RAG app for this

[ - ] chrimony 0 points 4 monthsJan 5, 2024 16:52:17 ago (+0/-0)

Not me, but I'm curious what kind of resources you threw at it.

[ - ] Niggly_Puff [op] 1 point 4 monthsJan 5, 2024 16:53:38 ago (+1/-0)

A 3060 12gb graphics card. It has just enough vram to load a 13b model on it.

[ - ] chrimony 0 points 4 monthsJan 5, 2024 17:03:53 ago (+0/-0)

Thanks for the info. How much hard drive space did you need? And how long did it take to train/load?

[ - ] Niggly_Puff [op] 1 point 4 monthsJan 5, 2024 17:11:38 ago (+1/-0)

I use oobabooga and silly tavern. With 1 13b model the folder is about 17 gigs. It's portable so you can set it up, and then put it on a usb stick and transfer between machines without having to re-set it up.

[ - ] chrimony 0 points 4 monthsJan 5, 2024 17:13:44 ago (+0/-0)

Thanks. That's surprisingly moderate. Does it have access to live Internet?

[ - ] Niggly_Puff [op] 1 point 4 monthsJan 5, 2024 17:24:04 ago (+1/-0)

No it doesn't. I am curious now if that is even possible, I dont think so. I know openAI's gpt4 can do it, but that's more of a system of models instead of just a LLM.