submitted by Conspirologist to Screenshot 3 monthsJan 26, 2025 04:38:47 ago (+20/-2) (files.catbox.moe)
TheYiddler 0 points 3 months ago
You need a quantized model. Quantizing down to four bits massively reduces it's memory footprint and allows it to run on smaller hardware. Any quant below four and it turns to mush.
TheYiddler 0 points 3 months ago
You need a quantized model. Quantizing down to four bits massively reduces it's memory footprint and allows it to run on smaller hardware. Any quant below four and it turns to mush.