Mixtral 8x7B - An offline Open AI better than GPT 3.5 in Gladys?

Development
1

An Open AI in Gladys? If there’s a crazy dev in the room!!! :crazy_face:

Worth checking it out, it’s apparently mind-blowing!!!

2

I’m not a big expert in AI and how it works, but it’s clear that ChatGPT is becoming one option among others.

I’m eyeing this: GitHub - mudler/LocalAI: 🤖 The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed, P2P and decentralized inference

1 Like
3

Eeeehhh super interesting @guim31, installable in a container and full of features that would be interesting to interface with Gladys…@pierre-gilles if you’re listening to us! :crazy_face:

image
image656×569 26.2 KB

1 Like
4

Very interesting, but it probably wouldn’t be very practical to use or quick to run on a Raspberry…

5

There are very lightweight versions of 1 or 2 GB that run with less than 4 GB of RAM and that will probably be enough.
However, on a mini PC it might not run anymore.

This might give you some ideas:

1 Like
6

You still need a good GPU :grin:

7

Exactly, and besides, I think you need an NVIDIA GPU because it uses CUDA cores, for better optimization.

8

Given the performance it demands, I think it’s still a bit overkill, but it’s worth keeping a close eye on, because I think it could actually become interesting before long.