While this is certainly a cool concept, local voice assistants like this are currently a novelty. Cool to play around with, though!
You can expect around 5 seconds processing time to start generating the response to a basic question on a very basic model like Llama 3 8B.
For context, using Moondream2 (as recommended) on a RasPi 5, it takes around 50 seconds to process an image taken by the Camera and start generating a description.
The main reason for this is that I wanted to create a voice assistant that is completely offline and doesn’t require any internet connection. This is because I wanted to ensure that the user’s privacy is protected and that the user’s data is not being sent to any third party servers.
Weird reasoning, why constrain the assistant to pi’s power and memory when you can use it for interaction and utilize another, more powerful server at home
While the system is designed to be entirely local, it is also possible to easily connect it to some external APIs or services if you want to enhance the conversation
Add comment