PS: This mode is not super stable and needs to be worked on

PPS: This project will be discontinued for some time since I'll be working on my thesis until 2024. However there are already so many things that can be improved so I'll be back!

What you'll need:

• An OpenAI account and API key; (check FAQs below for the alternatives)
• PicoVoice account and a free AccessKey; (optional)
• ElevenLabs account and free Api Key (optional);
• langChain API keys for web surfing (news, weather, serpapi, google-serp, google-search... they are all free)
• ffmpeg ;
• Python virtual environment (Python>=3.9 and <3.10);
• Some credit to spend on ChatGPT (you can get three months of free usage by signing up to OpenAI) (suggested);
• CUDA version >= 11.2;
• An IBM Cloud account to exploit their cloud-based text-to-speech models (tutorial)(optional);
• A (reasonably) fast internet connection (most of the code relies on API so a slower connection might result in a longer time to respond);
• mic and speaker;
• CUDA capable graphic engine (my Torch Version: 2.0 and CUDA v11.7 pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117);
• Patience :sweat_smile:

you can rely on the new setup.bat that will do most of the things for you.

GitHub overview

MAIN script you should run: openai_api_chatbot.py if you want to use the latest version of the OpenAI API Inside the demos folder you'll find some guidance for the packages used in the project, if you have errors you might check these files first to target the problem. Mostly is stored in the Assistant folder: get_audio.py stores all the functions to handle mic interactions, tools.py implements some basic aspects of the Virtual Assistant, voice.py describes a (very) rough Voice class. Agents.py handle the LangChain part of the system (here you can add or remove tools from the toolkits of the agents) The remaining scripts are supplementary to the voice generation and should not be edited.

INSTALLATION TUTORIAL

Automatic installation

You can run setup.bat if you are running on Windows/Linux. The script will perform every step of the manual installation in sequence. Refer to those in case the procedure should fail. The automatic installation will also run the Vicuna installation (Vicuna Installation Guide)

Manual Installation

Step 1: installation, accounts, APIs...

Environment

1. Make a new, empty virtual environment with Python 3.8 and activate it (.\venv_name\Scripts\activate );
2. pip install -r venv_requirements.txt; This might take some time; if you encounter conflicts on specific packages, install them manually without the ==<version>;
3. install manually PyTorch according to your CUDA VERSION;
4. Copy and paste the files you'll find in the folder whisper_edits to the whisper folder of your environment (.\venv\lib\site-packages\whisper\ ) these edits will add just an attribute to the whisper model to access its dimension more easily;
5. install TTS;
6. Run their script and check everything is working (it should download some models) (you can alternatively run demos/tts_demo.py);
7. Rename or delete the TTS folder and download the Assistant and other scripts from this repo
8. Install Vicuna following the instructions on the Vicuna folder or by running: cd Vicuna call vicuna.ps1 Manual instructions will instruct you to follow the Vicuna Installation Guide
9. paste all your keys in the env.txt file and rename it to .env (yes, remove the txt extension)
10. Check everything works (following)

Checks

• Verify your graphic engine and CUDA version are compatible with PyTorch by running torch.cuda.is_available() and torch.cuda.get_device_name(0) inside Pyhton; .
• run tests.py. This file attempt to perform basic operations that might raise errors;
• [WARNING] Check the FAQs below if you have errors;
• You can check the sources of error by running demos in the demos folder;

Step 2: Language support

• To have answers spoken in your language you should first check if your language is supported by the speech generator at https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-voices;
• If it's supported, add or change the languages inside VirtualAssistant.__init__() ;

• Remember: The loaded Whisper is the medium one. If it performs badly in your language, upgrade to the larger one in the __main__() at whisper_model = whisper.load_model("large"); but I hope your GPU memory is large likewise.

Step 3: Running (openai_api_chatbot.py):

When running, you'll see much information being displayed. I'm constantly striving to improve the readability of the execution, the whole project is a huge beta, forgive slight variations from the screens below. Anyway, this is what happens in general terms when you hit 'run':

• Preliminary initializations take place, you should hear a chime when the Assistant is ready;
• When awaiting for triggering words is displayed you'll need to say Jarvis to summon the assistant. At this point, a conversation will begin and you can speak in whatever language you want (if you followed step 2). The conversation will terminate when you 1) say a stop word 2) say something with one word (like 'ok') 3) when you stop making questions for more than 30 seconds

• After the magic word is said, the word listening... should then appear. At this point, you can make your question. When you are done just wait (3 seconds) for the answer to be submitted;
• The script will convert the recorded audio to text using Whisper;
• The text will be analyzed and a decision will be made. If the Assistant believes it needs to take some action to respond (like looking for a past conversation) the langchain agents will make a plan and use their tool to answer.
• Elsewise, the script will then expand the chat_history with your question, it will send a request with the API and it will update the history as soon as it receives a full answer from ChatGPT (this may take up to 5-10 seconds, consider explicitly asking for a short answer if you are in a hurry);
• The say() function will perform voice duplication to speak with Jarvis/Someone's voice; if the argument is not in English, IBM Watson will send the response from one of their nice text-to-speech models. If everything fails, the functions will rely on pyttsx3 which is a fast yet not as cool alternative;

• When any of the stop keywords are said, the script will ask ChatGPT to give a title to the conversation and will save the chat in a .txt file with the format 'CurrentDate_Title.txt';
• The assistant will then go back to sleep;

Keywords:

• to stop or save the chat, just say 'THANKS' at some point;
• To summon JARVIS voice just say 'JARVIS' at some point;

not ideal I know but works for now

History:

• [11 - 2022] Deliver chat-like prompts from Python from a keyboard
• [12 - 2022] Deliver chat-like prompts from Python with voice
• [2 - 2023] International language support for prompt and answers
• [3 - 2023] Jarvis voice set up
• [3 - 2023] Save conversation