If you are running local llms, AMA is one of the best options out there

Introduction

In a recent comment on my Discord channel, Justin asked if it’s possible to use a model loaded up on LM Studio instead of using AMA (Adaptive Model Artifacts). This got me thinking, why not use AMA with local GPT? Local GPT is a project that allows you to chat with your documents on your local devices or private cloud using the power of llms (large language models). In this video, I will show you how to use a model running on AMA with local GPT, with just two extra lines of code.

Setting up Local GPT

The first step is to set up local GPT. To do this, you need to clone the local GPT repository. You can do this by copying the repository link and using the Git clone command. Once the repository is cloned, you need to create a virtual environment for local GPT using the command “conda create -n localGPT python=“. Activate the virtual environment using the command “conda activate localGPT”. Next, install all the required packages using the command “pip install -r requirements.txt”.

Preparing Documents

In order to use local GPT, you need to have some documents to chat with. The local GPT repository comes with the original Ora paper in the source documents folder. However, you will need to provide your own documents. Copy your documents into the source documents folder. If you are using a preconfigured virtual machine from MK compute, the welcome document will guide you on how to move your documents to the remote virtual machine.

Creating a Vector Store

Next, you need to create a vector store for the documents. This can be done by running the python script “injest.py” in the local GPT repository. This script takes the documents in the folder, creates chunks, computes embeddings, and puts them in a vector store. The time taken for this step depends on the number of files you have.

Configuring OpenAI Local

In order to run an llm using OpenAI Local (olama), you need to download and install AMA. Go to the OpenAI website, download AMA, and select the appropriate option based on your operating system. Follow the installation instructions provided. Once AMA is installed, choose an llm model to run. In this example, we will use the “Mistol” model. Copy the command for running the model and paste it into your terminal. The first time you run the command, it will download the model, so it may take some time. Once the model is downloaded, it will be ready to start interacting with.

Using AMA with Local GPT

Now that we have both AMA and local GPT set up, we can use AMA with local GPT. In the local GPT repository, open the “run_localgpt.py” file. Import the “oLama” package from the “lchaim” library using the line “from lchaim.llms import oLama”. Please note that the version of lchaim used in this example is relatively older, but it is the version that supports AMA. Add this line to the file. Next, go to the line where the model is loaded (line 140) and comment out the existing code. Replace it with the line “lm = oLama(model_name=’Mistol’)”, where ‘Mistol’ is the name of the AMA model you are using. Save the changes.

Running Local GPT with AMA

With the setup complete, you can now run local GPT with AMA. In the terminal, navigate to the local GPT directory and run the command “python run_localgpt.py”. The output will show that the embedding model is loaded, but not the llm from the constants file, as we are running the llm from AMA. You can now interact with the model by asking questions or starting a conversation.

Conclusion

Using AMA with local GPT allows you to leverage the power of llms in your local environment. With just a few lines of code, you can integrate AMA models into your local GPT framework. Local GPT is a flexible framework that allows you to bring in different components from various sources. Give it a try and let me know what you think!