In this tutorial we will be converting an existing Gemma LLM LoRa file from HuggingFace to a LoRa file compatible with Google Gemini for use with the Syntattic LLM Unreal plugin.

Prerequistes​

Converting an existing LoRa model to Tensorflow lite format requires a Python development environment, in this example we will be running a Jupyter file using Google Colab. You will also need a Huggingface account and be willing to connect the Colab notebook to your Google drive.

Training a LoRa​

To train the LoRa we are going to use a modified version of the Colab notebook that can be found here. The modified version is stripped down to just the code that is needed to convert a Google gemma LoRa. Each block of code contains detailed instructions on how to customise the script to create an arbitrary LoRa.

You can compare against the original file to make the modifications to train a LoRa in the “Falcon 1B”, “StableLM 3B” and “Phi 2” formats. The Jupyter notebook contains comments that explain what to modify in each section.

The Colab notebook has three sections which do the following :

• Imports the required dependencies.
• Downloads the required files for the gemini model and the LoRa we want to apply and builds a simple UI to enter your Huggingface credentials and a button to start the conversion to Tensorflow lite format.
• Saves the converted files to your Google Drive.

Using the LoRa with Unreal​

To use the converted LoRa files with Unreal :

• Download the LLM and LoRa file from your Google Drive.
• If the LLM + LoRa file size is over 2GB
  • Place the LLM on a publicly accessible server.
  • Set the “LLM download location” to the web address of the file location on the server.
• If the LLM + LoRa file size is less than 2GB
  • Select “Package” as the delivery method in the project settings.
  • Press the “LLM model” field in the project settings and select the LLM file.
  • Change the LoRa dropdown to “Enabled” .
  • Press the “LoRa model” field in the project settings and select the LoRA file.
  • Further information, including troubleshooting and tips can be found here.