Google Releases Gemma Model: A Game Changer in AI
Introduction
Last week, Google released their second version of their proprietary models called Gemini 1.5. However, this morning, they shocked the whole industry with the release of an open weights model which they named Gemma. Gemma is a family of lightweight state-of-the-art open models inspired by the research done for Gemini models.
Technical Details
Google has released two variations of Gemma – a 2 billion parameter model and a 7 billion parameter model. These models are state-of-the-art in their category in terms of performance benchmarks. Although not completely open source, they are open weights models similar to what Meta and MRAL AI have done.
Responsible Generative AI Toolkit
Google also released a responsible generative AI toolkit, providing safeguards for user input and model responses. This framework ensures model safety and addresses criticisms of the output produced by Gemini models.
Technical Report
According to benchmarks results, the Gemma model outperforms similar sized models on 11 out of 18 text-based tasks. It excels in math and coding tasks, showcasing significant performance improvements.
Model Architecture
Gemma is based on a transformer decoder and features a context window of 8,000 tokens. It includes multi-query attention and rope embedding for extending the context window. The activation function used in Gemma is GGU activation, different from the traditional ReLU activation.
Training Data
The 2 billion model is trained on 2 trillion tokens, while the 7 billion model is trained on 6 trillion tokens. Gemma focuses on English data and does not support multilingual capabilities.
Fine-Tuning and Licensing
Google provides Google Colab notebooks for testing Gemma and offers resources for fine-tuning the model within Caros NLP. The terms of use permit responsible commercial usage and distribution for all organizations.
Conclusion
Google’s release of the Gemma model represents a significant advancement in AI technology. The model’s performance on various benchmarks and its technical details make it a game-changer in the industry. Stay tuned for more updates and content on Gemma as the community explores its potential.
