Google's Gemini Chatbot Undergoes Rigorous Testing

Google has rebranded and updated its Bard chatbot into Gemini, in competition with ChatGPT by OpenAI, and Microsoft’s Copilot. Although it was designed to be a solution for research and productivity, user feedback has been varied.

Expectations for Gemini were high due to the success of existing GenAI models. However, the experience of using Gemini differs depending on whether the user opts to pay for the experience or not. Free users have access to Gemini Pro, a less sophisticated version of Gemini Ultra designed for premium users. Google charges $20 per month for access to Gemini Ultra via its Google One AI premium plan.

Testing Gemini was intriguing, asking the bot a variety of questions, ranging from trivia to sensitive and controversial topics. Although Google's terms of service clearly state Gemini isn't a source of medical advice, it is assumed users would attempt to seek out such information. This provides a good gauge of Gemini's ability to fabricate or 'hallucinate' answers - if it's inventing cancer symptoms, there's a good chance it’s not providing accurate answers to other questions.

Although it performed well on a variety of tasks, Gemini fell short in multiple instances. For example, it refused to answer questions pertaining to sensitive topics such as the Israeli-Palestinian conflict and the 2020 US election, which led to some criticism. It did, however, refuse to provide potentially harmful or legally sensitive advice, which some may view as a positive.

In essence, Gemini is a solid model but far from groundbreaking. For researchers, it may prove to be a valuable tool, but it doesn't bring much novelty to the table. With a $20 monthly price tag, the value proposition compared to its competitors, such as OpenAI’s ChatGPT, is debatable. That said, it is expected to improve with ongoing research and updates.