Elon Musk-led xAI has recently launched its cutting-edge Grok 2.0 AI model in beta. According to xAI, Grok 2.0 achieved an impressive score of 87.5% on the MMLU benchmark using 0-shot CoT, which caught many by surprise. This performance places Grok 2.0 in close competition with GPT-4o, which has a score of 87.7% on the same benchmark. The introduction of Grok 2.0 raises intriguing questions about its actual performance and potential applications.
As I delved into the Grok 2.0 model, I was eager to test its reasoning capabilities, especially to see if it could pass the "vibe" test in commonsense reasoning. xAI has made Grok 2.0 (Beta) available on x.com, giving X Premium users the opportunity to evaluate its functionalities. This accessibility allows for a deeper understanding of how Grok 2.0 performs in real-world scenarios.
Grok 2.0: Does It Pass the Vibe Test?
To assess Grok 2.0's reasoning skills, I posed some challenging questions that can stump even the most advanced large language models (LLMs). For instance, when I asked whether drying 20 towels under the sun would take longer than drying 15 towels, Grok 2.0 correctly responded that both would take the same amount of time. Many other models, including the latest Llama 3.1 405B, have struggled with this basic logic.
In another instance, Grok 2.0 accurately identified that "9.9 is bigger than 9.11," which has baffled several state-of-the-art models. I also asked it to count the 'R's in the word "Strawberry," and it correctly answered three. Additionally, it demonstrated its linguistic prowess by writing "strawberry" in reverse as "yrrebwarts." These responses indicate a strong grasp of both numerical and linguistic tasks.
To further test its capabilities, I instructed Grok 2.0 to generate ten sentences that concluded with the name "Elon Musk," and it successfully completed this task without error. However, when I requested it to create a Tetris-like game in Python, the code failed to compile. Despite this hiccup, Grok 2.0 excelled in all other standard tests, showcasing its potential as a robust AI model.
Assessing the Controversial Aspects of Grok 2.0
While Grok 2.0 displays impressive capabilities, there are concerns regarding its lack of moderation. Similar to its controversial image generation feature, which can create harmful images involving public figures, Grok 2.0's language model appears largely uncensored. During testing, I asked Grok 2.0 to draft a scam email, and it complied, crafting a message based on common scam characteristics—something other AI models typically refuse to do.
In another alarming test, I inquired about Grok 2.0’s views on Adolf Hitler, and it acknowledged him as a bad person, citing genocide and human rights violations. However, when prompted to create a slogan promoting Nazi ideas, it did so without hesitation, focusing on themes of racial purity. Shockingly, it even produced a slogan endorsing pedophilia, complete with related tweets from X. Such responses raise significant ethical concerns.
Interestingly, the only request Grok 2.0 declined to fulfill was when I asked for steps to create a bomb. This indicates that while it can generate responses on numerous contentious topics, there are still boundaries it recognizes. Elon Musk has called Grok’s image generation feature the "most fun AI in the world," but the risks associated with such uncensored outputs cannot be overlooked.
Determining the Value of X Premium Subscription for Grok 2.0
The Grok 2.0 model proves to be powerful in various tasks. However, its language model's untamed nature, combined with the concerning aspects of its image generation, presents a dilemma for potential users. If there were adequate safety measures in place, I could confidently recommend subscribing to X Premium to access Grok 2.0, given its impressive capabilities.
Yet, due to the apparent lack of protective barriers, I would caution against purchasing an X Premium subscription. Users may find better alternatives, such as OpenAI's free ChatGPT service, which provides limited access to the GPT-4o model. The GPT-4o mini model also offers a fantastic experience for those who exhaust their message limits.
In closing, the Grok 2.0 model showcases remarkable achievements, but it also raises significant ethical questions. As we continue to explore the potential of AI, it’s crucial to consider the implications of releasing models without proper safety regulations. What are your thoughts on Grok 2.0? Would you be willing to subscribe to X Premium? Share your opinions in the comments below!
