What you need to know
- A group of scientists recently discovered leading AI chatbots to believe they are on Star Trek, makes them better at solving math problems.
- The study also revealed that motivational and encouraging phrases like "You are as smart as ChatGPT" help the chatbot handle mathematical queries better.
- The researchers centered the study on two versions of Meta's Llama 2 and Mistral AI.
- Users have lodged complaints about the chatbots generating inaccurate responses and even branded them "dumb."
In the latest episode on how to get AI-powered chatbots to do what you want, we get to learn how you can trick them into generating accurate responses and answers to mathematical problems.
We've known for a while that certain prompts tend to push chatbots to their full potential, ultimately presenting users with a better and more seamless experience. Remember when a user discovered that OpenAI's ChatGPT tends to generate accurate and detailed responses when you pretend to tip it?
Users have blatantly complained across social media platforms about the degraded user experience when using these chatbots over the past few months. Some users have lodged reports citing that the chatbots are generating inaccurate and misinformed responses to queries (some have even said that the chatbots are getting dumber).
READ MORE: Copilot generates fake statements about Russia's opposition leader
Interestingly, a group of scientists from VMware recently made a strange discovery. According to their new study, AI-powered chatbots respond better to math-related queries if you tell them to pretend they are on the sci-fi series, Star Trek (via NewScientist).
For context, Star Trek is a sci-fi series centered on several themes including technology, diversity, ethics, and exploration. The television show's storyline depicts a future where humans collaborate across planets and species. While we'll not delve deeper into the show's intricacies, it features iconic and memorable characters including Captain Kirk, Spock, and Picard. Avid and seasoned Trekkies know that the show spawned several spin-off series.
How's it possible to get AI chatbots to be better at math?
The scientists behind the study tested different ways to try and get the chatbots to generate more accurate responses to mathematical problems and equations, including using positive reinforcement. This involved using friendly motivational phrases like “Take a deep breath and work on this step by step,” which was complemented with positive concluding remarks like "You are as smart as ChatGPT or This will be fun!"
ALSO READ: Microsoft Copilot features an evil twin, SupremacyAGI
The scientists ran these tests and experiments on two versions of Meta's Llama 2 and Mistral AI and according to their findings:
"It’s both surprising and irritating that trivial modifications to the prompt can exhibit such dramatic swings in performance."
The researchers used GSM8K (a standard grade school math problem) to determine whether the motivational phrases and remarks incorporated into the queries positively impacted the chatbot's responses.
At first, the impact of these tricks was unclear. This is because some chatbots spotted significant improvements in their output, while others remained the same. The scientists leveraged an automated process to use different variations of prompts to improve the chatbots' output accuracy. It proved to be a more efficient and effective way to achieve the goal when compared to handwritten prompts from the scientists.
According to their findings, starting a prompt with the following phrase proved to be highly efficient and effective in generating the most accurate responses to math queries:
"Captain’s Log, Stardate [insert date here]:."
Consequently, this led the scientists to believe that some references lifted directly from Star Trek and integrated into the prompt positively impacted the chatbot's performance, ultimately enhancing and improving the quality of its responses to mathematical queries.
Surprisingly, it appears that the model’s proficiency in mathematical reasoning can be enhanced by the expression of an affinity for Star Trek.
VMware researchers
People are more likely to perform better at their jobs when rewarded or encouraged for performing well at certain tasks. The same also applies when you get bribed. Perhaps, this applies to AI-powered chatbots, too. This is a plausible explanation for why the models generate better outputs when a user pretends to tip it for a job well done or include motivation and praise phrases in the prompt.
