By: Shreyash Singh RIG Inc Intern Researcher

According to Wikipedia, Artificial General Intelligence (AGI) is the hypothetical ability of an artificially intelligent agent to understand or learn any intellectual task that a human being can. Simply put, Artificial General Intelligence refers to the intelligent systems and robots we see in sci-fi movies and literature, that we were promised sometime in the early 21st century.

There can be two types of AI systems— “Narrow AI”, which can perform only a specific task, and “Strong AI” or Artificial General Intelligence (AGI), that can perform any task as well as a human being, but this is a much harder problem and there isn’t a clear roadmap to solve it. Many researchers have been saying that AGI is still decades away, and some even believe that it will never happen. AGI was far from achievable, until now! The introduction of OpenAI’s latest language model — GPT-3 (Generative Pre-trained Transformer 3) is the first model to challenge this status quo.

The OpenAI team first introduced GPT-3 in a research paper in May 2020, and in July they gave access to beta testers to test the model. GPT-3 has been used for text summarization, writing poetry and stories, generating code and chatbots. These are AI problems that can already be achieved by other systems. So, what makes GPT-3 so special? GPT-3 is one of the most powerful language models. OpenAI’s previous model, GPT-2, which was released in 2019, was already able to achieve surprisingly good performances by generating fairly convincing text streams when prompted with an opening sentence, but, GPT-3 is a huge step forward. The model has 175 billion parameters compared to the GPT-2’s 1.5 billion and is trained on large datasets with hundreds of billion words.

Some Examples of GPT-3’s abilities

Since the launch of GPT-3, developers and users have been flooding the internet with impressive things the model has been able to achieve.

“I am open to the idea that a worm with 302 neurons is conscious, so I am open to the idea that GPT-3 with 175 billion parameters is conscious too.” — David Chalmers

  • Here’s an excerpt posted by a twitter user of an interesting story generated by GPT-3 when only prompted with the title, the author’s name and the first “It”, the rest was done by GPT-3.

    Figure 1. GPT-3 Generated conversation (source: twitter-@quasimondo)

  • Amazingly, GPT-3 can even do things that its creators did not think of. After OpenAI started providing beta access of its API to select developers, some of them showed that it was possible to have GPT-3 generate functional JavaScript code from a natural language prompt. The video below shows an impressive GPT-3 model which generates JSX code for you when given a description in English.

    Sharif Shameem on Twitter: “This is mind blowing. With GPT-3, I built a layout generator where you just describe any layout you want, and it generates the JSX code for you.” @sharifshameem

What makes GPT-3 so powerful

GPT-3 is essentially a transformer model. Transformers are sequence-to-sequence deep learning models that are capable of producing a stream of text given an input sentence or word [3]. Transformers are designed for text generation tasks such as chatbots, text summarization and machine language translation. The figure below shows a transformer model that translates a sentence in French into English.

Figure 2. Transformer model for Machine Translation.

Based on the original paper that introduced this model, GPT-3 was trained using a combination of the following large text datasets:

  • Common Crawl
  • WebText2
  • Books1
  • Books2
  • Wikipedia Corpus

The final dataset contained a large portion of web pages from the internet, a giant collection of books, and all of Wikipedia. Researchers used this dataset with hundreds of billions of words to train GPT-3 to generate text in English and several other languages.

Limitations of GPT-3

Like all technological advancements over time, GPT-3 is definitely not perfect and that is the reason many still remain skeptical about achieving Artificial General Intelligence. Though GPT-3 can generate coherent sentences which may or may not make sense; GPT-3 is not immune to one of the fundamental problems of AI, bias. Similar to other ML models, GPT also suffers from bias mainly due to the type of data available. Take a look at this example posted by Twitter user Jerome Pensenti (@an_open_mind) where the sentences generated by GPT-3 seem to be hateful and misogynist.

Figure 3. Jerome Pesenti on Twitter: “#gpt3 is surprising and creative but it’s also unsafe due to harmful biases. Prompted to write tweets from one word – Jews, black, women, holocaust – it came up with these ( We need more progress on #ResponsibleAI before putting NLG models in production.”

What the future holds

GPT-3 might not be perfect but it does in a sense “generalize” any language task that is thrown at it — although the levels of performance vary. The question then becomes, can we call this AGI, or is it just a “Strong-AI”? The GPT-3 model is what we know as an ‘ungrounded’ model, which means that is has only vague notions of the problem it is trying to solve. For instance, it isn’t clear if GPT-3 “knows” that J.K Rowling is real and Harry Potter and the Dementors are not. However, a human being with the same limitations, being forced to learn only by the written word would still learn like any other human being. In turn, we are not sure that being grounded is a condition for intelligence.

With GPT becoming more powerful with each version released, and with a massive 175 billion parameters, we begin to think, are we close to the bottleneck and will making improvements become more and more difficult? But Google has decided to step-up their game and come up with a model that is six times bigger than GPT-3. With an incredible 1.6 trillion parameters, Google’s novel Switch Transformer aims to set the bar higher than ever before. Apart from having a huge number of parameters, with this model Google is essentially unveiling a new method to maximize the parameter count in a simple and computationally efficient way.

If that wasn’t enough, a Chinese AI institute has unveiled a new NLP model that is even more sophisticated than those created by OpenAI and Google. The WuDao 2.0 is an NLP model that was designed by the Beijing Academy of Artificial Intelligence (BAAI) and developed with the help of over 100 scientists from multiple organisations around the world. It uses 1.75 trillion parameters to simulate conversations, generate summaries and even create recipes.

Final Thoughts

These machine learning models are not perfect but provide us tremendous insight into what is achievable and if we can correctly utilize the resources and optimize the algorithms. The final product is still a major issue. With failing chatbots and generated sentences which make little sense, we get an idea of the limitations of these machine learning models.

RIG’s Dynamic TrustTM addresses complexities involved with the connections across the Artificial Intelligence network and determines a Trust level based on the usage history and intended data. This trust level is continuously changed based on the agents’ conditions. Dynamic Trust can help the machine learning systems make better final decisions by determining a trust level for the output produced by the model considering various parameters and conditions. This may eliminate some issues like inappropriate responses by a chatbot or generated text which might be racist or derogatory.

Artificial Intelligence has come a long way in the past decade and with the introduction of models like GPT-3 and WuDao 2.0, it is clear that there are still a lot of obstacles to overcome before we achieve Artificial General Intelligence, but we are definitely much closer than we were a decade ago.