AI talk: State of AI at the end of 2022

Dec. 16, 2022 / By V. “Juggy” Jagannathan, PhD

For my last blog post of 2022, I will explore the explosion of new research results reported in two major conferences that happened in the past few weeks, first is the annual Neural Information Processing (NeurIPS) conference and second is the Empirical Methods in Natural Language Processing (EMNLP) annual conference. I will also look at some concurrent news events and their implication for the overall state of AI.

NeurIPS 2022 

Just to get a perspective of the scale of innovation happening, NeurIPS published 2,900 papers from a pool of 9,634 submissions, and, for the first time, 10,000 people attended in person in New Orleans, and another 3,000 attended online. For a good overview of the entire conference, check out this YouTube video by Zeta Alpha.

One of the best markers for what is groundbreaking is the award given to top research papers. NeurIPS recognized 15 papers as outstanding and this post by a Nvidia researcher (also one of the award recipients) summarizes the papers quite succinctly. The awards highlight a string of advances in developing better ways to optimize models.

It is now possible to get photorealistic image generation from textual descriptions using what have been dubbed “diffusion models” (from Google) – a trend that started with the release of DALL*E from OpenAI.

Research from Allen AI focused on generating 3D interactive simulation environments of houses, which are ideal to train robots/agents (check out their demo world). A related effort from Nvidia provides rich simulation environments based on Minecraft to allow the training of AI agents using reinforcement learning.

What do the remaining papers cover? This blog has a nice word cloud created from the papers at the conference. Large language models (LLMs) continue to be a dominant topic area – how to make them small, how to make them efficient and how to combine them with other methods such as reinforcement learning and graph models.

Diffusion models are another big area and we already noted one of the outstanding papers is in this area, above. Automatic code generation and explainability of model output are always popular topics. Brain inspired architecture is another area. Of course, we don’t necessarily see the value of these methodical advances until something big shows up to convince us what is possible.

Deep learning pioneer Geoffrey Hinton gave the ending keynote presentation, which is in the realm of brain inspired architecture. He talked about a forward-forward architecture, motivated by the fact that the brain does not have a back propagation mechanism and is still able to do remarkable things. This from a person who literally wrote the book on back propagation! Hinton also opines that the future of deep learning is analog.

EMNLP 2022 

EMNLP used a hybrid format with Abu Dhabi as the conference location. They selected 1,400 papers from a pool of 4,190 submissions of different types. The papers and participants are literally from all over the world! The conference was also host to 24 workshops that brought researchers together to explore a wide variety of topics. Some of the themes explored include common sense reasoning, interpretability and models that leverage the notion of human in the loop.

One interesting theme in EMNLP is captured by this sentiment from Dr. Neil Cohn in his keynote speech: “Human expression is naturally multimodal.” To get a glimpse of his work, check out the visual language lab. In the industry track there was a speech from HuggingFace about their open repository of more than 75,000 models. The impact of HuggingFace and other open source projects cannot be underestimated. There is work done by big tech and then there is everybody else. Big tech has created monster models with incredible capabilities – while the rest of the world, including academia have relied on the open source models to advance the science.

State of AI

The breaking news during the NeurIPS conference was the release of ChatGPT from OpenAI, but with no attendant paper. Combining a derivative of GPT-3 with reinforcement learning from human feedback (RLHF), ChatGPT follows instructions and provides a cogent response. The responses have been stunning in their clarity and verbosity. More than a million people have tried out the trial interface from OpenAI. One of its capabilities is to write code given a description or even debug a piece of code to suggest what is wrong.

It makes errors for sure, particularly with math. Also, its knowledge base is frozen to 2021. Still, it’s capability is impressive to say the least. Google gives you a set of hits and the most relevant document, while ChatGPT answers your question in lucid prose. Check out some samples posted here. There are plenty of criticisms, with one observer saying it is good at producing “fluent BS” – like giving a long well-formed response on why crushed glass can be nutritional to consume! In fact, just this week, Stack Overflow, a popular question/answer platform has banned the use of ChatGPT fearing massive misinformation generation.

LLMs have progressed remarkably, but there are still significant problems to be solved before anyone can claim we are nearing the magic marker of artificial general intelligence (AGI). Foremost among the remaining challenges is dealing with common sense, physical reasoning, safety, ethics and multimodality. This year saw debate on the issue of whether these models are sentient. Needless to say, they are not and unlikely to be any time in the next decade.

Still with more than 4,000 research papers published just in the last few weeks, the AI revolution continues to amaze. We are clearly in a pivotal moment in AI.  

Acknowledgement 

I would like to acknowledge my colleagues who attended the NeuriPS and EMNLP conferences and shared their experiences and impressions: Mojtaba, Alireza, Arindam, Longxiang, John and Federico.

I am always looking for feedback and if you would like me to cover a story, please let me know! Leave me a comment below or ask a question on my blogger profile page.

“Juggy” Jagannathan, PhD, is an AI evangelist with four decades of experience in AI and computer science research.