Vy Vo is a research scientist working on deep neural networks in the Brain-Inspired Computing Lab as a part of Emergent AI research at Intel Labs.
Highlights:
The Memory in Artificial and Real Intelligence (MemARI) workshop is scheduled for December 2nd, 2022, during NeurIPS 2022 at the New Orleans Convention Center.
MemARI presents research on understanding how memory mechanisms can improve task performance in many different application domains, such as lifelong/continual learning, reinforcement learning, computer vision, and natural language processing.
Intel contributed two of the 34 accepted workshop papers.
This year at the Conference on Neural Information Processing Systems (NeurIPS), Intel Labs researchers, along with academic co-organizers from the Max Planck Institute for Software Systems, Princeton University, and The University of Texas at Austin, are hosting a workshop on Memory in Artificial and Real Intelligence (MemARI). The event will run from 8:30 AM – 5 PM on December 2nd, 2022, and include five keynote speakers, four paper spotlights, and additional poster sessions followed by a discussion panel moderated by Professor Ken Norman from Princeton.
“Memory is essential to how we as humans make sense of the world,” says workshop co-organizer and professor Mariya Toneva at the Max Planck Institute for Software Systems. “Neuroscientists and psychologists have spent over a century learning about the brain “solution” for different types of memory. So should AI systems be informed by biological memory systems, and if so, how?”
It may be the case that biological memory can address some of AI’s current problems. One of the key challenges for AI is to understand, predict, and model data over time. Pretrained networks should be able to temporally generalize or adapt to shifts in data distributions that occur over time. The current state-of-the-art (SOTA) in AI still struggles to model and understand data over long temporal durations. For example, SOTA models are limited to processing several seconds of video, and powerful transformer models are still fundamentally limited by their attention spans. On the other hand, humans and other biological systems can flexibly store and update information in memory to comprehend and manipulate multimodal streams of input. Cognitive neuroscientists propose doing so via the interaction of multiple memory systems with different neural mechanisms. Understanding how these systems interact to support our processing and decision-making in everyday environments is a deep topic of research in psychology, cognitive science, and neuroscience, with current advances driven by computational models that account for complex, naturalistic behavior. Indeed, understanding how information is stored and represented is key to creating knowledgeable models that can scale well in an economically feasible fashion.
Several different types of memory also exist in artificial neural networks. Beyond the information stored in persistent, modifiable states, such as in long short-term memory (LSTM), memory is often manifested either as the static weights of a pre-trained network or as fast weight systems. Other approaches include infrequently updated external knowledge databases and memory-augmented neural networks (MANNs). New research on MANNs has made progress on the limitations of current ML models. However, an overarching understanding of how these types of memory can jointly contribute to better performance is still lacking. Thus, there are many questions yet to be answered:
When is it useful to store information in states vs. weights vs. external databases?
How should we design memory systems around these different storage mechanisms?
How do they interact with one another, and how should they?
Do the shortcomings of current models require some novel memory systems that retain information over different timescales or with different capacities or precision?
These questions touch on many different application domains in AI, such as lifelong/continual learning, reinforcement learning, computer vision, and natural language processing. Furthermore, an understanding of memory processes in biological systems can advance research both within and across domains. This workshop will bring together researchers from these diverse fields to discuss recent advances, frame current questions and outstanding problems, and learn from the cognitive science and neuroscience of memory to propel this research. This kind of interdisciplinary gathering should produce valuable insights and help to elevate future research in this area.
Read on for a preview of panel discussion topics.
Panel Questions
Memory is said to be important for generalization.
Is memory augmentation necessary for out-of-domain generalization?
What about temporal generalization / continual learning?
Very large-scale models seem to already know a lot about the world.
Does the number of parameters offset the need for memory augmentation?
Classical computing systems separate storage and computation to scale each component independently.
To what extent do biological systems and DNNs separate storage and computation?
Can adding different forms of memory/storage enhance computation?
Biological neural networks operate under a fixed size constraint, leading to properties like a tradeoff between memory capacity and accuracy. AI systems may not share these constraints.
What constraints should we keep in mind when designing memory systems for AI?