The following introduces research visions, and problems currently being worked on.
Previously, I have spent about 10 years investigating first-principle derivation of Deep Neural Networks (DNNs).
To connect the dots, the journey started with the marvel over DNNs’ human-surpassing performance on vision tasks. There is a saying that “the greatest discovery of the 19th century was that the laws of nature are linear, and the greatest discovery of the 20th century was that they are not”. The key in DNNs is the nonlinear activation function, and thus at the beginning I studied how to understand the nonlinear activation function through harmonic analysis, statistical physics, and group theory in my M. Phil thesis.
The study led to the idea that a neural network could be characterized as a feedback control loop, where each layer processes information, and receives feedback from the next layer, which recursively from the output layer (which computes loss). In this loop, each input example is an information perturbation to the network, and the weights are wired in response to those perturbations to reduce loss and thus reduce errors in decisions. The impact of neural weights and activation function could be bounded by singular values (of the weight matrices); and thus to prevent a network from overreacting to examples, we restrict the network’s singular values to a band around 1. This restriction keeps the information propagation at the edge of chaos, and improves generalization. This work was published on TPAMI.
This idea of perturbation naturally connects to adversarial examples/perturbations, and thus we also study the network’s behaviors in response to adversarial training, and found that adversarial training could be understood as a form of regularization that is too strong. The takeaway is that adversarial examples are not bugs, but features, and fixing them requires foundational work. This work is published on ICML.
Those two works led me to crystallize the idea that the training of a DNN is a learning process that reduces informational perturbations, which previously was an idea in the background. More specifically, these previous two works tries to give a macroscopic characterization of the network, and further pursuing this idea led me to study how to connect microscopic neurons to macroscopic behaviors of DNNs, where the perturbation reduction process is characterized as a statistical mechanical characterization of DNNs, crystallizing the ideas in two papers—a letter introducing the main ideas, and a more technical article (where most technicalities are put in the ~100 pages appendices). An introductory blog is also written to introduce the article intuitively.
The basic idea is that Deep Neural Network (DNN) is a phenomenological model of biotic intelligence, and DNNs could be studied similarly to physics through symmetries, though the symmetries here are not conservative symmetries in physics, but a symmetry perhaps that might form a different paradigm, and is referred as adaptive symmetry. The learning process could be formalized as a symmetries-breaking process where these symmetries are broken by information perturbations to build a model of the world that could predict the future, such that the uncertainty of survival and reproduction is decreased; that is, to adapt to the world.
Those adaptive symmetries could be understood as choices that would be made by us with equal chance—humans are organic system as well:
Two roads diverged in a yellow wood,
And sorry I could not travel both. — Robert Frost
And chosen choices are broken symmetries.
As the first result of this characterization, it explains why DNNs have such strong fitting capacity that could find almost zero risk when it is large enough. Technically, there exists concentration of measures phenomena induced by aforementioned adaptive symmetries for large DNNs that enable learning of the network to stay within a phase where arbitrary learning errors could be reduced.
Those errors could be understood as cognitive errors that lead to mistakes (in the datasets, or environment). Or analogically, the wrong choices we have made in life. And in an oversimplifying way, the less errors we make, the more sophisticated situations we could handle, and the more complex our model perhaps is.
And to summarize this process in the format of the catch phrases in complexity science (e.g., “order from noise”), the training of organic system might be referred as biotic or cognitive complexity from adaptive-symmetries breaking.
The adaptive-symmetries breaking work sets the first step, or foundation, for the subsequent efforts. More specifically, the previous works analyze the simplest yet most fundamental setting, i.e., object classification; it is conceptually clean, yet fundamental intelligence arises from a being embodied in an environment whose reduction of survival uncertainty requires building a feedback and control loop (that is, daily feedback from environment and decisions on what actions to take) with the environment.
Therefore, the next stage is to study realistic open systems that interact with their environments. The potential complexity of the system and complexity of the environment should be sufficiently large, such that a level that we denote as intelligence in daily sense could be reached. This area of study is roughly known as Embodied Intelligence.
There are two fitting pilot problems to study this area, for example. First, the current breakthrough of large language model (LLM) is such an uncertainty-reducing system in a semantic word worlds; the system is trained by minimizing perplexity, another word for uncertainty. For LLM, the environment is the Internet, and thus LLM agents are systems that interact with the environment in the sense of embodied intelligence. Second, the next industrial-revolution scale innovation probably comes from cybernetic/control systems such as self-driving cars, and those are classic embodied systems. Those two problems are major challenges to realize the transformational impact of artificial intelligence on societies.
The goal of decreasing uncertainty of reproduction is decomposed into two problems by biotic intelligence by timescales: the prediction under, and the prediction above hundreds of milliseconds. The mechanism that predicts at the fast timescale is known as instincts, and the mechanism that predicts at the slow timescale is known as consciousness, reasoning, memory, and other modules function at our daily life timescale. These are known as system 1 and system 2.
Therefore, the study of embodied intelligence is broken down into studying these two mechanisms. The frontier of the research thus has reached the state where we have a system close to human instincts in language processing—this is known as pretrained Foundation Models. And the slow system is being investigated largely under the name of Agent, which requires long horizontal reasoning and interaction with the Internet, and also involves system 2 concepts such as long term memory. The bridge of the two is investigated under the name of Alignment, where the foundation model is aligned to human intentions, and becomes an agent.
Some clarification might be needed here, given that alignment has a connotation to safety issues. The developmental progress of an organism is a process where it explores possible actions in the environment, and accumulates a repository of action sequences such that it could be more fit to perpetuate in the environment. Therefore, to align a model is to align this developmental progress, and thus the capability alignment and human value alignment (i.e., the common denotation of “aligning” catastrophic superintelligence) are deeply intertwined. They both might be studied under the theme to find the right formalism of interaction between the environment (e.g., Internet) and the model/agent.
More concretely, some concrete angles that we are working on are briefly introduced in the following.
To move beyond the word world, current foundation models lack instincts in the physical world. Large language model builds this kind of instinct through pretraining on trillions of worlds, and this instinct is an abstraction of the word world. However, this way of building abstract over word world might not naively generalize to other categories of signals. Natural words are already coarse-grained (in other words, semantic) information packet of the words, and thus predicting next-word sidesteps the problem of building coarse-grained abstraction of the world from raw signals (e.g., pixels). An intelligence in the physical world (e.g., autonomous robots) needs somehow to have a mechanism to build abstraction from raw signals that are directly wired with control. A foundation model for control needs such abstraction capability, and this abstraction might manifest as animal-level instinct to navigate in the physical world.
A significant capability of the slow system manifests as long-horizon decision or reasoning capability. We study this problem under the setting of LLM. As discussed previously, next-word prediction might be understood as something that might be called instincts, and it lacks long-horizon decision capability to be useful as an competent assistant (e.g., one that we could truly delegate complex tasks). To make a language model “understand” our request, a mechanism of feedback and long-chain reasoning need to be incorporated. Currently, this ability is nascent in language model, and we are studying how to enable long-horizon reasoning and decision that require tool usage in LLM. This is probably what Q* or Gemini aims for.
The previous directions focus more on the problem side, and the previous problems likely would enable the new development of reinforcement learning (RL) algorithms. Intuitively, humans could make skills instinctive through deliberate practice, and also come up with high quality solution through careful reasoning and self critique. We also works on formalism of those two behaviors. For the latter, it is roughly known as language feedback or AI feedback for alignment. The idea is that generation is a NP problem, while discrimination is a P problem, and thus a good enough model could self-critique and self-improve. In addition to enabling scalable alignment, this might provide solution to hard RL problems such as sparse reward, and instability in RL training.