Modern deep learning methods have achieved significant performance on various tasks by transforming the data in a layer-by-layer manner. The residual learning, which introduces skip-connection to deep structures, further overcomes vanishing of gradients and empowers the deep learning methods with deeper structures and thus more powerful expression ability. However, in the perspective of researchers in dynamic systems (control, applied physics, systems biology etc.), such a residual-net framework is the approximation of differential equation (DE) systems by Euler method (which is a 1st order numerical method in solving DEs' initial value problems). Thus, fundamental questions are raised: Can we view (residual-) deep models as dynamic systems and then use differential equation theory as a framework to analyze them? Can we design new and more powerful deep models inspired by dynamic systems? What are the benefits by introducing DE theory to deep learning? What are the potential new applications?

The boldest goal of this tutorial is to bridge the gap between the modern deep learning methods in computer science and DE theory (developed in control, applied math, physics, systems biology, numerical computation, etc.), and further to broaden the horizon of the deep learning methods with an emphasis on deep learning methods on graphs (also termed networks) and their applications. We will show that differential deep learning on graphs are powerful tools to model the structures and dynamics of networks and to generate molecular graphs. We summarize the contents covered in this tutorial as follows.

We will start by introducing the basic idea of Differential Deep Learning (DDL). We motivate the audiences by connecting the dots between deep learning models and dynamic systems modeled by differential equations. Basic ideas of differential equations (DEs) and physics models are illustrated. We will show examples of how to make deep structures into differential equation systems. We will conclude this part by summarizing the importance and values of differential deep learning.

In the perspective of dynamic systems, the residual learning framework, e.g. Residual-Net, is the approximation of differential equation (DE) systems by Euler method (1st order numerical method in solving DEs' initial value problems). We show the evolutionary path from CNN to Residual-Net and further to ODE-Net. Inspired by the CNNs developed for images, Graph neural networks (GNN) are proposed to deal with combinatorial graph data. We show how to get Residual-GNN, ODE-GNN, and our neural-dynamics-on-complex-network model. The promising experimental results demonstrate our model's capability of jointly capturing the structure, dynamics and semantics of complex systems.

Normalizing flows are one of the most promising deep generative models. We show how to use flow-based models for graphs, especially on molecular graph generation. We introduce our MoFlow model which achieved the state-of-the-art performance on molecular graph generation.

"Data Science lacking a model of reality may be statistics but hardly a science. --Judea Pearl". In physics, differential equations are one of the most successful models which describe the time-varying reality well. We motivate the problem by briefly introducing the history of network science. Then we show how to get DEs from statistical distribution functions by constructing dynamic systems. We show the relationships between DEs and the distributions functions. Many DEs in network science and statistical physics can be found in a principled way}.

Three applications of differential deep learning on graph models are discussed: molecular graph generation, namely to generate novel molecules with optimized properties; learning dynamics on graphs, namely to predict temporal change or final states of complex systems; and mechanism discovery in graphs, namely to find dynamical laws of complex systems and distributions.

We conclude the tutorial by introducing related research topics. The limitations and potential future directions are then discussed. We will leave some time for questions and answers, feedback and further discussions.

• Introduction: Differential Deep Learning and their graph applications (45 min) [PDF] [PPT]
• Introduction to graphs and differential equations.
• What is differential deep learning? Integrating differential equations into deep models.
• Importance and challenges.
• Graph applications: molecular graph generation, learning dynamics on graphs, and mechanism discovery.
• Molecular graph generation (45 min) [PDF]
• Drug discovery.
• Molecular graph generation and normalizing flow models.
• Our MoFlow model for molecular graph generation
• Experiments on molecule generation, reconstruction, visualization and optimization.
• Learning dynamics on graphs. (45 min) [PDF]
• Learning dynamics of complex systems.
• Our differential graph neural networks.
• Interpretations from Residual-Net, GNN, RNN, and temporal GNN.
• Experiments on continuous-time network dynamics prediction, structured sequence prediction, and node classification.
• Mechanism discovery in graphs (45 min) [PDF]
• From data to statistics, and further to dynamical models.
• Density estimation in computer science vs. mechanism discovery in network science.
• A theorem bridging distributions and differential equations.
• Data-driven discovery of differential equations and distributions.
• Conclusion: Discussions and Future Directions (30 min) [PDF] [PPT]

Due to size issue, contact my email for PPTs which have more animations than PDFs.

3 hours, 30 minutes, plus 30-minute break. This tutorial will be held at The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), February 7-12, 2020, 2:00 PM - 6:00 PM at Sutton North Room, the Hilton New York Midtown, New York, New York, USA. Friday, February 7, 2020

All the researchers and practitioners engaged in data mining and machine learning are welcome. Basic knowledge on deep learning, graph mining, and differential equations is preferred but not required. The estimated number of participants is 100.

Chengxi Zang is currently a Postdoctoral Research Associate collaborating with Dr. Fei Wang in the Weill Cornell Medicine. He got his PhD from Tsinghua University in 2019 with Excellent Ph.D. Award in Tsinghua University (top 3%). Dr. Zang joined in Cornell since February 2019. Dr. Zang has worked extensively on data-driven dynamical modeling of complex social and biological systems. For example, he is the first one who studied the evolutionary dynamics, social dynamics, and human behaviors of WeChat at billion scale. He also investigated millions of information cascades in Tencent Weibo. His methodologies are adopted from data mining, network science and machine learning. Right now, he focused on automated drug design by differential deep learning methods. Dr. Zang has published many papers in top data mining and machine learning conferences/journals including KDD, ICDM, TKDE etc.

Fei Wang is an Associate Professor of Department of Healthcare Policy and Research and Computer Science Field Member at Cornell University. He has in total 26 AAAI/IJCAI publications. His google scholar profile is here