Automatic Differentiation in Machine Learning: a Survey
Calculating derivatives of complex functions, especially those in machine learning models, is a critical but difficult task. Manual derivation is error-prone and infeasible for large models.
The paper surveys Automatic Differentiation (AD), reviewing forward mode and reverse mode. Reverse mode is shown to be exceptionally efficient for functions with many inputs and a single output (the loss function).
Reverse-mode AD is the foundational algorithm that powers the training of virtually all deep learning models — commonly known in the ML community as backpropagation.
The paper is a survey rather than a novel method, and focuses primarily on deep learning applications of AD.
Automatic differentiation is the enabling technology for training the neural networks at the core of the software agent (UI understanding), the typing robot (computer vision), and the humanoid robot (perception and control).
📇 Summary flashcard — 13 analytical fields for this paper
خلاصه
A comprehensive survey of Automatic Differentiation (AD), clarifying that backpropagation is an application of reverse-mode AD.
نمای سریع
Connects the mathematical concept of differentiation with its automated implementation in frameworks like TensorFlow and PyTorch.
یافتههای کلیدی
Reverse-mode AD is the foundational algorithm that powers the training of virtually all deep learning models — commonly known in the ML community as backpropagation.
هدف
To provide a unified view of AD for the ML community and show why reverse mode suits neural network training.
روش
The paper surveys Automatic Differentiation (AD), reviewing forward mode and reverse mode. Reverse mode is shown to be exceptionally efficient for functions with many inputs and a single output (the loss function).
نتایج
Systematizes AD theory and application in ML as a standard reference.
نتیجهگیری
Automatic Differentiation is mature and essential technology for modern AI.
مفاهیم کلیدی
automatic-differentiation، backpropagation، deep-learning، machine-learning، optimization، calculus
مطالعهی بیشتر
https://jmlr.org/papers/v18/17-468.html
تحلیل
Bridges the numerical-computation and machine-learning communities' independent discoveries of the same technique.
محدودیتها
The paper is a survey rather than a novel method, and focuses primarily on deep learning applications of AD.
کارهای آینده
AD for probabilistic models, higher-order derivatives, and novel hardware accelerators.
کاربرد عملی
Automatic differentiation is the enabling technology for training the neural networks at the core of the software agent (UI understanding), the typing robot (computer vision), and the humanoid robot (perception and control).
◀ References (this paper draws on) (0)
No references recorded.
Cited by (0) ▶
No paper cites this one yet.
Prerequisite learning path for this paper