Deep learning
For the TV series episode, see Deep Learning (South Park).
In machine learning, deep learning (DL) focuses on using multilayered neural networks to perform tasks such as classification, regression, and representation learning. The field takes inspiration from biological neuroscience and works by stacking artificial neurons into layers and "training" them to process data. The word "deep" refers to the use of many layers in the network, which can range from three to several hundred or thousands. Methods can be supervised, semi-supervised, or unsupervised.
Some common deep learning network designs include fully connected networks, deep belief networks, recurrent neural networks, convolutional neural networks, generative adversarial networks, transformers, and neural radiance fields. These designs have been applied to fields including computer vision, speech recognition, natural language processing, machine translation, bioinformatics, drug design, medical image analysis, climate science, material inspection, and board game programs. In many of these areas, they have produced results comparable to, and in some cases better than, those of human experts.
Early neural networks were inspired by how information is processed and shared between nodes in biological systems, especially the human brain. However, current neural networks are not meant to copy how the brain of a living thing works. They are generally seen as low-quality models for that purpose.
Overview
Most modern deep learning models are based on multi-layered neural networks such as convolutional neural networks and transformers. Some also use propositional formulas or hidden variables arranged in layers, as in deep generative models such as deep belief networks and deep Boltzmann machines. Deep learning is a type of machine learning algorithms that passes data through a series of layers, turning it into a more and more abstract representation. In an image recognition model, the input is an image, stored as a tensor of pixels. The first layer may find basic shapes like lines and circles, the second may combine them into edges, the third may find a nose and eyes, and the fourth may recognize a face.
Before deep learning, people often had to design feature engineering by hand to prepare data. With deep learning, the model discovers useful features from the data on its own, though people still need to adjust settings such as the number of layers. The word "deep" refers to how many layers the data passes through. Most researchers agree that a system is deep when it has more than two layers of transformations, and deeper models can often pick out better features. Deep learning can also work with feedforward neural networks and recurrent neural networks, and it can be built one layer at a time using a greedy method. Because unlabeled data is more common than labeled data, deep learning can also be used for unsupervised learning, as in deep belief networks. The term was introduced to machine learning by Rina Dechter in 1986, and to artificial neural networks by Igor Aizenberg and colleagues in 2000, in the context of Boolean threshold neurons.
Interpretations
Deep neural networks are usually explained in one of two ways: the universal approximation theorem or probabilistic inference. The classic theorem is about how well feedforward neural networks with one hidden layer of limited size can approximate continuous functions. In 1989, George Cybenko published the first proof for sigmoid activation functions. In 1991, Kurt Hornik extended this to multi-layer networks. Later work also showed that the idea holds for activation functions like Kunihiko Fukushima's rectified linear unit.
For deep neural networks, the theorem looks at networks with a limited width whose depth can grow. Lu et al. proved that with ReLU activation, a network whose width is larger than its input dimension can approximate any Lebesgue integrable function. If the width is smaller or equal, the network is not a universal approximator. The probabilistic view comes from machine learning. It uses ideas of inference and of optimization, including training and testing, which relate to fitting and generalization. This view treats the activation nonlinearity as a cumulative distribution function and led to the use of dropout as a regularizer in neural networks. It was introduced by researchers including Hopfield, Widrow and Narendra, and became popular through surveys such as the one by Bishop.
History
Early ideas about networks of artificial neurons appeared long before modern computers. Frank Rosenblatt proposed the perceptron in 1958, and in 1965 Alexey Ivakhnenko and Lapa published the first working deep learning method. Researchers such as Shun'ichi Amari and Kunihiko Fukushima made important early steps, including a method called backpropagation that helps networks learn from their mistakes. In the 1980s and 1990s, Yann LeCun and Jürgen Schmidhuber built networks that could read handwritten numbers and remember information over long sequences. Speech recognition and other tools were improved along the way, but progress was slow because computers were not yet powerful enough.
Deep learning grew quickly in the 2000s and 2010s. Using graphics processing units (GPUs) made training large networks much faster, which helped a network called AlexNet win the large ImageNet competition in 2012. Researchers then created new tools, such as word2vec for understanding words and the residual neural network for training very deep networks. Other systems learned to create images, and generative adversarial networks became popular. In 2018, Yoshua Bengio, Geoffrey Hinton, and Yann LeCun received the Turing Award for their work on deep neural networks.
Main article: Artificial neural network
Artificial neural networks (ANNs) are computer systems inspired by the biological neural networks in animal brains. They learn to do tasks by studying examples, usually without being given special instructions for each task. For instance, to recognize cats in pictures, an ANN can study example images labeled "cat" or "no cat" and then use what it learned to find cats in new images. They are most useful for problems that are hard to solve with rule-based programming. An ANN is built from connected units called artificial neurons, which pass signals to one another. These neurons are usually arranged in layers, and signals travel from the first (input) layer to the last (output) layer. Neural networks are used for many tasks, including computer vision, speech recognition, machine translation, and medical diagnosis.
A deep neural network is a neural network with many layers between its input and output layers. The word "deep" refers to these many layers. At first, the network gives its connections random values, called "weights." When it makes a mistake, the network adjusts these weights so that it gets better over time. Deep networks can have problems, such as "overfitting," where they learn rare details in the training data instead of general patterns. Methods such as dropout, which randomly leaves out some units during training, can help reduce this problem. Training these networks can also take a lot of time and computer power.
Hardware
Since the 2010s, better machine learning methods and computer hardware have made it faster to train deep neural networks with many layers. By 2019, graphics processing units (GPUs), often with AI-specific improvements, had replaced CPUs as the main tool for training large AI systems in commercial cloud computing. OpenAI estimated that the computing power used in the largest deep learning projects from AlexNet (2012) to AlphaZero (2017) grew 300,000-fold, doubling about every 3.4 months.
Special electronic circuits called deep learning processors were designed to speed up deep learning. Examples include neural processing units (NPUs) in Huawei cellphones and cloud computing servers, such as tensor processing units (TPU) in the Google Cloud Platform. Cerebras Systems also built a system called the CS-2 for handling very large deep learning models. It is based on the largest processor in the industry, the second-generation Wafer Scale Engine (WSE-2).
Researchers are exploring new materials and designs to make deep learning hardware more energy-efficient. Some work uses semiconductors that are only a few atoms thick. In 2021, J. Feldmann et al. proposed a light-based hardware accelerator that uses integrated photonics to process data very quickly, which could be useful for data-heavy AI applications.
Applications
Main article: Speech recognition
Main article: Computer vision
Main article: Natural language processing
For more information, see Drug discovery and Toxicology.
Main article: Recommender system
Main article: Bioinformatics
Main article: Epigenetic clock
Deep learning is used in many areas. Large-scale speech recognition is one of its first and most convincing successes, and all major commercial speech recognition systems are based on it. Deep learning image recognition has become "superhuman", producing more accurate results than human contestants. This first happened in 2011 with traffic signs and in 2014 with human faces. It is also used in visual art, such as identifying the style period of a painting. In language, Google Translate uses a large long short-term memory network and supports over one hundred languages. Word embedding, such as word2vec, is another key technique in this field.
Deep learning has also been used in science and medicine. AtomNet, a deep learning system, was used to predict new candidate biomolecules for diseases such as the Ebola virus and multiple sclerosis. In bioinformatics, deep neural networks have shown strong performance in predicting protein structure. In 2020, AlphaFold reached a level of accuracy significantly higher than all previous computational methods. Deep learning has also been used for fraud detection and to analyze medical images. In November 2023, researchers at Google DeepMind and Lawrence Berkeley National Laboratory announced GNoME, an AI system that discovered over 2 million new materials. GraphCast, a deep learning model, can predict weather conditions for up to 10 days.
Relation to human cognitive and brain development
Deep learning is closely related to ideas about how the brain development works, especially the growth of the neocortex, which were proposed by cognitive neuroscientists in the early 1990s. Those ideas were turned into computer models, so they came before today's deep learning systems. In these ideas, chemicals such as nerve growth factor help the brain self-organize, a bit like the neural networks used in deep learning. Both work through layers, where each layer takes in information from the layer before it and passes its results on, creating a self-organizing stack of transducers that fit their surroundings.
Scientists have tried many ways to test whether deep learning models resemble the brain. Some researchers changed the backpropagation method to make it more like real brain processes. Others think that learning without labeled examples, using generative models or deep belief networks, may be closer to how the brain works. Although a full comparison between the human brain and deep networks has not yet been made, some similarities have been noticed, such as in how the units in deep learning compute and how they represent what they see.
Commercial activity
Facebook's AI lab performs tasks such as automatically tagging uploaded pictures with the names of the people in them. Google's DeepMind Technologies developed a system that could learn to play Atari video games using only the pixels on the screen as input. In 2015, they showed their AlphaGo system, which learned the game of Go well enough to beat a professional Go player. Google Translate uses a neural network to translate between more than 100 languages. In 2017, Covariant.ai was launched to bring deep learning into factories.
As of 2008, researchers at The University of Texas at Austin (UT) developed a framework called TAMER, which helped robots or computer programs learn tasks by interacting with a human instructor. In 2018, a newer method called Deep TAMER was introduced through a collaboration between the U.S. Army Research Laboratory (ARL) and UT researchers. It used deep learning to let a robot learn new tasks by watching, either through video or by observing a human in person. The robot then practiced the task with coaching from its trainer, who gave feedback such as "good job" and "bad job".
Criticism and comment
See also: Explainable artificial intelligence
Deep learning has drawn criticism and comment, sometimes from people outside computer science. A main concern is that the theory behind some methods is unclear. Common deep learning systems learn through well-understood gradient descent, but the theory behind other methods, such as contrastive divergence, is less clear.[citation needed] Deep learning is often treated as a black box, and most results are confirmed through testing rather than theory. A series of images showing the inner states of deep neural networks drew wide interest, with the original research notice receiving well over 1,000 comments and, for a time, being the most accessed article on The Guardian's website. A study used Innovation Diffusion Theory (IDT) and Google Trends data to look at how deep learning spread in BRICS and OECD countries. Some systems also make errors, such as confidently labeling unrecognizable images as ordinary categories or misclassifying tiny changes to correctly classified images. Goertzel suggested these limits could make it hard to fit such systems into artificial general intelligence (AGI) designs, and that learning a grammar could help with commonsense reasoning based on production rules, a basic goal of artificial intelligence (AI). As deep learning moves into the real world, researchers have found that neural networks can be hacked or fooled through an "adversarial attack", in which inputs are changed so the system finds a match that humans would not see. Examples include doctored images, psychedelic spectacles that fooled a facial recognition system, stickers on stop signs, and sounds that made Google Now open a web address. One defense is reverse image search with a site such as TinEye. In "data poisoning", false data is slipped into a training set. Many training datasets are also created or labeled by people, often through low-paid clickwork such as on Amazon Mechanical Turk, or through microwork that people may not notice. The philosopher Rainer Mühlhoff describes five ways human work is captured to make training data: gamification, "trapping and tracking" such as CAPTCHAs or click-tracking on search results pages, using social motivations such as tagging faces on Facebook, information mining from quantified-self devices such as activity trackers, and clickwork.

Quiz time!
5 questions about Deep learning
What does the word "deep" in deep learning refer to?
Keep exploring
This article is a child-friendly adaptation of the Wikipedia article on Deep learning, available under CC BY-SA 4.0.
Images from Wikimedia Commons. Tap any image to view credits and license.