A few days ago, I wrote about a new research paper from Google that has been causing a lot of debate in the machine learning(ML) and artificial intelligence(AI) communities. “One Model to Learn Them All” proposes an approach to combine several deep learning models in areas such as image recognition, natural language processing or speech analysis on a single algorithm that can solve problems across different domains. In my previous article, I mentioned that the principles behind Google’s model have been around for decades within the machine learning research circles. Specifically, University of Washington’s ML researcher Pedro Domingos has been an active proponent of the idea of a universal learning algorithm.
In his book “The Master Algorithm”, Domingos introduces the idea of a universal learner (humblingly named Alchemy ;) ) that combines the main algorithms of the top five ML schools:
— Symbolists: Symbolists focus on solving problems by manipulating symbols and replacing expression by other expressions. Their main algorithm is, typically, inverse deduction which generalizes knowledge by making predictions about it.
— Connectionists: Connectionists focus on solving problems by simulating brain functions. Their main algorithm is backpropagation which compares the results of a model against expectations and adjusts the structure of a neural network accordingly.
— Bayesians: Bayesian focus on solving uncertainty via probabilistic inference. Not surprisingly, their master algorithm is the Bayes Theorem.
— Evolutionaries: Evolutionaries focus on improving learning structures by inspecting knowledge. Their master algorithm is genetic programming.
— Analogizers: Analogizers improve learning by recognizing similarities. Their main algorithm is support vector machines which tracks and combines inputs to derive new predictions.
The core of Domingo’s master algorithm theory( and it is just that, a theory ;) ) is to combine the five ML leading schools using a cohesive model. Specifically, the master algorithm focuses on three main areas:
— Representation: This is the area in which a learner forms a representation of the knowledge related to a model. For instance, symbolists rely on logic to model knowledge while connectionists focus on neural networks.
— Evaluation: This is the area in which the accuracy of models is evaluated. For instance, connectionists use continuous error measures( ex: squared errors) to evaluate the distance between predicted results and true values. Bayesians rely on posterior probability for the same task.
— Optimization: This is the are in which the master algorithm selects the highest scoring among its model. In order to achieve that task, symbolists rely on inverse deduction while connectionists are likely to leverage gradient descent algorithms.
Combining ML algorithms from the main five ML schools across the representation, evaluation and optimization stages produces some very interesting multi-model algorithms. Markov Logic Networks(MLN) are a great example of this technique which combines logic and probability for knowledge representation ,making it one of the most flexible ML models for general learning.
Master algorithms such as Google’s MultiModel are likely to become essential in order to simulate human intelligence. Without a doubt, multi-model algorithms can be considered on eof the greatest challenges of the next decade of machine learning.