Connectionists and Symbolists: Learning Rules From Noisy Data

In his book “The Master Algorithm”, artificial intelligence researcher Pedro Domingos explores the idea of a single algorithm that can combine the major schools of machine learning. The idea is, without a doubt, extremely ambitious but we are already seeing some iterations of it. Last year, Google published a research paper under the catchy title of “One Model to Learn Them All” that combines heterogeneous learning techniques under a single machine learning model. This week, Alphabet’s subsidiary DeepMind took another step towards multi-model algorithms by introducing a new technique called Differentiable Inductive Logic Programming(DILP) that combines logic and neural networks into a single model to extract rules from noisy data.

DILP brings together two of the major machine learning schools. Connectionists try to model knowledge by imitating representations of the brain in the form of neural networks and have been the driving force behind movements such as deep learning. Symbolists rely on logic to model knowledge based on well-understood rules. Both schools have well known advantages and drawbacks. Symbolist systems based on inductive logic programming(ILP) tend to generalize knowledge efficiently and they are semi-immune to overfitting. Also, ILP systems tend to be a great fit in transfer learning scenarios in which a trained model can be copied and reused in other models. The main limitation of ILP systems is their struggle with noisy or ambiguous data which is so common in deep learning scenarios.

Connectionist systems tend to work well in environments with noisy data and can efficiently handle uncertainty and ambiguity. However, they tend to be expensive to train and version. Also, the knowledge learned from connectionist system is very hard to follow and understand which contrasts with the clarity of symbolist model. For years, many experts have highlighted the theoretical value on combining robust connectionist learning with symbolic relational learning. DILP is certainly a step on the right direction.

Conceptually, DILP combines neural networks with ILP to provide a model that can process noisy and ambiguous data while also generalizing well and avoiding deterioration. By combining the best of both worlds, DILP is a technique that differs from connectionist models in the sense that can generalize knowledge symbolically while also differing from traditional symbolist models by generalizing knowledge visually. The following matrix might help to illustrate the comparison between the three schools of thought.

DILP is very creative approach to bring together two of the major tribes in machine learning. Combining the intuitive knowledge of connectionist systems with the conceptual knowledge of symbolists is a step closer to emulate human cognition and, maybe, a step closer to Domingo’s master algorithm.

CEO of IntoTheBlock, Chief Scientist at Invector Labs, I write The Sequence Newsletter, Guest lecturer at Columbia University, Angel Investor, Author, Speaker.