Building deep learning solutions in the real world is a journey full of challenges. Most of us are familiar with the difficulties finding sophisticated talent with enough understanding of deep learning algorithms and technologies. However, the challenges of implementing deep learning solutions expand way beyond the deficit technical talent.
During the implementation of a real world deep learning program, technologists are constantly faced with roadblocks in areas such as training, optimization, model capacity management, testing and many other unconventional aspects that don’t appear natural to mainstream developers. Among the areas that require a careful strategy during the implementation of deep learning systems, debugging stands out on own as one of the challenges capable of driving the most skillful technologists absolutely crazy.
What Makes Deep Learning Debugging So Challenging?
In one word: unpredictability. When we write static code for a web or mobile application we know the expected behavior of every algorithms as this one is determined by the code itself. In the case of deep learning applications, the behavior of the algorithm is determined by the knowledge built during training instead of by static code. For instance, wif we train a convolutional neural network(CNN) and it achieves 3% error rate, its practically impossible to understand whether the behavior has been optimal or not without a deep analysis.
To the unpredictability challenges, we should add the fact that most deep learning models produced neural networks with hundreds of thousands or millions of nodes and many hidden units which makes debugging nothing short of a nightmare. Not surprisingly, despite great tools such as TensorFlow’s TensorBoard, the tools and frameworks for debugging deep learning models are still in their infancy. So what do we do in order to effectively debug deep learning programs? There are a few tips that might help.
Practical Tips for Debugging Deep Learning Programs
1 — Visualize the Network and its Results
A pretty obvious point. When building a deep learning application, it is imperative to leverage tools that can help to visualize the connected graph and the results of the model based on certain inputs. This will give developers a visually intuitive way to reason through the model and try to understand the behavior of the algorithms.
2 — Analyze Training and Test Errors
The training and test errors in a deep learning model can offer helpful clues about potential problems before they occur. For instance, if a model is overfitting( test error is high) but the training error remains low, then is likely that there are errors in the algorithm. However, if the training error is high then the model is underfitting and we are likely to find an error in the training procedure.
3 — Test with Small Datasets
Building on the previous point; if a model is underfitting, we need to determine whether is a code or data defect. A way to achieve that is to test with a very small number of examples. If the model fails, then it is most likely due to issues with the code.
4 — Monitor Activations and Gradient Values
Keeping an eye on the activations of hidden units and the values of the gradients are essential measures to optimize a deep learning model. The number of node activations are an important metric to understand if a neural network is saturated. Similarly, getting a histogram view of the value of the gradients is a super helpful technique to understand the potential for future optimizations in the model.