A Neural Network Told Me to Do it: Behavior Analysis for Machine Intelligence Part II

Last week I wrote about the great potential for technologies that analyze and understand the behavior of machine intelligence(MI) programs. Borrowing some concepts from cognitive psychology, we introduced the concept of machine intelligence behavior analysis(MIBA). Today, I would like to deep dive into some of the key elements, I believe, are necessary for the adoption of MIBA solutions.

From a conceptual standpoint, MIBA focuses on understanding and analyzing the decision many dynamics of MI models like neural networks. Beyond the many benefits of understanding how MI models build knowledge and make decisions, MIBA capabilities are essential to create the correct regulatory environment for MI technologies claimed by many (including Tesla-SpaceX CEO Elon Musk) to be paramount to our survival in the era of super intelligence. Not surprisingly, as explained in the first part of this essay, MIBA represents a great opportunity for startups and venture capitalists looking to power the next generation of MI technologies. So expectations are high; but what are really the relevant features that we need in the first group of MIBA solutions? I’ve been thinking about that question for a while and I’ve compiled a few ideas that might be worth discussing.

1 — What Did You Do? MIBA via Model Monitoring

We need a Google Analytics for machine/deep learning models. The increasing fragmentation in the MI platform space makes it extremely hard to effectively monitor MI models across different frameworks. A consistent platform for instrumenting, visualizing and analyzing the behavior of MI models will go a long way helping us understand the behavior of intelligent applications.

2 — How Did You Do That? MIBA via Interrogation

Asking questions is a natural vehicle used by humans in order to understand and reason about a particular subject. Similarly, MIBA techniques can benefit from interfaces that allow human experts or application to interrogate an MI model in order to better understand its behavior.

3 — Please Explain That To Me: MIBA via Knowledge Abstraction and Visualization

MI knowledge representations such as neural networks or Markov Chains are fairly complex structures which can be hard to navigate and understand. In order to streamline MIBA capabilities, I believe we need tools that can extrapolate MI knowledge structures onto representations that use elements such as visualizations or natural language narratives which can be easily understood by human experts.

4 — You Should Not Do That: MIA via Ethics and Regulation

Once we are able to understand the behavior of MI agents, we can start creating regulatory and ethic frameworks that constraint and influence the actions of MI programs. Instead of relying heavily on human supervision like we do today, I believe the new MIBA regulatory technologies will be embedded in MI models to guide their behavior and decisions.

5 — How Much do You Know? MIBA via Automated Testing

Just like modern institution today, we will need MIBA tools that regularly evaluate MI agents to detect specific behavioral profiles. From that perspective, MIBA tools might become the psychologist of MI applications :). In that context, MIBA tools would interact with MI models looking for well-established behavioral patterns and, if necessary, improve their behavior with new knowledge and logic.

CEO of IntoTheBlock, Chief Scientist at Invector Labs, I write The Sequence Newsletter, Guest lecturer at Columbia University, Angel Investor, Author, Speaker.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store