Title: Debugging Skynet: A Machine Learning Approach to Log Analysis

Description:

"We present a novel system for log forensics that is based on Machine Learning. The system enables classification of log entries in realtime and alerting in case of impending production failures. The system is designed for high scalability and realtime performance. In order to achieve this, a linear classification engine was trained on features extracted from user behavior, real world user queries and Community Q&A sites. The ML tools used in the system include principles from Deep Learning, as well as Support Vector Machines (SVM). The realtime engine was implemented using linear SVM and random forests. Training of these supervised algorithms was bootstrapped using human intelligence and several simple heuristics and was later refined by incorporating user feedback into the training stages. The system takes advantage of the BIg Data that is now available on the web and relies on modern scalable solutions such as Spark, EMR and Hadoop in order to achieve a fully elastic and scalable solution."

Bio:

Ianir Ideses heads the algorithms research and development at Logz.io. Prior to Logz.io, Ianir was Chief Scientist at Shine Security and Algorithm System Architect at Superfish. Ianir holds a Ph.D. from Tel-Aviv University and specializes in machine learning, computer vision, and image processing.