Artwork

内容由Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®提供。所有播客内容(包括剧集、图形和播客描述)均由 Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® 或其播客平台合作伙伴直接上传和提供。如果您认为有人在未经您许可的情况下使用您的受版权保护的作品,您可以按照此处概述的流程进行操作https://zh.player.fm/legal
Player FM -播客应用
使用Player FM应用程序离线!

Real-time Threat Detection Using Machine Learning and Apache Kafka

29:18
 
分享
 

Manage episode 348328528 series 2355972
内容由Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®提供。所有播客内容(包括剧集、图形和播客描述)均由 Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® 或其播客平台合作伙伴直接上传和提供。如果您认为有人在未经您许可的情况下使用您的受版权保护的作品,您可以按照此处概述的流程进行操作https://zh.player.fm/legal

Can we use machine learning to detect security threats in real-time? As organizations increasingly rely on distributed systems, it is becoming more important to analyze the traffic that passes through those systems quickly. Confluent Hackathon ’22 finalist, Géraud Dugé de Bernonville (Data Consultant, Zenika Bordeaux), shares how his team used TensorFlow (machine learning) and Neo4j (graph database) to analyze and detect network traffic data in real-time. What started as a research and development exercise turned into ZIEM, a full-blown internal project using ksqlDB to manipulate, export, and visualize data from Apache Kafka®.
Géraud and his team noticed that large amounts of data passed through their network, and they were curious to see if they could detect threats as they happened. As a hackathon project, they built ZIEM, a network mapping and intrusion detection platform that quickly generates network diagrams. Using Kafka, the system captures network packets, processes the data in ksqlDB, and uses a Neo4j Sink Connector to send it to a Neo4j instance. Using the Neo4j browser, users can see instant network diagrams showing who's on the network, allowing them to detect anomalies quickly in real time.
The Ziem project was initially conceived as an experiment to explore the potential of using Kafka for data processing and manipulation. However, it soon became apparent that there was great potential for broader applications (banking, security, etc.). As a result, the focus shifted to developing a tool for exporting data from Kafka, which is helpful in transforming data for deeper analysis, moving it from one database to another, or creating powerful visualizations.
Géraud goes on to talk about how the success of this project has helped them better understand the potential of using Kafka for data processing. Zenika plans to continue working to build a pipeline that can handle more robust visualizations, expose more learning opportunities, and detect patterns.
EPISODE LINKS

  continue reading

章节

1. Intro (00:00:00)

2. What is the Ziem Project? (00:05:03)

3. How do you use ksqlDB? (00:12:15)

4. Creating network visualizations with Neo4j and Neovis.js (00:14:00)

5. Machine learning plans with Ziem (00:17:50)

6. Supervised vs. non-supervised machine learning (00:20:07)

7. Future use cases for Ziem (00:22:30)

8. How to get started with TensorFlow (00:24:29)

9. It's a wrap! (00:27:33)

265集单集

Artwork
icon分享
 
Manage episode 348328528 series 2355972
内容由Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®提供。所有播客内容(包括剧集、图形和播客描述)均由 Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® 或其播客平台合作伙伴直接上传和提供。如果您认为有人在未经您许可的情况下使用您的受版权保护的作品,您可以按照此处概述的流程进行操作https://zh.player.fm/legal

Can we use machine learning to detect security threats in real-time? As organizations increasingly rely on distributed systems, it is becoming more important to analyze the traffic that passes through those systems quickly. Confluent Hackathon ’22 finalist, Géraud Dugé de Bernonville (Data Consultant, Zenika Bordeaux), shares how his team used TensorFlow (machine learning) and Neo4j (graph database) to analyze and detect network traffic data in real-time. What started as a research and development exercise turned into ZIEM, a full-blown internal project using ksqlDB to manipulate, export, and visualize data from Apache Kafka®.
Géraud and his team noticed that large amounts of data passed through their network, and they were curious to see if they could detect threats as they happened. As a hackathon project, they built ZIEM, a network mapping and intrusion detection platform that quickly generates network diagrams. Using Kafka, the system captures network packets, processes the data in ksqlDB, and uses a Neo4j Sink Connector to send it to a Neo4j instance. Using the Neo4j browser, users can see instant network diagrams showing who's on the network, allowing them to detect anomalies quickly in real time.
The Ziem project was initially conceived as an experiment to explore the potential of using Kafka for data processing and manipulation. However, it soon became apparent that there was great potential for broader applications (banking, security, etc.). As a result, the focus shifted to developing a tool for exporting data from Kafka, which is helpful in transforming data for deeper analysis, moving it from one database to another, or creating powerful visualizations.
Géraud goes on to talk about how the success of this project has helped them better understand the potential of using Kafka for data processing. Zenika plans to continue working to build a pipeline that can handle more robust visualizations, expose more learning opportunities, and detect patterns.
EPISODE LINKS

  continue reading

章节

1. Intro (00:00:00)

2. What is the Ziem Project? (00:05:03)

3. How do you use ksqlDB? (00:12:15)

4. Creating network visualizations with Neo4j and Neovis.js (00:14:00)

5. Machine learning plans with Ziem (00:17:50)

6. Supervised vs. non-supervised machine learning (00:20:07)

7. Future use cases for Ziem (00:22:30)

8. How to get started with TensorFlow (00:24:29)

9. It's a wrap! (00:27:33)

265集单集

すべてのエピソード

×
 
Loading …

欢迎使用Player FM

Player FM正在网上搜索高质量的播客,以便您现在享受。它是最好的播客应用程序,适用于安卓、iPhone和网络。注册以跨设备同步订阅。

 

快速参考指南