https://peertube.watch/videos/watch/655eaad7-9211-4471-9309-8427fb296b16 Teams of all shapes and sizes benefit from near real-time analytics. In this session, I will present a project template that can serve as the foundation to build one such high performing system, powered by Apache Kafka, QuestDB, Grafana OSS, and Jupyter Notebook. The first step of a data pipeline is ingestion, and even though we could directly ingest into a fast database, I will use Apache Kafka to ingest data. We will see how to use Python, JavaScript, and Go to send messages into Kafka. Now, we need an analytics database, and for real-time data, a time-series database seems like a good match. I will demonstrate how to use QuestDB, an Apache 2.0 licensed project, to ingest and query data in milliseconds or faster. Data analytics often require a graphical dashboard. For this purpose, I will use Grafana OSS, where we will create a couple of real-time charts updating several times per second. And, of course, it's 2024, so you might want to delve into some data science. No worries. I will demonstrate how Jupyter Notebook can be used to read from your database and perform interactive data exploration and time-series forecasting. This will be a demo-driven presentation and I will share all the code I use so you can use it as a starting point. Speaker: javier ramirez, Track: Main Track - Janson, Room: Janson, When: Saturday 3 February 2024, 16:00–16:50 (Brussels time), Schedule: FOSDEM 2024 event page, Slides & links pesentation (slides), Licence: Creative Commons Attribution 2.0 Belgium (CC BY 2.0 BE) (as stated in the FOSDEM video archive). Attribution: “Ingesting and analyzing millions of events per second in real-time using open source tools” by javier ramirez, FOSDEM 2024. Original recording: 2024/janson/fosdem-2024-2871-ingesting-and-analyzing-millions-of-events-per-second-in-real-time-using-open-source-tools.av1.webm from the FOSDEM video archive. This channel is an unofficial mirror; FOSDEM is a free and non-commercial event organised by the community for the community. Wed, 21 Oct 2026 23:05:33 GMT https://validator.w3.org/feed/docs/rss2.html PeerTube - https://peertube.watch https://peertube.watch/lazy-static/avatars/3b284627-b75f-409e-ac2e-10f4c2946f43.png https://peertube.watch/videos/watch/655eaad7-9211-4471-9309-8427fb296b16 All rights reserved, unless otherwise specified in the terms specified at https://peertube.watch/about and potential licenses granted by each content's rightholder.