https://peertube.watch/videos/watch/e19bf51d-acbf-437e-ba7d-07669ea03f5d with Apache Spark and Apache Beam As distributed data parallel systems, like Spark, are used for more mission-critical tasks, it is important to have effective tools for testing and validation. This talk explores the general considerations and challenges of testing systems like Spark through spark-testing-base and other related libraries. With over 40% of folks automatically deploying the results of their Spark jobs to production, testing is especially important. Many of the tools for working with big data systems (like notebooks) are great for exploratory work, and can give a false sense of security (as well as additional excuses not to test). This talk explores why testing these systems are hard, special considerations for simulating "bad" partioning, figuring out when your stream tests are stopped, and solutions to these challenges. Speaker: Holden Karau, Track: Testing and automation, Room: H.2213, When: Saturday 3 February 2018, 15:05–15:50 (Brussels time), Schedule: FOSDEM 2018 event page, Slides & links Slides one slideshare, Talk feedback to Holden, Licence: Creative Commons Attribution 2.0 Belgium (CC BY 2.0 BE) (as stated in the FOSDEM video archive). Attribution: “Testing and Validating distributed systems” by Holden Karau, FOSDEM 2018. Original recording: 2018/H.2213/validating_spark.webm from the FOSDEM video archive. This channel is an unofficial mirror; FOSDEM is a free and non-commercial event organised by the community for the community. Mon, 19 Oct 2026 21:50:21 GMT https://validator.w3.org/feed/docs/rss2.html PeerTube - https://peertube.watch https://peertube.watch/lazy-static/avatars/3b284627-b75f-409e-ac2e-10f4c2946f43.png https://peertube.watch/videos/watch/e19bf51d-acbf-437e-ba7d-07669ea03f5d All rights reserved, unless otherwise specified in the terms specified at https://peertube.watch/about and potential licenses granted by each content's rightholder.