https://peertube.watch/videos/watch/806b3c95-5c52-4672-a8b7-a773cbacf3e2 With Spark 4 in the pipeline for this year, many of us are looking at what will be involved in upgrading to the latest and greatest Spark. This talk will look at the open-source tooling we use to automate upgrading thousands of our Spark pipelines (from Spark 2.X -> 3.4) and how to we used a variation of the write-audit-publish technique to validate the new pipelines in production for pipelines that might have less testing than ideal. Seeing is a pre-requisite to believing, so the talk will include a short demo showing how the spark-upgrade tool works on a demo pipeline complete with "live" validation. In this talk, you will learn how to: Upgrade your Spark pipelines without crying* Validating Spark (and other similar) pipelines even when you don't trust the tests (by extending the write-audit-publish pattern) on top of Iceberg, Hudi, or Delta Lake. Time permitting, I will end with some exciting new (but non-backward compatible) changes coming in Spark 4 (tentatively scheduled for June, but it's software). *Not a guarantee, some upgrades may still cause tears Related links: https://github.com/holdenk/spark-upgrade - the upgrade tool being discussed https://scalacenter.github.io/scalafix/ - the code mod tool we used for scala https://pybowler.io/ - the python code refactoring tool we extended https://sqlfluff.com/ - the sql linting / cleanup tool we extended Speaker: Holden Karau, Track: HPC, Big Data & Data Science devroom, Room: UA2.118 (Henriot), When: Saturday 3 February 2024, 15:00–15:30 (Brussels time), Schedule: FOSDEM 2024 event page, Slides & links Slides, Licence: Creative Commons Attribution 2.0 Belgium (CC BY 2.0 BE) (as stated in the FOSDEM video archive). Attribution: “Automating Spark (and Pipeline) Upgrades While "Testing" in Production” by Holden Karau, FOSDEM 2024. Original recording: 2024/ua2118/fosdem-2024-1970-automating-spark-and-pipeline-upgrades-while-testing-in-production.av1.webm from the FOSDEM video archive. This channel is an unofficial mirror; FOSDEM is a free and non-commercial event organised by the community for the community. Tue, 20 Oct 2026 22:13:19 GMT https://validator.w3.org/feed/docs/rss2.html PeerTube - https://peertube.watch https://peertube.watch/lazy-static/avatars/3b284627-b75f-409e-ac2e-10f4c2946f43.png https://peertube.watch/videos/watch/806b3c95-5c52-4672-a8b7-a773cbacf3e2 All rights reserved, unless otherwise specified in the terms specified at https://peertube.watch/about and potential licenses granted by each content's rightholder.