Apache Spark

Three strengths

• Built for speed. For iterative and interactive work, keeping data in memory can improve performance by an order of magnitude. 3
• Your choice of language. You can work with it in Python, SQL, Scala, Java or R. 1
• Batch and live data. It processes data in batches and as real-time streams, on one machine or across clusters of thousands. 1

What it is

When a business has more data than one computer can handle, the answer is to split the work across many computers and combine the results. A group of computers working together like this is called a cluster. Apache Spark is the engine that does the splitting: It's a multi-language engine for executing data engineering, data science and machine learning on single machines or clusters. 1

It was built to fix a weakness in what came before. Earlier frameworks such as MapReduce and Dryad let people write parallel computations without worrying about how work was shared out or what happened when a machine failed. But to reuse data between steps, they had to write it out to storage each time, which added overheads that could dominate the running time. 3

Spark's answer was a new idea called the resilient distributed dataset, which lets programs run computations in memory across a cluster and still recover if a machine fails. In the researchers' words, keeping data in memory "can improve performance by an order of magnitude" for iterative algorithms and interactive data exploration. 3

Spark started as a research project at the UC Berkeley AMPLab in 2009, was open sourced in early 2010, and moved to the Apache Software Foundation in 2013. 2 The project reports that thousands of companies, including 80% of the Fortune 500, use it. 1 Its SQL support runs standard ANSI SQL queries, so analysts can ask questions of huge datasets in a language they already know. 1

We choose it when the volume of data is more than a single database can comfortably process, and when the same job has to scale up as your data grows.

How we use it

Spark carries our bulk data work. When a Data Solutions project has to process large volumes of data in one go, Spark is the engine that does the heavy lifting. Airflow decides when those jobs run and in what order, and Databricks gives us a platform built around Spark for larger or more varied data.

That's Data Solutions work, and for you it means pipelines that keep up as your data grows. Where information needs to arrive the moment it's created, we use WebSockets and serverless event-driven pipelines instead.

References

1. The Apache Software Foundation, "Apache Spark: Unified engine for large-scale data analytics", spark.apache.org. https://spark.apache.org/ (accessed 29 October 2026)
2. The Apache Software Foundation, "Spark History", spark.apache.org. https://spark.apache.org/history.html (accessed 29 October 2026)
3. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker and Ion Stoica, "Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing", 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12), 2012. https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf (accessed 29 October 2026)