Databricks

Three strengths

• One place for data. Its lakehouse combines enterprise data warehouses and data lakes in a single platform. 1
• Built on open source. Its founders created Apache Spark, Delta Lake, MLflow and Unity Catalog, all open source projects. 2
• One set of permissions. Unity Catalog gives one set of permissions across the lakehouse and supports secure data sharing. 1

What it is

Most businesses keep information in many places: databases, spreadsheets, files and apps. To learn anything from it, someone has to bring it together, clean it up and make it easy to question. There are two older ways to do that. A data warehouse is a tidy, structured store that's fast for reports. A data lake is cheap storage for raw files of any kind, but it's harder to query. Databricks is a platform built around a middle path called a lakehouse, which "combines enterprise data warehouses and data lakes". 1

In a 2021 paper, researchers from Databricks, UC Berkeley and Stanford argued that the traditional data warehouse will wither and be replaced by the lakehouse. They described it as built on open data formats such as Apache Parquet, with first-class support for machine learning and data science, and strong performance. 3 Its authors include Databricks staff, so read it as the case for the approach from the people who build it.

The company was founded in 2013 by the original creators of the lakehouse architecture and of open source projects including Apache Spark, Delta Lake and MLflow. 2 The platform runs in your own cloud account, integrating with your cloud storage and security. 1 Day to day it covers:

• ETL, which means extract, transform, load: pulling data from its sources, reshaping it, and loading it where it can be used. It's built on Apache Spark and Delta, with logic written in SQL, Python or Scala. 1
• SQL and dashboards, so the people asking questions can query the data directly. 1
• Notebooks, shared documents where analysts mix code, charts and commentary. 1
• Machine learning and AI tooling, including MLflow. 1

It's in real use: 3.4% of developers surveyed in 2025 said they use Databricks SQL. 4

We choose it when the data is large or varied enough that a single database would struggle, and when a client wants the data and the analysis to live in one governed place.

How we use it

Databricks is our choice for larger or more varied data: many sources, mixed shapes, and volumes that outgrow an ordinary database. It does the heavy lifting alongside Spark, and Airflow schedules the jobs that feed it. For clients who run on Microsoft, we usually reach for Fabric and Power BI instead.

That's Data Solutions work. For you, it means data from separate systems arrives in one place and can be queried together, and Unity Catalog gives you one set of permissions to manage.

References

1. Databricks, "What is Databricks?", Databricks documentation. https://docs.databricks.com/aws/en/introduction/ (accessed 29 October 2026)
2. Databricks, "About us", databricks.com. https://www.databricks.com/company/about-us (accessed 29 October 2026)
3. Michael Armbrust, Ali Ghodsi, Reynold Xin and Matei Zaharia, "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics", 11th Conference on Innovative Data Systems Research (CIDR), 2021. https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf (accessed 29 October 2026)
4. Stack Overflow, "2025 Developer Survey: Technology", survey.stackoverflow.co, 2025. https://survey.stackoverflow.co/2025/technology (accessed 29 October 2026)