Question
You have an Azure Databricks workspace.
You are creating a Lakeflow Spark Declarative Pipelines (SDP) pipeline that scales automatically.
You need to configure compute for the pipeline. The solution must minimize operational costs and administrative effort.
What should you use?
- A. serverless compute Correct
- B. a single-node, all-purpose cluster
- C. an all-purpose cluster that uses autoscaling
- D. an existing SQL warehouse
- E. a job cluster that uses autoscaling
Explanation
Serverless pipelines run on Databricks-managed compute, which removes most infrastructure configuration, and Databricks recommends serverless for new pipelines. They add vertical autoscaling, choosing the most cost-efficient instance types, on top of horizontal autoscaling. An all-purpose or job cluster with autoscaling still leaves you to size and maintain the compute, and a single-node cluster cannot scale out. A SQL warehouse is compute for SQL queries, not a compute option you assign to run a pipeline.
