Question
You have an Azure Databricks workspace.
You are creating a Lakeflow Spark Declarative Pipelines (SDP) pipeline that scales automatically.
You need to configure compute for the pipeline. The solution must minimize operational costs and effort.
What should you use?
- A. the existing SQL warehouse
- B. an all-purpose cluster that uses autoscaling
- C. a job cluster that uses autoscaling Correct
- D. a single-node, all-purpose cluster
Explanation
Pipeline compute is created for the update and shut down automatically when it is not in use, which is the job-style lifecycle that keeps cost and effort low, and autoscaling adds and removes workers as the workload changes. A SQL warehouse serves SQL queries and is not the compute that runs a pipeline update. An all-purpose cluster is meant for interactive work and keeps costing money between runs. A single node cannot scale at all, so it fails the main requirement.
