Free DATA-ENG-ASSOC sample questions

Correct answer: The major and minor versions of the local Python environment do not exactly match the cluster's Python version.

Databricks Connect requires that the major and minor Python versions of the local client environment match the version on the Databricks cluster exactly. In this case, the local version is 3.11.5 while the cluster version is 3.11.2. Even though the major version (3) and minor version (11) match, the strict requirement often extends to the patch version for full compatibility, but the major/minor mismatch is the key principle. An expired PAT would result in an authentication error, not a `Py4JError`. A firewall issue would likely cause a timeout. An incorrect cluster ID would result in a 'not found' error.

Correct answers: The `targets` section to define distinct profiles for each environment (dev, staging, prod).; The `variables` section within each target to substitute values like cluster IDs or secret scopes.

The `targets` section is specifically designed to manage environment-specific configurations. Each target can have its own workspace URL, root path, and variable definitions, allowing the same bundle to be deployed to different environments with the correct settings.

Variables, defined within each target, allow for parameterization of the bundle's resources. This is how you would specify a different cluster policy ID for production versus development, or reference a different secret scope for database credentials in each environment.

Correct answer: `.option("cloudFiles.schemaEvolutionMode", "rescue")`

The `rescue` mode for `cloudFiles.schemaEvolutionMode` is designed for this exact scenario. It instructs Auto Loader to infer the schema and, if a data type change is detected (like numeric to string), it adds the new column to the `_rescue` column. For schema evolution, the correct option is to enable schema evolution on the write stream using `.option("mergeSchema", "true")` and handle rescued data. However, among the given choices, `rescue` mode is the feature specifically designed to handle columns with mixed data types by rescuing them, which aligns with the observed behavior. While `mergeSchema` is also needed, `schemaEvolutionMode` `rescue` directly addresses the problem of incompatible data types appearing in a column.

Correct answer: Do not grant any privileges on the Bronze tables to any users or groups, but grant `SELECT` and `MODIFY` on the specific Bronze tables to the service principal.

Unity Catalog operates on a default-deny model. To meet the requirement, you should avoid granting any broad permissions at the catalog or schema level. The correct approach is to grant the necessary, specific privileges (`SELECT` to read, `MODIFY` might be needed for the job's operations) directly to the service principal on the target Bronze tables. This ensures that only that principal can access the sensitive data, enforcing the principle of least privilege.

Correct answer: True

This statement is true. The primary purpose of cluster pools is to reduce cluster start and auto-scaling times by maintaining a set of idle, ready-to-use instances. When a job requests a cluster attached to a pool, it gets its nodes from this warm pool instead of requesting new instances from the cloud provider, which significantly shortens the startup latency.

Correct answer: The code fails to apply any data cleansing, validation, or schema enforcement rules before writing to the Silver layer.

The primary purpose of the Silver layer is to store cleansed, validated, and enriched data. This code snippet moves data directly from the source (Bronze) to Silver using a blind overwrite without any intermediate transformation steps. This violates the core principle of the Medallion architecture, as it bypasses the crucial data quality and shaping processes that should occur between the Bronze and Silver layers.

Correct answer: `@dlt.table(comment="Enriched data")` `@dlt.expect_or_drop("valid_key", "join_key IS NOT NULL")`

To create a physical table that can be queried later, `@dlt.table` is the correct decorator. To enforce a quality rule where invalid records are dropped while allowing the pipeline to continue, `@dlt.expect_or_drop` is the appropriate function. `@dlt.expect_or_fail` would halt the pipeline, and `@dlt.expect` would only record the violation statistics without dropping or failing.

Correct answer: Delta Sharing open sharing protocol

Delta Sharing is the open protocol specifically designed for sharing live data from Databricks to any external consumer, regardless of their platform. It works by creating shares and recipients, then providing the recipient with a secure, one-time activation link to download a credential file. This allows them to access the shared data using various connectors (like Power BI, Tableau, pandas) without needing a Databricks account. Lakehouse Federation is for querying external data sources from within Databricks, not sharing data out.

Correct answer: A serverless SQL warehouse.

Serverless SQL warehouses are ideal for this use case. They provide instant compute, automatically scale up and down to handle concurrent query loads, and eliminate the need for administrators to manage cluster configurations. This directly addresses the requirements for performance on variable workloads and reduced management overhead. A job cluster is for automated jobs, and an all-purpose cluster requires manual management and is less efficient for handling concurrent SQL queries.

Correct answer: A `merge` operation with `whenMatchedUpdate` and `whenNotMatchedInsert` clauses.

The `merge` operation is specifically designed for this type of 'upsert' (update or insert) scenario. It allows you to define conditions for matching records between a source DataFrame and a target Delta table. The `whenMatchedUpdate` clause specifies the update logic for existing records, while the `whenNotMatchedInsert` clause handles the insertion of new records, all within a single, atomic transaction.