Free Apache-Spark-Developer sample questions

Correct answer: A broadcast variable

A broadcast variable is the correct choice for distributing a large, read-only dataset to all worker nodes. The driver serializes the variable and sends it to each executor only once, where it is cached in memory. Tasks on that executor can then access the data locally without causing repeated network transfers. Accumulators are used for aggregating results back to the driver. Caching the DataFrame would work, but broadcast is specifically designed for this 'side data' distribution pattern and is generally more efficient for joins. A UDF is for custom logic, not data distribution.

Correct answer: A wide transformation that requires a shuffle of data between executors.

Spark creates a new stage at each shuffle boundary. Wide transformations, such as `groupBy()`, `join()`, or `repartition()`, require data to be redistributed (shuffled) across the network between executors. This redistribution marks the end of one stage and the beginning of another. Narrow transformations can be pipelined within a single stage. Actions trigger the execution of a job, which consists of one or more stages, but the action itself doesn't define the stage boundary; the shuffle does.

Correct answers: Cause: Data skew on the `user_id` key. Solution: Implement salting by adding a random key to the skewed `user_id` values before grouping.; Cause: Data skew on the `user_id` key. Solution: Enable Adaptive Query Execution (AQE) and configure `spark.sql.adaptive.skewJoin.enabled`.

Correct answer: `year = 2023 AND month = 1 AND day IN (1, 2, 3, 4, 5, 6, 7)`

For partition pruning to be effective, the filter conditions must be applied directly to the partition columns. This allows Spark's Catalyst optimizer to read the file system metadata and skip reading the data files in partitions that do not match the filter. Applying functions to the partition columns (like `concat` or `to_date`) can prevent the optimizer from pushing down the predicate, forcing a full table scan. The correct approach is to filter directly on the raw partition columns.

Correct answer: Update Mode

Update mode is specifically designed for use cases involving aggregations. In this mode, only the rows that were updated in the result table since the last trigger will be written to the sink. This is perfect for outputting running counts where only the users with new events in the current micro-batch will have their counts updated in the output. Complete mode would rewrite the entire result table, and Append mode is not supported for aggregations without a watermark.

Correct answer: 1. `enriched_df = sales_df.join(products_df, 'product_id', 'left_outer')` 2. `aggregated_df = enriched_df.groupBy('product_category').agg(sum('sale_amount'))` 3. `aggregated_df.write.format('delta').mode('overwrite').partitionBy('product_category').save(path)`

This is the correct and most efficient sequence. First, the sales data is joined with the product data to add the `product_category`. A left outer join is appropriate to ensure no sales are dropped if a product is missing from the dimension table. Second, the enriched data is grouped by the `product_category` to calculate the sum of sales. Performing the join before aggregation is crucial to have the category available. Finally, the aggregated result is written to a Delta table, correctly using `overwrite` mode and partitioning by `product_category` for query performance.

Correct answer: True

This statement is true. The primary purpose of Spark Connect is to decouple the client application (where the SparkSession is created and DataFrame logic is defined) from the Spark driver and cluster. It introduces a client-server architecture where the client sends unresolved logical plans to a Spark Connect server running on the cluster, which then translates them into Spark's physical plan for execution on the executors.

Correct answer: A Pandas UDF (Vectorized UDF)

A Pandas UDF (also known as a vectorized UDF) is the best choice. It processes data in batches as pandas Series or DataFrames, which allows for efficient, vectorized computations using libraries like NumPy or SciPy. This avoids the high overhead of serialization/deserialization and row-by-row processing that occurs with standard Python UDFs, leading to significant performance improvements.

Correct answer: It is a process that runs on one of the worker nodes inside the cluster, maintains the SparkSession, and coordinates task execution.

In cluster mode, the driver program is launched on a worker node within the cluster. Its main responsibilities are to host the `main()` method of the application, create the SparkSession, analyze and schedule jobs, and coordinate the execution of tasks on the executors. The executors are the processes that actually run the computation tasks.

Correct answer: SaveMode.ErrorIfExists

`SaveMode.ErrorIfExists` is the correct option. It is the default behavior. If the target location already exists, a `AnalysisException` is thrown. `Append` adds data, `Overwrite` replaces it, and `Ignore` silently does nothing if the location exists.