The implementation of linear regression in Spark ML first attempts to solve the linear regression problem using matrix decomposition, but this method does not scale well to large datasets with a large number of variables.
Which of the following approaches does Spark ML use to distribute the training of a linear regression model for large data?

1. A. Logistic regression
2. B. Spark ML cannot distribute linear regression training
3. C. Iterative optimization
4. D. Least-squares method
5. E. Singular value decomposition

Correct Answer: C
For large datasets with many variables, Spark ML distributes the training of a linear regression model using iterative optimization methods. Specifically, Spark ML employs algorithms such as Gradient Descent or L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno) to iteratively minimize the loss function. These iterative methods are suitable for distributed computing environments and can handle
large-scale data efficiently by partitioning the data across nodes in a cluster and performing parallel updates.References:
✑ Spark MLlib Documentation (Linear Regression with Iterative Optimization).

Which of the following describes the relationship between native Spark DataFrames and pandas API on Spark DataFrames?

1. A. pandas API on Spark DataFrames are single-node versions of Spark DataFrames withadditional metadata
2. B. pandas API on Spark DataFrames are more performant than Spark DataFrames
3. C. pandas API on Spark DataFrames are made up of Spark DataFrames and additional metadata
4. D. pandas API on Spark DataFrames are less mutable versions of Spark DataFrames

Correct Answer: C
The pandas API on Spark DataFrames are made up of Spark DataFrames with additional metadata. The pandas API on Spark aims to provide the pandas-like experience with the scalability and distributed nature of Spark. It allows users to work with pandas functions on large datasets by leveraging Spark??s underlying capabilities. References:
✑ Databricks documentation on pandas API on Spark: pandas API on Spark

A data scientist has defined a Pandas UDF function predict to parallelize the inference process for a single-node model:

They have written the following incomplete code block to use predict to score each record of Spark DataFramespark_df:

Which of the following lines of code can be used to complete the code block to successfully complete the task?

1. A. predict(*spark_df.columns)
2. B. mapInPandas(predict)
3. C. predict(Iterator(spark_df))
4. D. mapInPandas(predict(spark_df.columns))
5. E. predict(spark_df.columns)

Correct Answer: B
To apply the Pandas UDFpredictto each record of a Spark DataFrame, you use themapInPandasmethod. This method allows the Pandas UDF to operate on partitions of the DataFrame as pandas DataFrames, applying the specified function (predictin this case) to each partition. The correct code completion to execute this is simply mapInPandas(predict), which specifies the UDF to use without additional arguments orincorrect function calls.References:
✑ PySpark DataFrame documentation (Using mapInPandas with UDFs).

A data scientist has produced three new models for a single machine learning problem. In the past, the solution used just one model. All four models have nearly the same prediction latency, but a machine learning engineer suggests that the new solution will be less time efficient during inference.
In which situation will the machine learning engineer be correct?

1. A. When the new solution requires if-else logic determining which model to use to compute each prediction
2. B. When the new solution's models have an average latency that is larger than the size of the original model
3. C. When the new solution requires the use of fewer feature variables than the original model
4. D. When the new solution requires that each model computes a prediction for every record
5. E. When the new solution's models have an average size that is larger than the size of the original model

Correct Answer: D
If the new solution requires that each of the three models computes a prediction for every record, the time efficiency during inference will be reduced. This is because the inference process now involves running multiple models instead of a single model, thereby increasing the overall computation time for each record.
In scenarios where inference must be done by multiple models for each record, the latency accumulates, making the process less time efficient compared to using a single model. References:
✑ Model Ensemble Techniques

A machine learning engineer would like to develop a linear regression model with Spark ML to predict the price of a hotel room. They are using the Spark DataFrametrain_dfto train the model.
The Spark DataFrametrain_dfhas the following schema:

The machine learning engineer shares the following code block:

Which of the following changes does the machine learning engineer need to make to complete the task?

1. A. They need to call the transform method on train df
2. B. They need to convert the features column to be a vector
3. C. They do not need to make any changes
4. D. They need to utilize a Pipeline to fit the model
5. E. They need to split thefeaturescolumn out into one column for each feature

Correct Answer: B
In Spark ML, the linear regression model expects the feature column to be a vector type. However, if thefeaturescolumn in the DataFrametrain_dfis not already in this format (such as being a column of type UDT or a non-vectorized type), the engineerneeds to convert it to a vector column using a transformer likeVectorAssembler. This is a critical step in preparing the data for modeling as Spark ML models require input features to be combined into a single vector column.
References
✑ Spark MLlib documentation forLinearRegression:https://spark.apache.org/docs/latest/ml-classification- regression.html#linear-regression