MLlib: Main Guide - Spark 3.5.3 Documentation - Apache Spark
MLlib is Spark’s machine learning (ML) library. Its goal is to make practical machine learning scalable and easy. At a high level, it provides tools such as: ML Algorithms: common learning algorithms such as classification, regression, clustering, and collaborative filtering.
ML Pipelines - Spark 3.5.3 Documentation - Apache Spark
In this section, we introduce the concept of ML Pipelines. ML Pipelines provide a uniform set of high-level APIs built on top of DataFrames that help users create and tune practical machine learning pipelines. Table of Contents. Main concepts in Pipelines. DataFrame.
MLlib - Apache Spark
MLlib is Apache Spark's scalable machine learning library. Ease of use. Usable in Java, Scala, Python, and R. MLlib fits into Spark 's APIs and interoperates with NumPy in Python (as of Spark 0.9) and R libraries (as of Spark 1.5).
Classification and regression - Spark 3.5.3 Documentation
The spark.ml implementation supports random forests for binary and multiclass classification and for regression, using both continuous and categorical features. For more information on the algorithm itself, please see the spark.mllib documentation on random forests.
Extracting, transforming and selecting features - Spark 3.5.3 …
Extracting, transforming and selecting features. This section covers algorithms for working with features, roughly divided into these groups: Extraction: Extracting features from “raw” data. Transformation: Scaling, converting, or modifying features. Selection: Selecting a subset from a larger set of features.
Clustering - Spark 3.5.3 Documentation - Apache Spark
From the abstract: PIC finds a very low-dimensional embedding of a dataset using truncated power iteration on a normalized pair-wise similarity matrix of the data. spark.ml ’s PowerIterationClustering implementation takes the following parameters: k: …
Collaborative Filtering - Spark 3.5.3 Documentation - Apache Spark
Explicit vs. implicit feedback. The standard approach to matrix factorization-based collaborative filtering treats the entries in the user-item matrix as explicit preferences given by the user to the item, for example, users giving ratings to movies.
ML Tuning - Spark 3.5.3 Documentation - Apache Spark
This section describes how to use MLlib’s tooling for tuning ML algorithms and Pipelines. Built-in Cross-Validation and other tooling allow users to optimize hyperparameters in algorithms and Pipelines. Table of contents. Model selection (a.k.a. hyperparameter tuning) Cross-Validation. Train-Validation Split.
BinaryClassificationEvaluator — PySpark 3.5.3 documentation
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra. Parameters. extradict, optional.
Linear Methods - RDD-based API - Spark 3.5.3 Documentation
Under the hood, linear methods use convex optimization methods to optimize the objective functions. spark.mllib uses two methods, SGD and L-BFGS, described in the optimization section. Currently, most algorithm APIs support Stochastic Gradient Descent (SGD), …