Spark Operator

This is the Helm chart for the Spark-on-Kubernetes Operator. Spark Operator is still under active development. Backward compatibility of the APIs is not guaranteed for alpha releases.

Customization of Spark pods, e.g., mounting arbitrary volumes and setting pod affinity, is currently experimental and implemented using a Kubernetes Mutating Admission Webhook, which became beta in Kubernetes 1.9. The mutating admission webhook is disabled by default but can be enabled if there are needs for pod customization.

Prerequisites

The Operator requires Kubernetes version 1.8 and above because it relies on garbage collection of custom resources. If customization of the driver and executor pods (through mounting custom configMaps and volumes) is desired, then the Mutating Admission Webhook needs to be enabled and it only became beta in Kubernetes 1.9.

Overview

Spark Operator aims to make specifying and running Spark applications as easy and idiomatic as running other workloads on Kubernetes. It uses Kubernetes custom resources for specifying, running, and surfacing status of Spark applications. For a complete reference of the custom resource definitions, please refer to the API Definition.It requires Spark 2.3 and above that supports Kubernetes as a native scheduler backend.

Spark Operator currently supports the following list of features:

• Supports Spark 2.3 and up.
• Enables declarative application specification and management of applications through custom resources.
• Automatically runs spark-submit on behalf of users for each SparkApplication eligible for submission.
• Provides native cron support for running scheduled applications.
• Supports customization of Spark pods beyond what Spark natively is able to go through the mutating admission webhook, e.g., mounting ConfigMaps and volumes, and setting pod affinity/anti-affinity.
• Supports automatic application re-submission for updated SparkAppliation objects with the updated specification.
• Supports automatic application restart with a configurable restart policy.
• Supports automatic retries of failed submissions with optional linear back-off.
• Supports mounting local Hadoop configuration as a Kubernetes ConfigMap automatically via sparkctl.
• Supports automatically staging local application dependencies to Google Cloud Storage (GCS) via sparkctl.
• Supports collecting and exporting application-level metrics and driver/executor metrics to Prometheus.

Motivations

This approach is completely different than the one that has the submission client creates a CRD object. Having externally created and managed CRD objects offer the following benefits:

• Things, like creating namespaces and setting up RBAC roles and resource quotas, represent a separate concern and are better done before applications get submitted.
• With the external CRD controller submitting applications on behalf of users, they don't need to deal with the submission process and the spark-submit command. Instead, the focus is shifted from thinking about commands to thinking about declarative YAML files describing Spark applications that can be easily version controlled.
• Externally created CRD objects make it easier to integrate Spark application submission and monitoring with users' existing pipelines and tooling on Kubernetes.
• Internally created CRD objects are good for capturing and communicating application/executor status to the users, but not for driver/executor pod configuration/customization as very likely it needs external input. Such external input most likely needs additional command-line options to get passed in.
• Additionally, keeping the CRD implementation outside the Spark repository gives us a lot of flexibility in terms of functionality to add to the CRD controller. We also have full control over code review and release process.