Spark Operator

Teams running repeated Spark jobs may want Kubernetes to manage their definitions and report execution status. Kubeflow Spark Operator watches application resources and submits the corresponding Spark work, with support for scheduled applications and lifecycle handling. It adds that management layer around Spark rather than replacing Spark’s processing engine. Controller retries can repeat work, so applications need suitable output and recovery behavior, and the operator’s version compatibility must be checked separately from the Spark runtime image.

Current guidance

The recovered upstream identity corresponds to the Kubeflow Spark Operator, which provides SparkApplication and scheduled application resources and runs spark-submit on their behalf. Current upstream documentation provides its own Helm repository, describes the project as beta and identifies the v1beta2 API. This is a specific operator lineage, not every project called a Spark operator.

Review the selected controller’s version matrix and application image separately. Controller retries and automatic resubmission can repeat external writes, so applications need an appropriate output and recovery design. Namespace watching, driver service accounts, admission webhook behavior and CRD ownership also affect a shared cluster beyond a single Spark job.

Before replacing an old installation, export custom resources and inspect stored API versions. Do not casually delete CRDs while following a historical migration example: custom-resource deletion can remove the specifications needed for recovery. Rehearse conversion and controller handover in isolation, then test scheduled execution, failure retry and cleanup with an idempotent job. Retain streaming checkpoints and application dependencies explicitly; upgrading the operator alone does not prove that a different Spark major version can resume them.