Spark history server

When a Spark application finishes, its live driver interface may disappear just when someone needs to investigate the run. Spark History Server reads saved event logs and reconstructs a web view of that application’s history. It helps teams inspect completed work, provided the jobs wrote logs and the server can read them. It does not submit or schedule jobs. Event-log locations, storage credentials and retention are therefore central to making the interface useful after execution ends.

Current guidance

It does not schedule jobs or replace the live driver UI. The archived stable/spark-history-server chart named Spark 2.4.0 and a separate Lightbend container source, so the chart, packaged image and Apache Spark project must be reviewed independently.

The Spark monitoring guide explains that applications must enable event logging and write to storage that the history server can read. Match the application event-log location and the server’s history-log directory. For object storage, confirm the required filesystem libraries and credentials are present in the chosen image; an empty history page can indicate missing logs or failed storage access rather than a failed Spark job.

Plan event-log retention, replay memory and storage costs, especially for long-running applications. Follow the security documentation for event logs and history-server access: these records may expose job details and configuration. Test a completed job, an incomplete attempt and a large rolling log through the actual ingress path. Pin a compatible server image and verify the old chart’s APIs before reuse. No Spark job or object-store integration was executed in this resource review.