[SPARK-53156][CORE] Track Driver Memory Metrics when the Application ends

### What changes were proposed in this pull request?
Report a heartbeat on the driver when the application stops.

### Why are the changes needed?
When the application proactively terminates due to some memory issues at the driver (SparkOOM, result size too large, etc...), due to metric sampling issues we will often miss this resourcing problem in the memory metrics and in the event log. We will abort the job before we capture accurate metrics for the driver. If we report an additional heartbeat (metric collection at the driver) on application termination than we will be able to better reflect the memory usage in the event log, shs, etc.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested
Tested with a custom job that collected a large amount of data to the driver, and otherwise had very low driver memory usage, (low # partitions no other data structures used at driver), without the change we witnessed that the peak memory usage at the driver was low <~100MiB, with this change we witness the higher memory usage reflected.
<img width="1723" height="230" alt="image" src="https://github.com/user-attachments/assets/fb442550-a262-453e-b6e2-f47e1e9f11b1" />

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #51882 from ForVic/vsunderl/report_driver_heartbeat.

Lead-authored-by: ForVic <vsunderland@linkedin.com>
Co-authored-by: Victor Sunderland <64456855+ForVic@users.noreply.github.com>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>

Author: 2 people

core/src/main/scala/org/apache/spark/Heartbeater.scala

@@ -48,6 +48,13 @@ private[spark] class Heartbeater(
     heartbeater.scheduleAtFixedRate(heartbeatTask, initialDelay, intervalMs, TimeUnit.MILLISECONDS)
   }
 
+  /**
+   * Reports a heartbeat.
+   */
+  def doReportHeartbeat(): Unit = {
+    reportHeartbeat()
+  }
+
   /** Stops the heartbeat thread. */
   def stop(): Unit = {
     heartbeater.shutdown()

core/src/main/scala/org/apache/spark/SparkContext.scala

@@ -2934,8 +2934,14 @@ class SparkContext(config: SparkConf) extends Logging {
     _driverLogger.foreach(_.startSync(_hadoopConfiguration))
   }
 
-  /** Post the application end event */
+  /** Post the application end event and report the final heartbeat */
   private def postApplicationEnd(exitCode: Int): Unit = {
+    try {
+      _heartbeater.doReportHeartbeat()
+    } catch {
+      case t: Throwable =>
+        logInfo("Unable to report driver heartbeat metrics when stopping spark context", t);
+    }
     listenerBus.post(SparkListenerApplicationEnd(System.currentTimeMillis, Some(exitCode)))
   }