内容纲要
概要描述
在部分业务场景下，需定期对 Quark 服务执行重启操作，以此释放缓存与连接资源，减少节点内存开销，改善 JVM 垃圾回收效果，进而提升 Quark 计算引擎的执行效率。
由于直接重启会造成 Quark 计算服务业务中断，本文提出灰度重启方案，能够在保障业务正常使用的前提下完成服务重启。方案已在多个客户现场实践使用，效果显著，推荐大家使用。
重启周期可根据实际情况灵活设置，理论上越频繁效果越好，比如每日一次；
slipstream组件不支持配置灰度重启
详细说明
功能设计：
- 根据cron表达式定义时间段来重启executor，如22-2，在22点00分到第二天2点59分间重启注册时间在22点前的executor；如果超过2点59分即使有executor未重启也不会再继续重启操作
- 重启前会先禁止任务调度到这些executor上，等与这些executor有关联的任务跑完，然后重启
- 支持参数配置热更新
使用方法：
quark组件新增如下参数配置，然后配置服务，重启quark。
| 参数 | 值 | 配置文件 | 说明 |
|---|---|---|---|
| quark.executor.restart.time.scheduled.enabled | true | hive-site.xml | 默认为false，为true表示开启定时重启功能 |
| quark.executor.restart.time.scheduled.cron.expression | * * 22-02 * * ? |
hive-site.xml | 默认为空，该表达式需要严格匹配正则^* * [^*].+$，22-02代表的是22点00分00秒到02点59分59秒（Quartz Cron） |
| quark.executor.restart.time.scheduled.timeout | 1800000 | hive-site.xml | 默认14400000，单位毫秒。executor会等待任务结束后再去重启，该配置为等待的最大时间 |
| quark.executor.restart.time.scheduled.batch.size | 2 | hive-site.xml | 默认1，一批一批重启executor，该配置指定了批大小 。注意：该配置必须小于executor的总数 |
| quark.executor.restart.time.scheduled.force | true | hive-site.xml | 默认false，该配置定义了超时等待${quark.executor.restart.time.scheduled.timeout} ms后是否要强制重启executor |
热更新
config global time.based.executor.restart.plan [key] [value] 注：热更新生效的前提条件是quark.executor.restart.time.scheduled.enabled = true
监控日志
- 可以通过
grep TimeBasedExecutorRestartPlan quark-server.log来查看被重启的executor - 可以通过
grep GracefulShutdownExecutorOperator quark-server.log来查看处于下线（禁止任务调度）和被重启的executor
2026-01-26 22:00:16,964 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Offlined executors 16
2026-01-26 22:00:16,965 ERROR cluster.CoarseGrainedSchedulerBackend: [sparkDriver-akka.actor.default-dispatcher-25()] - Asking executor 16 to suicide, exitCode 42
2026-01-26 22:00:16,966 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Stopped executors 16
2026-01-26 22:00:16,966 WARN executor.TimeBasedExecutorRestartPlan: [TimeBasedExecutorRestartPlan()] - Successfully stopped executor 16
2026-01-26 22:00:16,967 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Offlined executors 17
2026-01-26 22:00:16,967 ERROR cluster.CoarseGrainedSchedulerBackend: [sparkDriver-akka.actor.default-dispatcher-17()] - Asking executor 17 to suicide, exitCode 42
2026-01-26 22:00:16,967 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Stopped executors 17
2026-01-26 22:00:16,967 WARN executor.TimeBasedExecutorRestartPlan: [TimeBasedExecutorRestartPlan()] - Successfully stopped executor 17
2026-01-26 22:00:16,968 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Offlined executors 18
2026-01-26 22:00:16,968 ERROR cluster.CoarseGrainedSchedulerBackend: [sparkDriver-akka.actor.default-dispatcher-3()] - Asking executor 18 to suicide, exitCode 42
2026-01-26 22:00:16,969 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Stopped executors 18
2026-01-26 22:00:16,969 WARN executor.TimeBasedExecutorRestartPlan: [TimeBasedExecutorRestartPlan()] - Successfully stopped executor 18
2026-01-26 22:00:16,969 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Offlined executors 19
2026-01-26 22:00:16,969 ERROR cluster.CoarseGrainedSchedulerBackend: [sparkDriver-akka.actor.default-dispatcher-25()] - Asking executor 19 to suicide, exitCode 42
2026-01-26 22:00:16,970 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Stopped executors 19
2026-01-26 22:00:16,970 WARN executor.TimeBasedExecutorRestartPlan: [TimeBasedExecutorRestartPlan()] - Successfully stopped executor 19DBAService页面，执行器-历史活跃执行器，也能看到，执行器ID为16~19的历史记录：
metric-util 工具 /var/log/aquila/transwarp-ops/restart.record 也会记录重启操作，可视为合理输出。
FAQ
Q1: quark服务未能够正常灰度重启的原因
根据关键字 TimeBasedExecutorRestartPlan 检索日志，能够看到如下信息，Effected tasks 892952 still unfinished after 480000 ms， 表示 sqlid 为 892952 的这个sql语句，在 480000ms （quark.executor.restart.time.scheduled.timeout）内未能执行结束，quark.executor.restart.time.scheduled.force为false的情况下 不会强杀这个sql，最终在quark.executor.restart.time.scheduled.cron.expression的时间范围内未能重启成功。
2026-10-28 03:00:55,054 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Offlined executors 392,420
2026-10-28 03:08:55,059 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Effected tasks 892952 still unfinished after 480000 ms
2026-10-28 03:08:55,062 ERROR executor.TimeBasedExecutorRestartPlan: [TimeBasedExecutorRestartPlan()] - Stop executor 392,420 failed due to Graceful shutdown timeout 480000 ms
2026-10-28 03:08:55,063 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Offlined executors 421,422
2026-10-28 03:16:55,063 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Effected tasks 892952 still unfinished after 480000 ms
2026-10-28 03:16:55,064 ERROR executor.TimeBasedExecutorRestartPlan: [TimeBasedExecutorRestartPlan()] - Stop executor 421,422 failed due to Graceful shutdown timeout 480000 ms