内容纲要
概要描述
某些场景下我们需要定期重启quark服务,达到缓存/连接释放、降低节点内存使用、提高JVM垃圾回收效率等目的,加快quark计算引擎的执行效率。
该功能在银联和中山医院都有实际应用,反馈很好,其他客户现场可供参考。
重启窗口可以酌情选择,不方便重启的甚至可以考虑一周/一月重启一次。当然,重启越频繁,理论上效果会越好。
详细说明
功能设计:
- 根据cron表达式定义时间段来重启executor,如22-2,在22点00分到第二天2点59分间重启注册时间在22点前的executor;如果超过2点59分即使有executor未重启也不会再继续重启操作
- 重启前会先禁止任务调度到这些executor上,等与这些executor有关联的任务跑完,然后重启
- 支持参数配置热更新
使用方法:
quark组件新增如下参数配置,然后配置服务,重启quark。
| 参数 | 值 | 配置文件 | 说明 |
|---|---|---|---|
| quark.executor.restart.time.scheduled.enabled | true | hive-site.xml | 默认为false,为true表示开启定时重启功能 |
| quark.executor.restart.time.scheduled.cron.expression | * * 22-02 * * ? |
hive-site.xml | 默认为空,该表达式需要严格匹配正则^* * [^*].+$,22-02代表的是22点00分00秒到02点59分59秒(Quartz Cron) |
| quark.executor.restart.time.scheduled.timeout | 14400000 | hive-site.xml | 默认14400000,单位毫秒。executor会等待任务结束后再去重启,该配置为等待的最大时间 |
| quark.executor.restart.time.scheduled.batch.size | 1 | hive-site.xml | 默认1,一批一批重启executor,该配置指定了批大小 。注意:该配置必须小于executor的总数 |
| quark.executor.restart.time.scheduled.force | false | hive-site.xml | 默认false,该配置定义了超时等待${quark.executor.restart.time.scheduled.timeout} ms后是否要强制重启executor |
热更新
config global time.based.executor.restart.plan [key] [value]
注:热更新生效的前提条件是quark.executor.restart.time.scheduled.enabled = true
监控日志
- 可以通过
grep TimeBasedExecutorRestartPlan quark-server.log来查看被重启的executor - 可以通过
grep GracefulShutdownExecutorOperator quark-server.log来查看处于下线(禁止任务调度)和被重启的executor
2026-01-26 22:00:16,964 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Offlined executors 16
2026-01-26 22:00:16,965 ERROR cluster.CoarseGrainedSchedulerBackend: [sparkDriver-akka.actor.default-dispatcher-25()] - Asking executor 16 to suicide, exitCode 42
2026-01-26 22:00:16,966 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Stopped executors 16
2026-01-26 22:00:16,966 WARN executor.TimeBasedExecutorRestartPlan: [TimeBasedExecutorRestartPlan()] - Successfully stopped executor 16
2026-01-26 22:00:16,967 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Offlined executors 17
2026-01-26 22:00:16,967 ERROR cluster.CoarseGrainedSchedulerBackend: [sparkDriver-akka.actor.default-dispatcher-17()] - Asking executor 17 to suicide, exitCode 42
2026-01-26 22:00:16,967 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Stopped executors 17
2026-01-26 22:00:16,967 WARN executor.TimeBasedExecutorRestartPlan: [TimeBasedExecutorRestartPlan()] - Successfully stopped executor 17
2026-01-26 22:00:16,968 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Offlined executors 18
2026-01-26 22:00:16,968 ERROR cluster.CoarseGrainedSchedulerBackend: [sparkDriver-akka.actor.default-dispatcher-3()] - Asking executor 18 to suicide, exitCode 42
2026-01-26 22:00:16,969 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Stopped executors 18
2026-01-26 22:00:16,969 WARN executor.TimeBasedExecutorRestartPlan: [TimeBasedExecutorRestartPlan()] - Successfully stopped executor 18
2026-01-26 22:00:16,969 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Offlined executors 19
2026-01-26 22:00:16,969 ERROR cluster.CoarseGrainedSchedulerBackend: [sparkDriver-akka.actor.default-dispatcher-25()] - Asking executor 19 to suicide, exitCode 42
2026-01-26 22:00:16,970 WARN executor.GracefulShutdownExecutorOperator: [TimeBasedExecutorRestartPlan()] - Stopped executors 19
2026-01-26 22:00:16,970 WARN executor.TimeBasedExecutorRestartPlan: [TimeBasedExecutorRestartPlan()] - Successfully stopped executor 19
DBAService页面,执行器-历史活跃执行器,也能看到,执行器ID为16~19的历史记录:

metric-util 工具 /var/log/aquila/transwarp-ops/restart.record 也会记录重启操作,可视为合理输出。
