{"id":6674,"date":"2021-09-03T11:05:37","date_gmt":"2021-09-03T03:05:37","guid":{"rendered":"https:\/\/nj.transwarp.cn:8180\/?p=6674"},"modified":"2021-09-03T11:05:35","modified_gmt":"2021-09-03T03:05:35","slug":"slipstream-%e9%87%8d%e8%a6%81%e7%9a%84%e5%8f%82%e6%95%b0%e8%af%b4%e6%98%8e","status":"publish","type":"post","link":"https:\/\/kbwp.transwarp.cn\/?p=6674","title":{"rendered":"Slipstream \u91cd\u8981\u7684\u53c2\u6570\u8bf4\u660e"},"content":{"rendered":"

\u6982\u8981\u63cf\u8ff0<\/h3>\n
\n

Slipstream \u91cd\u8981\u7684\u53c2\u6570\u8bf4\u660e<\/p>\n

\u8be6\u7ec6\u8bf4\u660e<\/h3>\n
\n

Mapred And Morphling mode \u5171\u540c\u7684\u53c2\u6570<\/h4>\n

1. \u5f15\u64ce\u7ea7\u522b\uff1a\u9700\u8981\u5728\u542f\u52a8\u524d\u901a\u8fc7Manager\u754c\u9762\u914d\u7f6e\u91cd\u542f\u751f\u6548<\/strong>
\nngmr.engine.mode (mapred) \u5f15\u64ce\u6267\u884c\u7684\u6a21\u5f0f\uff0c\u9ed8\u8ba4\u662fmapred\uff0c\u82e5\u8981\u4f7f\u7528\u4e8b\u4ef6\u9a71\u52a8\u6a21\u5f0f\u7684stream\uff0c\u5219\u8981\u5207\u6362\u5230morphling<\/p>\n

2. Application\u7ea7\u522b: \u8bbe\u7f6e\u5728appproperties\u4e2d\u7684\u53c2\u6570, application\u7684\u53c2\u6570\u5bf9\u4e8e\u5f53\u524dapp\u4e0b\u6240\u6709\u7684streamjob\u90fd\u662f\u6709\u6548\u7684\u3002
\n**<\/strong>\u82e5\u662f\u8981\u6bcf\u4e2astreamjob\u533a\u5206\u4e0d\u540c\u7684\u53c2\u6570\uff0c\u53ef\u4ee5\u8bbe\u7f6e\u5728jobproperties\u4e2d\u3002<\/p>\n

streamsql.use.eventmode (false) \u662f\u5426\u4f7f\u7528\u4e8b\u4ef6\u9a71\u52a8\uff0c\u5f53\u524d\u5982\u679c\u5f15\u64ce\u662fmorphling\uff0c\u4f1a\u9ed8\u8ba4\u5f00\u542f\u8be5\u53c2\u6570\uff0c\u6ce8\u610f\u4e8b\u4ef6\u9a71\u52a8\u548c\u975e\u4e8b\u4ef6\u9a71\u52a8\u65e0\u6cd5\u540c\u65f6\u5728\u540c\u4e00\u4e2aapp\u4e0b\u8fd0\u884c\u3002<\/p>\n

3. Job \u7ea7\u522b\uff1a\u8bbe\u7f6e\u5728jobproperties\u4e2d\uff0cjob\u7ea7\u522b\u53c2\u6570\u8bbe\u7f6e\u5230app\u7ea7\u522b\u5219\u5bf9app\u4e0b\u6240\u6709streamjob\u90fd\u6709\u6548<\/strong><\/p>\n

streamsql.use.eventtime (false) \u662f\u5426\u4f7f\u7528\u4e8b\u4ef6\u65f6\u95f4<\/p>\n

stream.id \u6d41\u4efb\u52a1\u540d\uff0c\u5373 jobId<\/p>\n

streamsql.max.running.sql (1000) \u6700\u5927\u6267\u884cSQL\u6570<\/p>\n

streamsql.max.running.task (1000) \u6700\u5927\u8fd0\u884c\u4efb\u52a1\u6570<\/p>\n

Mapred mode\u4e0b\u7684\u53c2\u6570<\/h4>\n

\u5f15\u64ce\u7ea7\u522b<\/strong>
\nstream.driver.checkpoint.dir checkpoint\u7684\u76ee\u5f55\uff0c\u53ef\u914d\u7f6e\u4e3ahdfs\u7684\u76ee\u5f55<\/p>\n

stream.driver.enable.autofailover (false) \u662f\u5426\u5f00\u542f\u81ea\u52a8failover<\/p>\n

stream.driver.ha.node (\/InceptorServer2) zk\u4e0a\u7684\u8282\u70b9\u4fe1\u606f<\/p>\n

App&Job \u7ea7\u522b<\/strong>
\nstream.number.receivers (-1) receiver\u7684\u4e2a\u6570, -1\u8868\u793a\u6839\u636e\u6570\u636e\u6e90\u6765\u8ba1\u7b97\uff0c\u4f8b\u5982\u51e0\u4e2apartition\u5c31\u542f\u52a8\u51e0\u4e2areceiver<\/p>\n

streamsql.enable.hdfs.batchflush (false) \u662f\u5426\u6253\u5f00 batch flush\uff0c \u9632\u6b62\u4ea7\u751f\u8fc7\u591a\u5c0f\u6587\u4ef6<\/p>\n

streamsql.hdfs.batchflush.size (100000) \u6253\u5f00batch flush \u6bcf\u6279\u6b21\u7684\u5927\u5c0f<\/p>\n

streamsql.hdfs.batchflush.interval.ms (60000) batch flush \u7684\u65f6\u95f4\u95f4\u9694<\/p>\n

streamsql.eventtime.zerotime (0) \u4e8b\u4ef6\u65f6\u95f4\u7684\u8d77\u70b9<\/p>\n

streamsql.eventtime.maxlag.on.disorder (-1) \u4e8b\u4ef6\u65f6\u95f4\u5141\u8bb8\u6d41\u6570\u636e\u4e71\u5e8f\u7684\u6700\u5927\u7684\u95f4\u9694\u65f6\u95f4\uff0c-1\u8868\u793a\u6570\u636e\u662f\u6709\u5e8f\u7684<\/p>\n

streamsql.infinitewindow.numreducers (8) infinite window reduce task \u7684\u6570\u91cf<\/p>\n

\u4e0e\u4efb\u52a1\u8c03\u5ea6\u6709\u5173<\/strong><\/p>\n

stream.batch.duration.ms (2000) \u6279\u4efb\u52a1\u6bcf\u6279\u7684\u5927\u5c0f,\u9ed8\u8ba42s\u4ea7\u751f\u4e00\u6279\u4efb\u52a1<\/p>\n

spark.streaming.blockInterval (-1) \u6bcf\u4e2ablock\u7684\u95f4\u9694\uff0c\u9ed8\u8ba4200ms\uff0c\u4e00\u62792\u79d2\u7684\u4efb\u52a1\u6700\u591a\u53ef\u5305\u542b10\u4e2ablock<\/p>\n

spark.streaming.blockQueueSize (-1) block \u6682\u5b58\u7684queue \u5927\u5c0f\uff0c\u9ed8\u8ba410<\/p>\n

spark.streaming.receiver.maxRate (-1) receiver\u6570\u636e\u63a5\u53d7\u7684\u6700\u5927\u901f\u7387\uff0c\u9ed8\u8ba4\u662f0\u8868\u793a\u4e0d\u53d7\u9650\u5236\uff0c\u5373\u4ece\u6570\u636e\u6e90\u5f80receiver \u6bcf\u79d2push \u6700\u5927\u6761\u6570\u7684\u6570\u636e<\/p>\n

\u8be5\u53c2\u6570\u901a\u8fc7\u63a7\u5236\u6570\u636e\u6d41\u91cf\u4ee5\u9632\u6b62\u7a81\u7136\u6570\u636e\u91cf\u589e\u5927\uff0c\u7cfb\u7edf\u5904\u7406\u4e0d\u8fc7\u6765\uff0c\u5bfc\u81f4\u5185\u5b58\u6ea2\u51fa\u7b49\u95ee\u9898\u3002<\/p>\n

HA\u76f8\u5173\u7684\u53c2\u6570<\/strong><\/p>\n

application.checkpoint.dir application \u5f00\u542fcheckpoint \u7684\u76ee\u5f55<\/p>\n

application.enable.wal (false) \u662f\u5426\u5f00\u542fWAL<\/p>\n

Morphling mode\u4e0b\u7684\u53c2\u6570<\/p>\n

1. \u5f15\u64ce\u7ea7\u522b<\/strong>
\n\u7f51\u7edc\u76f8\u5173\u53c2\u6570\uff0cslipstream shuffle\u901a\u8fc7netty\u8fdb\u884c\u7f51\u7edc\u901a\u4fe1<\/p>\n

ngmr.io.mode (NIO), Netty IO \u65b9\u5f0f\uff0c\u53ef\u9009EPOLL<\/p>\n

ngmr.io.directbufs (true), Netty\u662f\u5426\u4f7f\u7528\u5bf9\u5916\u5185\u5b58\uff0c\u9ed8\u8ba4\u662ftrue<\/p>\n

ngmr.memsegment.size (32768) \u7cfb\u7edf\u4e00\u4e2amemory page \u7684\u5927\u5c0f\uff0c\u5355\u4f4dbytes, \u7528\u4e8e\u8bbe\u7f6eNetworkBufferPool\u4e00\u4e2abuffer\u7684\u5927\u5c0f\uff0c\u8bbe\u7f6eNetty\u9ad8\u4f4e\u6c34\u4f4d\u503c<\/p>\n

ngmr.network.num.buffers (10240) NetworkBufferPool \u521d\u59cb\u5316\u5927\u5c0f\uff0c\u5373\u6709\u591a\u5c11\u4e2abuffer\u53ef\u7528<\/p>\n

ngmr.network.client.num.threads (0) Netty Client\u7aefEventLoop\u7ebf\u7a0b\u6c60\u5927\u5c0f\uff0c\u9ed8\u8ba40\u8868\u793a\u5f53\u524dCPU core\u6570\u76842\u500d<\/p>\n

ngmr.network.server.num.threads (0) Netty Server\u7aef EventLoop\u7ebf\u7a0b\u6c60\u5927\u5c0f\uff0c\u8bbe\u7f6e\u8303\u56f4\u5efa\u8bae\u4e3a[ numOfCores + 1, 2 * numOfCores]<\/p>\n

ngmr.network.client.timeout (120) \u5355\u4f4d\u79d2\uff0c client\u7f51\u7edc\u8d85\u65f6\u65f6\u95f4<\/p>\n

ngmr.network.sndrcv.buffer (0) Netty \u5e95\u5c42TCP\u4f7f\u7528\u7684buffer\u5927\u5c0f\uff0c0\u8868\u793a\u91c7\u7528Netty\u7684\u9ed8\u8ba4\u503c<\/p>\n

Linux\u4e0b\u8bfb\u5199\u7684\u9ed8\u8ba4\u503c\u914d\u7f6e\u5728\u6587\u4ef6\u4e2d\u3002
\n\u7b2c\u4e00\u4e2a\u8868\u793a\u6700\u5c0f\u503c\uff0c\u7b2c\u4e8c\u4e2a\u8868\u793a\u9ed8\u8ba4\u503c\uff0c\u7b2c\u4e09\u4e2a\u8868\u793a\u6700\u5927\u503c\u3002
\n\"file\"<\/p>\n

\u4f7f\u7528HA\u65f6\u9700\u8981\u914d\u7f6e\u7684\u989d\u5916\u53c2\u6570
\nspark.morphling.recovery.mode (standalone \/ zookeeper ) \u5982\u679c\u9700\u8981\u652f\u6301Exactly Once\uff0c\u5219\u914d\u7f6eZookeeper
\nspark.morphling.taskstate.backend (filesystem) task\u72b6\u6001\u7684\u5b58\u50a8\u7cfb\u7edf\uff0c\u53ef\u4ee5\u8bf4\u4f1amemory, filesystem, db. \u5f53\u524d\u53ea\u652f\u6301filesystem
\nspark.morphling.taskstate.checkpoint.directory ( hdfs:\/\/xxx\/checkpoints) Morphling Task\u4fdd\u5b58State\u7684checkpoints\u7684\u5730\u5740\uff0c\u901a\u5e38\u662f\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\uff0c\u4fdd\u8bc1\u4e0d\u540c\u7684executor\u90fd\u80fd\u8bbf\u95ee<\/p>\n

\u5f53\u4f7f\u7528Zookeeper\u7684RecoverMode\u65f6\uff0c\u76f8\u5173\u7684\u914d\u7f6e\u5982\u4e0b\uff1a
\nspark.morphling.completed.checkpoints.storage.dir (\/morphling\/completedcheckpoints) Morphling Driver\u7aef\u4fdd\u5b58\u4efb\u52a1\u72b6\u6001checkpoints\u7684\u5730\u5740<\/p>\n

Zookeeper Client\u76f8\u5173\u53c2\u6570\uff1a
\nspark.morphling.zookeeper.quorum (localhost:2181); Zookeeper\u7684quorum
\nspark.morphling.zookeeper.session.timeout (10000)\uff1b\u8d85\u65f6\u65f6\u95f4\uff0c\u5355\u4f4d\u4e3ams
\nspark.morphling.zookeeper.connect.timeout (5000)\uff1b\u8d85\u65f6\u65f6\u95f4\uff0c\u5355\u4f4d\u4e3ams
\nspark.morphling.zookeeper.max.retries (3); \u6700\u591a\u91cd\u8bd5\u8fde\u63a5\u6b21\u6570
\nspark.morphling.zookeeper.retry.wait (5000); \u6bcf\u6b21\u91cd\u8bd5\u7b49\u5f85\u65f6\u95f4\uff0c\u5355\u4f4d\u4e3ams<\/p>\n

Zookeeper\u4e0aZnode\u8def\u5f84\u76f8\u5173\u53c2\u6570\uff1a
\nspark.morphling.zookeeper.root.path (\/morphling); Morphling\u4f7f\u7528zookeeper\u7684Root Path
\nspark.morphling.zookeeper.namespace (\/default); \u5f53\u524d\u96c6\u7fa4\u4f7f\u7528\u7684\u5b50znode\uff0c\u7528\u4e8e\u533a\u5206\u591a\u96c6\u7fa4\u7684\u73af\u5883\uff0c\u548cRoot Path\u914d\u5408\u4f7f\u7528
\nspark.morphling.zookeeper.checkpoints.path (\/checkpoints); \u7528\u4e8e\u8bb0\u5f55morphling\u4eba\u7269\u4e2d\u5f53\u524d\u5b8c\u6574\u7684checkpoint\u7684\u4fe1\u606f
\nspark.morphling.submitted.job.path (\/runningjobs); \u5f53\u5f00\u542fmorphling.job.enable.auto.failover\u81ea\u52a8Failover\u65f6\uff0c\u4f1a\u4fdd\u7559\u63d0\u4ea4\u7684\u4efb\u52a1\u7684\u4fe1\u606f<\/p>\n

2. Application\u7ea7\u522b<\/strong>
\n\u82e5\u662f\u8981\u6bcf\u4e2astreamjob\u533a\u5206\u4e0d\u540c\u7684\u53c2\u6570\uff0c\u53ef\u4ee5\u8bbe\u7f6e\u5728jobproperties\u4e2d\u3002<\/p>\n

streamsql.use.eventmode (false) \u662f\u5426\u4f7f\u7528\u4e8b\u4ef6\u9a71\u52a8\uff0c\u5f53\u524d\u5982\u679c\u5f15\u64ce\u662fmorphling\uff0c\u4f1a\u9ed8\u8ba4\u5f00\u542f\u8be5\u53c2\u6570\uff0c\u6ce8\u610f\u4e8b\u4ef6\u9a71\u52a8\u548c\u975e\u4e8b\u4ef6\u9a71\u52a8\u65e0\u6cd5\u540c\u65f6\u5728\u540c\u4e00\u4e2aapp\u4e0b\u8fd0\u884c\u3002<\/p>\n

3. Job \u7ea7\u522b<\/strong>
\nstreamsql.use.eventtime (false) \u662f\u5426\u4f7f\u7528\u4e8b\u4ef6\u65f6\u95f4<\/p>\n

morphling.maxlag.on.disorder (-1) \u5141\u8bb8\u6d41\u6570\u636e\u4e71\u5e8f\u7684\u6700\u5927\u95f4\u9694\u65f6\u95f4\uff0c-1\u8868\u793a\u6570\u636e\u6709\u5e8f<\/p>\n

morphling.infinitewindow.reduce.tasks (-1), Infinite window \u9ed8\u8ba4reduce\u4efb\u52a1\u7684\u6570\u91cf, -1\u8868\u793a\u7b49\u4e8e\u5f53\u524dreduce task\u7684\u6570\u76ee<\/p>\n

morphling.result.auto.flush (false) \u662f\u5426\u5f00\u542f\u81ea\u52a8sink\uff0c\u82e5\u5f00\u542f\u5219\u6709\u6570\u636e\u5c31\u7acb\u523bflush\u5230\u76ee\u6807\u8868<\/p>\n

morphling.hdfs.flush.size (100000) \u6570\u636e\u5237\u65b0\u5230\u76ee\u6807\u8868\u7684batch\u5927\u5c0f\uff0c\u9ed8\u8ba410\u4e07\u6761\u6570\u636e\u4f1a\u5237\u65b0\u4e00\u6b21<\/p>\n

morphling.hdfs.flush.interval.ms (60000) \u6570\u636e\u5237\u65b0\u5230\u76ee\u6807\u8868\u7684\u9891\u7387\uff0c\u9ed8\u8ba460\u79d2\u5237\u65b0\u4e00\u6b21\uff0c\u4e0e\u6761\u6570\u7684\u914d\u7f6e\u9879\u76f8\u8f85\u76f8\u6210\uff0c\u6309\u5148\u6ee1\u8db3\u6761\u4ef6\u6267\u884c<\/p>\n

\u4f7f\u7528HA\u542f\u52a8\u4efb\u52a1\u65f6\u7684\u4e00\u4e9b\u4efb\u52a1\u7ea7\u522b\u7684\u53c2\u6570<\/p>\n

morphling.task.max.failures (-1), \u4efb\u52a1\u6700\u5927\u5141\u8bb8\u5931\u8d25\u7684\u6b21\u6570\uff0c\u82e5\u5931\u8d25\u4f1a\u81ea\u52a8\u6062\u590d\uff0c-1\u8868\u793a\u4e0d\u5141\u8bb8<\/p>\n

morphling.job.enable.checkpoint (false), \u662f\u5426\u5f00\u542fcheckpoint \u529f\u80fd<\/p>\n

morphling.job.checkpoint.interval (60000), Checkpoint \u7684\u65f6\u95f4\u95f4\u9694<\/p>\n

morphling.job.enable.auto.failover (false) \u5f00\u542f\u4efb\u52a1\u81ea\u52a8\u6062\u590d\uff0c\u914d\u5408 morphling.task.max.failues\u4f7f\u7528<\/p>\n

morphling.job.enable.wal.sink (false), \u4fdd\u8bc1exactly-once\u8bed\u4e49\uff0c\u662f\u5426\u6253\u5f00wal sink \u529f\u80fd<\/p>\n

morphling.wal.sink.committer.type (mysql\/voltdb) wal sink\u4f9d\u8d56\u7684\u5916\u90e8\u6570\u636e\u5e93\u7c7b\u578b\uff0c\u652f\u6301mysql\u548cvoltdb <\/p>\n

morphling.wal.sink.voltdb.jdbc.url<\/p>\n

morphling.wal.sink.mysql.jdbc.url commit type\u914d\u7f6e\u6210\u5565\u5c31\u7528\u54ea\u4e2a<\/p>\n

\u4f7f\u7528metrics\u529f\u80fd\u7684\u53c2\u6570<\/strong><\/p>\n

morphling.metrics.enable (true)<\/p>\n","protected":false},"excerpt":{"rendered":"

\u6982\u8981\u63cf\u8ff0 Slipstream \u91cd\u8981\u7684\u53c2\u6570\u8bf4\u660e \u8be6\u7ec6\u8bf4\u660e Mapred And Morphling mode ..<\/p>\n

<\/div>\n

Read more<\/a><\/p>\n","protected":false},"author":9,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-6674","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"acf":[],"_links":{"self":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/posts\/6674","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6674"}],"version-history":[{"count":0,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/posts\/6674\/revisions"}],"wp:attachment":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6674"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6674"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6674"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}