{"id":7622,"date":"2022-01-25T15:00:42","date_gmt":"2022-01-25T07:00:42","guid":{"rendered":"https:\/\/nj.transwarp.cn:8180\/?p=7622"},"modified":"2022-01-25T15:41:56","modified_gmt":"2022-01-25T07:41:56","slug":"slipstream-kafka-checklist","status":"publish","type":"post","link":"https:\/\/kbwp.transwarp.cn\/?p=7622","title":{"rendered":"Slipstream\/Kafka checklist"},"content":{"rendered":"<h3>\u6982\u8981\u63cf\u8ff0<\/h3>\n<hr \/>\n<p>\u672c\u6587\u4ecb\u7ecd\u5f53Slipstream\/Kafka \u6570\u636e\u79ef\u538b\u6216\u8005\u542f\u52a8\u5931\u8d25\u65f6\uff0c\u5982\u4f55\u6536\u96c6\u5f02\u5e38\u4fe1\u606f\uff0c\u4fbf\u4e8e\u6392\u67e5\u6839\u672c\u539f\u56e0\uff0c\u63d0\u4f9b\u5bf9\u5e94\u7684\u89e3\u51b3\u65b9\u6848<\/p>\n<h3>\u8be6\u7ec6\u8bf4\u660e<\/h3>\n<hr \/>\n<p><strong>\u5907\u4efd\u7ec4\u4ef6\u65e5\u5fd7<\/strong><\/p>\n<p>\u767b\u5f55\u7ec4\u4ef6\u5bf9\u5e94\u670d\u52a1\u5668linux\u547d\u4ee4\u5907\u4efd\u5230\u6570\u636e\u76d8\u76ee\u5f55<\/p>\n<p>\u786e\u8ba4\u8981\u5907\u4efd\u65e5\u5fd7\u6587\u4ef6 $ERR_LOG<\/p>\n<pre><code class=\"language-sql\">ls -ltr \/var\/log\/slipstream1\/hive-server.log*<\/code><\/pre>\n<p>\u5907\u4efd\u6587\u4ef6<\/p>\n<pre><code class=\"language-sql\">DATE=$(date +%Y%m%d) && mkdir -p \/mnt\/disk1\/log_err\/$DATE && cp $ERR_LOG \/mnt\/disk1\/log_err\/$DATE\n<\/code><\/pre>\n<p><strong>\u7ec4\u4ef6\u65e0\u6cd5\u542f\u52a8<\/strong><\/p>\n<pre><code class=\"language-sql\">1\uff09Pod\u72b6\u6001\n\n\u6267\u884c\u547d\u4ee4\u201ckubectl get pods -o wide | grep slipstream\u201d\uff0c\u83b7\u53d6\u547d\u4ee4\u8f93\u51fa\n\n\u5bf9\u4e8e\u6bcf\u4e00\u4e2apod\uff0c\u6267\u884c\u547d\u4ee4\u201ckubectl logs <pod\u540d\u5b57>\u201d\uff0c\u83b7\u53d6\u547d\u4ee4\u8f93\u51fa\n\n\u5bf9\u4e8e\u6bcf\u4e00\u4e2apod\uff0c\u6267\u884c\u547d\u4ee4\u201ckubectl describe po <pod\u540d\u5b57>\u201d\uff0c\u83b7\u53d6\u547d\u4ee4\u8f93\u51fa\n\n\u83b7\u53d6\u7ec4\u4ef6\u7684\u65e5\u5fd7\uff0c\u5728slipstream\u5b89\u88c5\u7684\u8282\u70b9\u4e0a\u8fdb\u5165\u8def\u5f84\u201c\/var\/log\/<slipstream\u5b9e\u4f8b\u540d>\u201d\uff0c\u4f8b\u5982\/var\/log\/slipstream1\uff0c\u83b7\u53d6\u4ee5\u4e0b\u65e5\u5fd7\u6587\u4ef6\uff1a\n\nhive-metastore.log*\uff08\u5982\u679cmetastore\u5171\u4eab\u7684inceptor\u7684\uff0c\u5c31\u53bbinceptor\u65e5\u5fd7\u8def\u5f84\u4e0b\u83b7\u53d6\u8be5\u65e5\u5fd7\uff09\u8fdb\u5165Manager\u9875\u9762\uff0c\u518d\u8fdb\u5165slipstream\u7ec4\u4ef6\uff0c\u53f3\u4e0a\u89d2\u66f4\u591a\u64cd\u4f5c\uff0c\u66f4\u65b0\u4f9d\u8d56\u91cc\u9762\uff0c\u67e5\u770b\u662f\u5426\u548cinceptor\u5171\u4eabMetaStore\n\nhive-server2.log*\n\ninceptor-executor.log*\n\n2\uff09\u5728Manager\u8282\u70b9\uff0c\u6536\u96c6\/var\/log\/transwarp-manager\/master\/transwarp-manager.log*\n\n3\uff09\u8fdb\u5165Manager\u9875\u9762\uff0c\u518d\u8fdb\u5165kafka\u7ec4\u4ef6\uff0c\u5de6\u4fa7\"\u76fe\u724c\"\u6309\u94ae\u67e5\u770b\u7ec4\u4ef6\u662f\u5426\u5f00\u4e86\u5b89\u5168\uff0c\u622a\u56fe\uff1b\u9488\u5bf94.x\u7248\u672c\u7684kafka\uff0c\u63d0\u4f9b\/etc\/<kafka\u5b9e\u4f8b\u540d>\/conf\/server.properties\u6587\u4ef6\u548c<TDH\u5b89\u88c5\u8def\u5f84>\/transwarp\/tdh_client\/kafka\/config\/server.properties\uff0c\u4f8b\uff1a\/mnt\/disk1\/transwarp\/tdh_client\/kafka\/config\/server.properties<\/code><\/pre>\n<p><strong>\u6570\u636e\u79ef\u538b<\/strong><\/p>\n<pre><code class=\"language-sql\">1\uff09\u8fdb\u5165Manager\u9875\u9762\uff0c\u518d\u8fdb\u5165kafka\u7ec4\u4ef6\u7684\u89d2\u8272\u9875\u9762\uff0c\u67e5\u770b\u5404\u89d2\u8272\u72b6\u6001\uff0c\u622a\u56fe\uff1b\n\n2\uff09\u8fdb\u5165Manager\u9875\u9762\uff0c\u518d\u8fdb\u5165slipstram\u7ec4\u4ef6\u7684\u89d2\u8272\u9875\u9762\uff0c\u67e5\u770b\u5404\u89d2\u8272\u72b6\u6001\uff0c\u622a\u56fe\uff1b\n\n3\uff09\u4efb\u52a1\u6267\u884c\u60c5\u51b5\n\n\u8fdb\u5165Manager\u9875\u9762\uff0c\u518d\u8fdb\u5165slipstrem\u7ec4\u4ef6\u7684\u89d2\u8272\u9875\u9762\uff0c\u901a\u8fc7slipstream-Server\u7684Link\u8fdb\u53bb4040\u9875\u9762\uff0c\u83b7\u53d6\u6d41\u4efb\u52a1SQL\u5bf9\u5e94\u4efb\u52a1\u7684job\uff0cstage\u548ctask\u9875\u9762\u7684\u622a\u56fe\n\n4\uff09\u83b7\u53d6\u7ec4\u4ef6\u7684\u65e5\u5fd7\uff0c\u5728slipstream\u5b89\u88c5\u7684\u8282\u70b9\u4e0a\u8fdb\u5165\u8def\u5f84\u201c\/var\/log\/<slipstream\u5b9e\u4f8b\u540d>\u201d\uff0c\u4f8b\u5982\/var\/log\/slipstream1\uff0c\u83b7\u53d6\u4ee5\u4e0b\u65e5\u5fd7\u6587\u4ef6\uff1a\n\nhive-metastore.log*\n\nhive-server2.log*\n\ninceptor-executor.log*\n\n5\uff09\u6839\u636e\u811a\u672c\u5224\u65ad\u6570\u636e\u79ef\u538b\u60c5\u51b5\u4ee5\u53ca\u79ef\u538b\u91cf\uff1a\n\n\u811a\u672c\u8def\u5f84\uff1amanager\u8282\u70b9\uff0c\/var\/lib\/transwarp-manager\/master\/content\/resources\/tdh_client\/kafka\/bin\n\n\u6267\u884c\u547d\u4ee4\uff1a.\/kafka-consumer-groups.sh --describe --group default-default.demo_stream\uff08stream\u540d\u5b57\uff09--bootstrap-server 192.168.31.33:9092\uff08kafka ip:port\uff09\n\nstream\u4efb\u52a1\u540d\u79f0\u83b7\u53d6\u65b9\u5f0f\uff1a\u5728beeline\u6a21\u5f0f\u4e0b\uff0c\u6267\u884cshow streams\uff1b<\/code><\/pre>\n<p><strong>\u6d41\u4efb\u52a1\u6267\u884c\u6162<\/strong><\/p>\n<pre><code class=\"language-sql\">1\uff09slipstream\u53c2\u6570\n\n\u8fdb\u5165Manager\u9875\u9762\uff0c\u518d\u8fdb\u5165slipstream\u7ec4\u4ef6\u7684\u914d\u7f6e\u9875\u9762\uff0c\u83b7\u53d6\u4ee5\u4e0b\u53c2\u6570\u7684\u503c\uff1a\n\ninceptor.executor.cores\n\ninceptor.executor.memory\n\ninceptor.server.memory\n\n2\uff09\u4efb\u52a1\u6267\u884c\u60c5\u51b5\n\n\u8fdb\u5165Manager\u9875\u9762\uff0c\u518d\u8fdb\u5165slipstream\u7ec4\u4ef6\u7684\u89d2\u8272\u9875\u9762\uff0c\u901a\u8fc7Inceptor Server\u7684Link\u8fdb\u53bb4040\u9875\u9762\uff0c\u83b7\u53d6\u6d41\u4efb\u52a1\u5bf9\u5e94\u7684job\uff0cstage\u548ctask\u9875\u9762\u7684\u622a\u56fe\n\n\u5982\u679c\u4e00\u4e2a\u4efb\u52a1\u957f\u65f6\u95f4\u6267\u884c\u4e0d\u5b8c\uff0c\u5bf9\u4e8e\u6bcf\u4e00\u4e2aexecutor\uff0c\u83b7\u53d6jstack\u4fe1\u606f\uff0c\u4f8b\u5982\uff1a\n\n\u6267\u884c\u547d\u4ee4\u201ckubectl exec -it {executor pod\u540d\u79f0} bash\u201d\u8fdb\u5165pod\n\n\u5728pod\u4e2d\uff0c\u6267\u884c\u547d\u4ee4\u201cjps\u201d\u83b7\u53d6executor\u8fdb\u7a0b\u7684pid\n\n\u6267\u884c\u547d\u4ee4\u201csudo -u hive jstack -l <executor pid> > \/tmp\/jstack.log\u201d\uff0c\u6536\u96c6\u8fd9\u4e2a\u8f93\u51fa\u6587\u4ef6<\/code><\/pre>\n<p><strong>\u6027\u80fd\u4fe1\u606f<\/strong><\/p>\n<p><strong>1.\u6709GC\u60c5\u51b5<\/strong><br \/>\n<strong>2.\u5185\u5b58\uff0ccpu\uff0cio\u8d44\u6e90\u504f\u9ad8<\/strong><\/p>\n<pre><code class=\"language-sql\">1\uff09Java\u5806\u6808\u4fe1\u606f jstack \u00d710 inceptor\/executor\n\n\u8bf4\u660e\uff1aGC\u4e00\u5b9a\u8981\u6536\u96c6\n\njstack\u5de5\u5177\uff1a\nPID=$(jps|grep HRegionServer| awk '{print$1}') && sudo -u hbase $(whereis jstack | awk -F':' '{print $2}') $PID\n\n2\uff09jmap  inceptor\/executor\n\n\u8bf4\u660e\uff1aGC\u4e00\u5b9a\u8981\u6536\u96c6\uff0cjmap\u4e0d\u4ec5\u80fd\u751f\u6210dump\u6587\u4ef6\uff0c\u8fd8\u53ef\u4ee5\u67e5\u8be2finalize\u6267\u884c\u961f\u5217\u3001Java\u5806\u548c\u6c38\u4e45\u4ee3\u7684\u8be6\u7ec6\u4fe1\u606f\uff0c\u5982\u5f53\u524d\u4f7f\u7528\u7387\u3001\u5f53\u524d\u4f7f\u7528\u7684\u662f\u54ea\u79cd\u6536\u96c6\u5668\u7b49\n\njmap\u5de5\u5177\uff1a\njmap -histo:live 24971 | more\n\n3\uff09heapdump\n\n\u8bf4\u660e\uff1a\u6bd4\u8f83\u5168\u9762\uff0c\u4f46\u6587\u4ef6\u8f83\u5927\n\njmap\u5de5\u5177\uff1a\njmap -dump:live,format=b,file=\/tmp\/dump.hprof 24971\n\n4\uff09GC\u65e5\u5fd7\n\n\u8bf4\u660e\uff1aGC\u4e00\u5b9a\u8981\u6536\u96c6\n\njstat\u5de5\u5177+\u7ec4\u4ef6gc\u65e5\u5fd7\uff1a\njstat -gcutil 3024 1000 10\n\n5\uff09SAR\u65e5\u5fd7\n\n\u8bf4\u660e\uff1a\u51fa\u73b0\u5185\u5b58\u3001CPU\u3001IO\u8d44\u6e90\u504f\u9ad8\u7684\u60c5\u51b5\u4e0b\uff0c\u9700\u8981\u8bb0\u5f55 \uff1b \u7cfb\u7edf\u72b6\u6001\u8fdb\u884c\u53d6\u6837\uff0c\u5e76\u6301\u7eed\u8bb0\u5f55\n\nsar\u547d\u4ee4\uff1a\n#\u6bcf10\u79d2\u91c7\u6837\u4e00\u6b21\uff0c\u8fde\u7eed\u91c7\u68373\u6b21\uff0c\u62a5\u544a\u8bbe\u5907\u4f7f\u7528\u60c5\u51b5\nsar -d 10 3 -p\n\n6\uff09ganglia\u622a\u56fe\n\nganglia\u9875\u9762\u96c6\u7fa4\u7684\u7b2c\u4e00\u4e2a\u8282\u70b9\uff0c\u53ef\u4ee5\u9009\u62e9\u7edf\u8ba1\u5468\u671f\nhttp:\/\/\u7b2c\u4e00\u4e2a\u8282\u70b9IP\/ganglia<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>\u6982\u8981\u63cf\u8ff0 \u672c\u6587\u4ecb\u7ecd\u5f53Slipstream\/Kafka \u6570\u636e\u79ef\u538b\u6216\u8005\u542f\u52a8\u5931\u8d25\u65f6\uff0c\u5982\u4f55\u6536\u96c6\u5f02\u5e38\u4fe1\u606f\uff0c\u4fbf\u4e8e\u6392\u67e5\u6839\u672c\u539f ..<\/p>\n<div class=\"clear-fix\"><\/div>\n<p><a href=\"https:\/\/kbwp.transwarp.cn\/?p=7622\" title=\"read more...\">Read more<\/a><\/p>\n","protected":false},"author":9,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-7622","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"acf":[],"_links":{"self":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/posts\/7622","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7622"}],"version-history":[{"count":0,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/posts\/7622\/revisions"}],"wp:attachment":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7622"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7622"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7622"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}