{"id":7116,"date":"2024-01-30T14:36:02","date_gmt":"2024-01-30T06:36:02","guid":{"rendered":"https:\/\/nj.transwarp.cn:8180\/?p=7116"},"modified":"2024-06-07T11:54:38","modified_gmt":"2024-06-07T03:54:38","slug":"%e5%bc%80%e6%ba%90flink-on-yarn-%e9%83%a8%e7%bd%b2","status":"publish","type":"post","link":"https:\/\/kbwp.transwarp.cn\/?p=7116","title":{"rendered":"\u5f00\u6e90Flink 1.11.1 on yarn-6.2.2 \u90e8\u7f72"},"content":{"rendered":"<h3>\u6982\u8981\u63cf\u8ff0<\/h3>\n<p>\u5728\u4e00\u4e9b\u573a\u666f\u4e2d\u4f7f\u7528\u5f00\u6e90Flink\u6267\u884c\u76f8\u5173\u6d41\u5904\u7406\u4efb\u52a1\uff0c\u9700\u8981\u5c06Flink\u4efb\u52a1\u63d0\u4ea4\u81f3Yarn\u4e0a\u3002\u6b64KB\u5c06\u6307\u5bfc\u80fd\u591f\u5728TDH\u5f00\u542f\u5b89\u5168\u7684\u63d0\u4ea4Flink ON Yarn\u7684Session\u3002<\/p>\n<p>\u6b64KB\u6848\u4f8b\u91c7\u7528Flink\u7248\u672c\u4e3aFlink 1.11.1\uff0cTDH\u7248\u672c\u4e3aTDH 6.2.2\uff08\u5f00\u542fguardian\u5b89\u5168\uff09<\/p>\n<hr \/>\n<h3>\u8be6\u7ec6\u8bf4\u660e<\/h3>\n<h4>1\u3001\u90e8\u7f72TDH-Client<\/h4>\n<p>\u5728\u9700\u8981\u5728\u63d0\u4ea4Flink\u7a0b\u5e8f\u7684\u673a\u5668\u4e0a\u63d0\u524d\u5b8c\u6210TDH-Clinet\u7684\u90e8\u7f72\uff0c\u53ef\u4ee5\u53c2\u8003KB-<a href=\"https:\/\/nj.transwarp.cn:8180\/?p=1105\" title=\"\u51c6\u5907TDH\u5ba2\u6237\u7aef\u73af\u5883\">\u51c6\u5907TDH\u5ba2\u6237\u7aef\u73af\u5883<\/a><\/p>\n<h4>2\u3001\u51c6\u5907Flink\u5ba2\u6237\u7aef<\/h4>\n<p>\u5728Flink\u5b98\u65b9\u6e20\u9053\u4e2d\u53ef\u4ee5\u4e0b\u8f7d\u7b26\u5408\u4ee3\u7801\u9700\u6c42\u7684Flink\u7248\u672c\uff0c\u5982\u672c KB\u4e2d\u4f7f\u7528\u7684<a href=\"https:\/\/archive.apache.org\/dist\/flink\/flink-1.11.1\/flink-1.11.1-bin-scala_2.11.tgz\" title=\"flink-1.11.1-bin-scala_2.11.tgz \">flink-1.11.1-bin-scala_2.11.tgz <\/a>\u3002<\/p>\n<p>\u5c06flink-1.11.1-bin-scala_2.11.tgz\u4e0a\u4f20\u81f3\u673a\u5668\u4e0a\u5e76\u89e3\u538b\u3002<\/p>\n<p><code>tar -zxvf flink-1.17.2-bin-scala_2.12.tgz<\/code><\/p>\n<h4>3\u3001 \u4fee\u6539Flink\u914d\u7f6e\u6587\u4ef6<\/h4>\n<p><strong>1\u3001\u6dfb\u52a0 jaas.conf \u6587\u4ef6<\/strong><\/p>\n<p><code>cp \/etc\/yarn1\/conf\/jaas.conf flink-1.11.2\/conf\/<\/code><\/p>\n<p><strong>2\u3001\u4fee\u6539 flink-conf.yaml<\/strong><\/p>\n<p><code>vi flink-1.11.2\/conf\/flink-conf.yaml<\/code><\/p>\n<p>\u6dfb\u52a0\u5982\u4e0b\u5185\u5bb9\uff1a<\/p>\n<pre><code class=\"language-shell\">env.java.home: \/usr\/java\/jdk1.8.0_25\/\n#\u7531\u4e8eTDH6.2.2\u9ed8\u8ba4java\u7248\u672c\u4e3ajdk1.7\uff0c\u9700\u8981\u624b\u52a8\u6307\u5b9aOn yarn\u8fd0\u884c\u65f6\u7684jdk\u7248\u672c\u4e3a1.8\ncontainerized.master.env.JAVA_HOME: \/usr\/java\/jdk1.8.0_25\/\ncontainerized.taskmanager.env.JAVA_HOME: \/usr\/java\/jdk1.8.0_25\/\njava.security.auth.login.config: \/home\/flink-1.11.2\/conf\/jaas.conf\nsecurity.kerberos.login.use-ticket-cache: true\nsecurity.kerberos.login.keytab: \/etc\/yarn1\/conf\/yarn.keytab\nsecurity.kerberos.login.principal: yarn\/test01@TDH\nsecurity.kerberos.login.contexts: Client\nclassloader.check-leaked-classloader: false<\/code><\/pre>\n<h4>5\u3001\u63d0\u4ea4yarn-session<\/h4>\n<p>\u6267\u884c\u547d\u4ee4\u8fd0\u884cFlink On Yarn\uff1a<\/p>\n<pre><code class=\"language-shell\">source \/root\/TDH-Client\/init.sh y y\nexport HADOOP_CLASSPATH=<code>hadoop classpath<\/code>\nexport HADOOP_CONF_DIR=\/etc\/yarn1\/conf\/\n.\/flink-1.11.2\/bin\/yarn-session.sh -d<\/code><\/pre>\n<p>\u5982\u679c\u8fd0\u884c\u6210\u529f\u4f1a\u6253\u5370\uff1a<\/p>\n<pre><code class=\"language-shell\">JobManager Web Interface: http:\/\/ts-cloudyi06:42781\n2021-11-18 11:21:50,394 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - The Flink YARN session cluster has been started in detached mode. In order to stop Flink gracefully, use the following command:\n$ echo \"stop\" | .\/bin\/yarn-session.sh -id application_1637139361677_0012\nIf this should not be possible, then you can also kill Flink via YARN's web interface or via:\n$ yarn application -kill application_1637139361677_0012\nNote that killing Flink might not clean up all job artifacts and temporary files.<\/code><\/pre>\n<p>\u5728\u6d4f\u89c8\u5668\u4e2d\u6253\u5f00yarn\u7ba1\u7406\u9875\u9762\u53ef\u4ee5\u770b\u5230flink-session\u5e94\u7528Running\u3002<br \/>\n<img decoding=\"async\" src=\"https:\/\/nj.transwarp.cn:8180\/wp-content\/uploads\/2021\/11\/\u622a\u5c4f2021-11-18-\u4e0a\u534811.29.15.png\" alt=\"cloudyi001\" \/><br \/>\n\u5728\u6d4f\u89c8\u5668\u4e2d\u8bbf\u95ee\u63a7\u5236\u53f0\u6253\u5370\u7684\u8fde\u63a5 <a href=\"http:\/\/ts-cloudyi06:42781\">http:\/\/ts-cloudyi06:42781<\/a> \u6216\u8005yarn\u754c\u9762\u4e0a\u7684Tracking UI\u94fe\u63a5\u90fd\u53ef\u4ee5\u8bbf\u95eeFlink\u7684\u7ba1\u7406\u754c\u9762\u3002<br \/>\n<img decoding=\"async\" src=\"https:\/\/nj.transwarp.cn:8180\/wp-content\/uploads\/2021\/11\/\u622a\u5c4f2021-11-18-\u4e0a\u534811.33.42.png\" alt=\"cloudyi002\" \/><\/p>\n<hr \/>\n<h3>\u5176\u4ed6\u4fe1\u606f<\/h3>\n<p>\u672cKB\u4ec5\u5982\u4f55\u8fd0\u884cFlinkOnYarn\u7684\u57fa\u672c\u64cd\u4f5c\uff0c\u5982\u679c\u9700\u8981\u8fd0\u884cJOB\u9700\u8981\u81ea\u884c\u6784\u5efaFlinkdemo\u7a0b\u5e8f\uff0c\u66f4\u591a\u5173\u4e8eFlink\u7528\u6cd5\u53c2\u89c1\uff1a<a href=\"https:\/\/nightlies.apache.org\/flink\/flink-docs-master\/zh\/\" title=\"Flink\u793e\u533a\u5b98\u65b9\u6587\u6863\">Flink\u793e\u533a\u5b98\u65b9\u6587\u6863<\/a><\/p>\n<p>\u5efa\u8bae\u60a8\u6784\u5efaFlinkDemo\u7a0b\u5e8f\u65f6\u4f7f\u7528jdk1.8.25\u7248\u672c\uff0c\u5426\u5219\u53ef\u80fd\u4f1a\u51fa\u73b0\u62a5\u9519:<\/p>\n<pre><code class=\"language-shell\">aused by: java.lang.Error: Failed to find GC Cleaner among available providers: [Legacy (before Java 9) cleaner provider, New Java 9+ cleaner provider]<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>\u6982\u8981\u63cf\u8ff0 \u5728\u4e00\u4e9b\u573a\u666f\u4e2d\u4f7f\u7528\u5f00\u6e90Flink\u6267\u884c\u76f8\u5173\u6d41\u5904\u7406\u4efb\u52a1\uff0c\u9700\u8981\u5c06Flink\u4efb\u52a1\u63d0\u4ea4\u81f3Yarn\u4e0a\u3002\u6b64KB\u5c06\u6307\u5bfc ..<\/p>\n<div class=\"clear-fix\"><\/div>\n<p><a href=\"https:\/\/kbwp.transwarp.cn\/?p=7116\" title=\"read more...\">Read more<\/a><\/p>\n","protected":false},"author":119,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[47],"tags":[],"class_list":["post-7116","post","type-post","status-publish","format-standard","hentry","category-third_part"],"acf":[],"_links":{"self":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/posts\/7116","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/users\/119"}],"replies":[{"embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7116"}],"version-history":[{"count":4,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/posts\/7116\/revisions"}],"predecessor-version":[{"id":13283,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=\/wp\/v2\/posts\/7116\/revisions\/13283"}],"wp:attachment":[{"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7116"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7116"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kbwp.transwarp.cn\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7116"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}