使用FlinkSQL查询MySQL写入Greenplum, 出现Java heap space问题,可以通过配置避免吗?
环境是本机用docker部署的1.11.2版flink集群,就一个task manager。
尝试在SQL Client里查一张50w数据的mysql表并写入GP,结果task manager报Java heap space异常。似乎是select的数据量太大,导致频繁GC。
试过配置scan.fetch-size和lookup.cache.max-rows,没有效果。请问有什么办法能避免呢?
Task manager报错信息如下:
java.lang.IllegalArgumentException: open() failed.Java heap space
at org.apache.flink.connector.jdbc.table.JdbcRowDataInputFormat.open(JdbcRowDataInputFormat.java:209) ~[flink-connector-jdbc_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:85) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
Caused by: java.sql.SQLException: Java heap space
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) ~[flink-sql-connector-mysql-cdc-1.0.0.jar:1.0.0]
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[flink-sql-connector-mysql-cdc-1.0.0.jar:1.0.0]
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[flink-sql-connector-mysql-cdc-1.0.0.jar:1.0.0]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955) ~[flink-sql-connector-mysql-cdc-1.0.0.jar:1.0.0]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1005) ~[flink-sql-connector-mysql-cdc-1.0.0.jar:1.0.0]
at org.apache.flink.connector.jdbc.table.JdbcRowDataInputFormat.open(JdbcRowDataInputFormat.java:206) ~[flink-connector-jdbc_2.12-1.11.2.jar:1.11.2]
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论