Spaksql Error：Kryo serialization failed: Buffer overflow

Question

我使用sparksql编写的job以yarn-client方式提交后出现了kryo序列化缓存溢出的情况，报错如下：

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 497, upwallet-cdh02): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 4. To avoid this, increase spark.kryoserializer.buffer.max value. 
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:299)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:240) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

我的代码相关参数设置如下：

 val conf = new SparkConf()
      .setAppName("Test")
      .set("spark.serializer","org.apache.spark.serializer.KryoSerializer")
      .set("spark.Kryoserializer.buffer.max","1024m")
      .set("spark.yarn.driver.memoryOverhead","1024")
      .set("spark.yarn.executor.memoryOverhead","2000")
      .set("spark.newwork.buffer.timeout","300s")
      .set("spark.executor.heartbeatInterval","30s")
      .set("spark.driver.extraJavaOptions","-XX:+UseG1GC -XX:+UseCompressedOops")
      .set("spark.executor.extraJavaOptions","-XX:+UseG1GC -XX:+UseCompressedOops")

我做过的尝试如下：

1、增大缓存 设置为原来的2倍，3倍，错误依旧
2、恢复默认值 64 ，错误依旧
3、取消序列化机制，错误依旧

求spark学习爱好者能够为小弟不吝赐教，先行感谢！！！