To increase SPARK kyroserializer.buffer.max

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

To address error like the one below

6 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 2.0 (TID 16, n06.domain.com): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 3. To avoid this, increase spark.kryoserializer.buffer.max value.

In CDH under SPARK look for spark-defaults.conf, add the below

One of the two values below shuld work (not sure which one)
spark.kryoserializer.buffer.max=64m
spark.kryoserializer.buffer.mb=64m

Deploy the config once done the modify

Note: The file below gets modified
/etc/spark/conf/spark-defaults.conf

Leave a Reply

avatar
3000
  Subscribe  
Notify of