To increase SPARK kyroserializer.buffer.max
To address the errors – like the one below 6 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 2.0 (TID 16, n06.domain.com): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 3. To avoid this, increase spark.kryoserializer.buffer.max value. Set In CDH under SPARK, look for spark-defaults.conf, add the below. One of the two values below shuld …