Yesterday we saw the first post about Spark vs Flink and today we are going to change some parameters
Spark vs Flink – Round 2
First we are going to change the number of threads
val sparkConf = new SparkConf() .setAppName("Weather Test") .setMaster("local[8]")
Now we can execute the script again and the results have changed
Total time: 24.036728s + 23.077092s = 47.11382s (135.671764s yesterday, about 35% of the time we spent)
Now we are going to change the slots and parallelism. You can do that editing the file in conf/flink-conf.yml. I’ve used 8 slots with 300Mb each and a parallelism of 8
Let’s see the results
Total time: 44s + 47s = 91s (143s yesterday, about 64% of the time we spent)