Spark vs Flink – Part II

Yesterday we saw the first post about Spark vs Flink and today we are going to change some parameters

Spark vs Flink – Round 2

First we are going to change the number of threads

val sparkConf = new SparkConf()
      .setAppName("Weather Test")
      .setMaster("local[8]")

Now we can execute the script again and the results have changed

Total time: 24.036728s + 23.077092s = 47.11382s (135.671764s yesterday, about 35% of the time we spent)

Now we are going to change the slots and parallelism. You can do that editing the file in conf/flink-conf.yml. I’ve used 8 slots with 300Mb each and a parallelism of 8

Let’s see the results

Total time: 44s + 47s = 91s (143s yesterday, about 64% of the time we spent)