Stream processing is a rising topic in the big data domain to process events in real time or close to real time. There are lots of modern solutions out there developers can choose from, e.g. :
and many more…
This blog aims at documenting student’s experiences with different stream processing engines for a practical course. The task is to develope an application on top of the selected stream processing engines, which will process real traffic data collected by cyclists via smartphone sensors (GPS, accelerometer, gyroscope, magnetometer, etc.). The dataset to process and the algorithm on which the application should base on is provided by Cyface. The goal is to generate a surface quality model of the public road network. The main challenge here is to process a very big amout of small geo-located sensor events.
For the first practical course held from Oct. 2017 till Feb. 2018, the two stream processing engines Apache Flink and Apache Spark have been choosen. Stay tuned!