Clustering streaming data in distributed environments based on belief propagation techniques
This study tries to examine a recent problem of computer science and specifically in data mining field, the online clustering of distributed streaming. This problem belongs to big data and analytics area, which means that we can’t apply the traditional techniques, software or databases to capture, process and analyze this data with low-latency and we need massive parallelism. Moreover, when these big data techniques applied to streaming data more challenges are emerged. On the other hand, the analyzing of streaming data will give a lot of advantages like real time insights of the data that will help to respond in emerging situations. The sequential and distributed fashion of the data produced from a variety of devices combined with the volume of them and constraints such as communication and storage make a major challenge the clustering of streaming data. In our study we address the problem of distributed clustering using two level of clustering approach, in first level, batch of data arrives in many distributed nodes in each time slot and the nodes performs clustering in these data extracting the most significant representatives of the batch (exemplars) which will be forwarded to the central node which in turn performs the second level of clustering in order to identify global patterns in the data arrived from every node The algorithm that we will try to implement uses belief propagation techniques in a distributed environment. The exemplars will feed back to the nodes with the appropriately modified weight which reflect their global significance. We adopt belief propagation techniques in both levels to perform streaming clustering.