Transforming IoT Data Preprocessing: A Holistic, Normalized and Distributed Approach

Data preprocessing is an integral part of Artificial Intelligence (AI) pipelines. It transforms raw data into input data that fulfill algorithmic criteria and improve prediction accuracy. As the adoption of Internet of Things (IoT) gains more momentum, the data volume generated from the edge is exponentially increasing that far exceeds any expansion of infrastructure. Social responsibilities and regulations (e.g., GDPR) must also be adhered when handling IoT data. In addition, we are currently witnessing a shift towards distributing AI to the edge. The aforementioned reasons render the distribution of data preprocessing to the edge an urgent requirement. In this paper, we introduce a modern data preprocessing framework that consists of two main parts. Part1 is a design tool that reduces the complexity and costs of the data preprocessing phase for AI via generalization and normalization. The design tool is a standard template that maps specific techniques into abstract categories and highlights dependencies between them. In addition, it presents a holistic notion of data preprocessing that is not limited to data cleaning. The second part is an IoT tool that adopts the edge-cloud collaboration model to progressively improve the quality of the data. It includes a synchronization mechanism that ensures adaptation to changes in data characteristics and a coordination mechanism that ensures correct and complete execution of preprocessing plans between the cloud and the edge. The paper includes an empirical analysis of the framework using a developed prototype and an automotive use-case. Our results demonstrate reductions in resource consumption (e.g., energy, bandwidth) while maintaining the value and integrity of the data.