Authors : Dong Jin Shin, Jong Min Eun, Ho Geun Lee, Myoung Gyun Lee, Jeong Min Park and Jeong Joon Kim
Abstract: Recently, various new technologies such as Ai, IoT, cloud and big data are being developed in line with the 4th industrial revolution. As the amount of various sensor data based on IoT is increased, many techniques are required to collect and analyze the data. Therefore, we want to present the analysis results through processing of big data. In IoT, sensor data can be various kinds and quantities such as ultrasonic waves, infrared rays, cameras and vibrations. This type of informal data is difficult to obtain the desired analytical results when applied to a general analysis program. In this study, we implemented a system that processes informal data by collecting, storing, processing and analyzing data. We used Raspberry Pi in IoT and generated web server log data. The generated web server log data is collected in real time using flume, a collection solution of big data. Storage is stored in the HDFS of the hadoop solution and the unwanted properties are refined through processing solutions Hive and Pig. At the end of the final refine process, we analyzed the files with R programming and Spark.
Dong Jin Shin, Jong Min Eun, Ho Geun Lee, Myoung Gyun Lee, Jeong Min Park and Jeong Joon Kim, 2018. Big Data-based Log Collection and Analysis in IoT Environments. Journal of Engineering and Applied Sciences, 13: 1064-1072.