Authors : S. Sravanthi and K. Thirupathi Rao
Abstract: Now a days the word MapReduce is synonymous with big data processing. Different flavors of it is available over Apache Hadoop being at the core of big data processing. For well versed Java developers there is the direct interaction with the core there is Hive for the SQL proficient one’s and there is Pig for procedural language aware developers. What ever the wrapper being used the core implementation of hadoop is simply is to divide the processing into two disjoint phases. One being the Map function and the other being the reduce function. So far many big data processing implementations are driven with the idea of equal distribution of workloads across processing nodes. We propose a dynamic distributed algorithm that is a processing aware job scheduler that assigns data processing nodes work load based on their prior performance throughputs. Extensive simulations using a 2.4 GB weather temperature conversion datasets demonstrates that our proposals can significantly reduce processing costs while prioritizing working nodes better compared to previous approaches.
S. Sravanthi and K. Thirupathi Rao, 2016. Efficient Big Data Analytics With Optimized Parallel Processing. International Journal of Soft Computing, 11: 312-318.