Description

MapReduce is a programming paradigm that makes it simple and efficient to process vast amount of data. It targets at very big clusters, where failures are no longer exceptions. Fault tolerance is vital to MapReduce, however, fault tolerance and recovery strategies in MapReduce perform poorly under failures. Currently fault tolerance is implemented at the task level, a task failure will lead to a re-execution of the whole task. Big data analysis has gained significant popularity within the last few years. The MapReduce framework presented by Google makes it easier to write applications that process vast amount of data. MapReduce targets at large commodity clusters where failures are not exceptions. Google’s MapReduce programming model serves for processing large data sets in a massively parallel manner. MapReduce is emerging as an important programming model for large-scale data-parallel applications such as web indexing, data mining, and scientific simulation. Hadoop is an open-source implementation of MapReduce enjoying wide adoption and is often used for short jobs where low response time is critical. Hadoop’s performance is closely tied to its task scheduler, which implicitly assumes that cluster nodes are homogeneous and tasks make progress linearly, and uses these assumptions to decide when to speculatively re-execute tasks that appear to be stragglers. This project presents BeTL which introduces slight changes to the execution flow of MapReduce, and makes it possible to gain a finer-grained fault tolerance.