Description

It is cost-efficient for a tenant with a limited budget to establish a vir tual MapReduce cluster by renting multiple virtual priva te ser vers (VPSs) from a VPS provider. To provide an appropriate scheduling scheme for this type of computing environment, we propose in this paper a hybrid job-driven scheduling scheme (JoSS for shor t) from a tenant’s perspective. JoSS provides not only job-level scheduling, but also map-task level scheduling and reduce-task level scheduling. JoSS classifies MapReduce jobs based on job scale and job type and designs an appropriate scheduling policy to schedule each class of jobs. The goal is to improve da ta locality for both map tasks and reduce tasks, avoid job star va tion, and improve job execution perfor mance. Two variations of JoSS are fur ther introduced to separately achieve a better map-data locality and a faster task assignment. We conduct extensive experiments toevalua te and compare the two variations with cur rent scheduling algorithms suppor ted by Hadoop. The results show that the two variations outperfor m the other tested algorithms in ter ms of map-data locality, reduce-da ta locality, and networ k overhead without incur ring significant overhead. In addition, the two variations are separately suitable for dif ferent MapReduce-work load scenarios and provide the best job perfor mance among all tested algorithms.