Effective Crowdsourcing Ranking Solution: Performance-Aware Working Group Selection and Efficient Result Aggregation

Crowdsourcing can harness grassroots intelligence to effectively solve complex ranking tasks. Two key issues for these tasks are how to recruit high-performance working groups and how to efficiently aggregate their submissions. In past work, many attempts have been made to maximize social benefits or motivate worker participation, without considering the performance and efficiency of task completion. Therefore, we investigate an effective crowdsourcing ranking solution which is concerned with the two problems of optimal working group selection and efficient result aggregation. A performance-aware working group selection method is proposed to solve the high time complexity of combinatorial optimization in working group selection with a fixed number of requester recruits. To improve the performance of aggregated results more efficiently, an efficient aggregation method based on differential evolution algorithm and Top-k pruning is proposed. Finally, the effectiveness of the proposed methods is verified by extensive experiments compared with recent related methods.