Crowdsourced Data Simulator

A program to simulate a synthetic dataset of annotations from a crowd to a set of multiple-choice questions. It accepts a configuration of indicators on the crowd, such as fraction of people with high accuracy, and the fraction of malicious labelers, and generates responses. Both single answer and multiple answer questions are supported. It can be used for experimenting with crowdsourced aggregation algorithms, such as the Dawid-Skene algorithm.

The program accepts classes of people, each having two parameters - (1) their percentage in the population, and (2) their range of probabilities of getting the answer to any question correctly. For example, one can specify that 20% of the population gets the answer correctly with probability between 0.6 and 0.7. The input also includes the number of questions to simulate, the number of options per question, the number of annotators to simulate, and whether the setting is single-answer or multiple-answer. It then provides simulated crowd labels and ground truths.

Instructions on setup, usage, and examples can be found on GitHub.

The program does not account for the difficulty of questions, and the simulated labels might be sometimes unnatural because of the predictable nature of the annotators. However, it could still be very useful to quickly generate synthetic data for testing aggregation algorithms.

This program was used for experimenting in the initial stages for the Fast Dawid-Skene project.

This program has been implemented in Python.