Edith Law (Harvard University)

Design Techniques for Crowdsourcing Complex Tasks

Human computation (a.k.a. crowdsourcing) systems are theoretically interesting because they challenge the way we currently think about and build intelligent systems. We now have to design the system to take into account factors that affect how people compute, including their motivation, cognitive limitations and expertise. Having access to both automated algorithms and many human computers also means that, as system designers, we must explicitly reason about the division of labor — between novices, experts, and machines — that will lead to the best computational outcomes.
There are numerous examples of human computation systems achieving remarkable feats — massively and rapidly labeling images (e.g., the ESP Game), digitizing books (e.g., reCAPTCHA), folding proteins (e.g., FoldIt), translating text (e.g., Duolingo). Yet, many of the problems tackled through crowdsourcing are simple, in that they require only basic perceptual abilities and common-sense knowledge, or that they can be handled by independent workers each having only a local view of solution. In this talk, I will describe several general design techniques for crowdsourcing complex tasks and specific examples of their use in developing a variety of human computation systems, including games with a purpose, and social computing platforms for planning and text summarization.
Can we extend existing crowdsourcing models to handle tasks that require substantially more expertise, such as research tasks involving the collection, annotation and analysis of scientific data? How can we lower the barrier of entry for scientists, who are domain experts but not necessarily technically savvy or familiar with crowdsourcing, to use crowdsourcing as a tool for their research? I will conclude by describing my research agenda on mixed-expertise crowdsourcing in the scientific domain, and a citizen science platform and research infrastructure, called Curio, for exploring an entirely new space of complex problems that can benefit from leveraging contributions from expert communities and non-expert crowds.

About the Speaker

Edith Law is a Center for Research on Computation and Society (CRCS) postdoctoral fellow at Harvard University. CRCS comprises of faculty from Harvard’s Computer Science Department and is affiliated with the Beckman Center for Internet and Society. Law’s research at Harvard focuses on understanding the role of expertise and machine intelligence in human computation systems, and how these systems can address problems in science and health care. She is working on Curio, a micro-task platform for crowdsourcing research tasks in the sciences and humanities, and Simplyput, a crowdsourcing project for improving health literacy and communication. Law graduated with a Ph.D. in Machine Learning at Carnegie Mellon University, where she worked with Tom Mitchell and Luis von Ahn on hybrid human and machine computation systems. Her research was supported in part by a Microsoft Graduate Research Fellowship.