skip to Main Content

UC San Diego Statement in Support of Asian Communities Impacted by Shootings in Atlanta

View Here

ucsd design lab edith law harvard university

Edith Law

February 12, 2014 12:00PM


Design Techniques for Crowdsourcing Complex Tasks

Human computation (a.k.a. crowdsourcing) systems are theoretically interesting because they challenge the way we currently think about and build intelligent systems. We now have to design the system to take into account factors that affect how people compute, including their motivation, cognitive limitations and expertise. Having access to both automated algorithms and many human computers also means that, as system designers, we must explicitly reason about the division of labor — between novices, experts, and machines — that will lead to the best computational outcomes.
There are numerous examples of human computation systems achieving remarkable feats — massively and rapidly labeling images (e.g., the ESP Game), digitizing books (e.g., reCAPTCHA), folding proteins (e.g., FoldIt), translating text (e.g., Duolingo). Yet, many of the problems tackled through crowdsourcing are simple, in that they require only basic perceptual abilities and common-sense knowledge, or that they can be handled by independent workers each having only a local view of solution. In this talk, I will describe several general design techniques for crowdsourcing complex tasks and specific examples of their use in developing a variety of human computation systems, including games with a purpose, and social computing platforms for planning and text summarization.
Can we extend existing crowdsourcing models to handle tasks that require substantially more expertise, such as research tasks involving the collection, annotation and analysis of scientific data? How can we lower the barrier of entry for scientists, who are domain experts but not necessarily technically savvy or familiar with crowdsourcing, to use crowdsourcing as a tool for their research? I will conclude by describing my research agenda on mixed-expertise crowdsourcing in the scientific domain, and a citizen science platform and research infrastructure, called Curio, for exploring an entirely new space of complex problems that can benefit from leveraging contributions from expert communities and non-expert crowds.


Edith Law is a Center for Research on Computation and Society (CRCS) postdoctoral fellow at Harvard University. CRCS comprises of faculty from Harvard’s Computer Science Department and is affiliated with the Beckman Center for Internet and Society. Law’s research at Harvard focuses on understanding the role of expertise and machine intelligence in human computation systems, and how these systems can address problems in science and health care. She is working on Curio, a micro-task platform for crowdsourcing research tasks in the sciences and humanities, and Simplyput, a crowdsourcing project for improving health literacy and communication. Law graduated with a Ph.D. in Machine Learning at Carnegie Mellon University, where she worked with Tom Mitchell and Luis von Ahn on hybrid human and machine computation systems. Her research was supported in part by a Microsoft Graduate Research Fellowship.

Read Next

Ucsd Design Lab Betg Altringer Harvard University

Beth Altringer (Harvard University)

February 26, 2014
Behavior for Better Innovation: Underrepresented obstacles to success in collaborative innovation projects
CSB 003 on the UCSD Campus
Ucsd Design Lab Bjorn Hartmann Uc Berkeley

Bjorn Hartmann (UC Berkeley)

Wednesday, May 17, 2017 at 4:00 PM
Interactive Design Tools for the Maker Movement
CSE1202 on the UCSD Campus
Ucsd Design Lab Darren Gergle Northwestern University

Darren Gergle (Northwestern University)

October 17, 2013
Understanding and Bridging the Wikipedia Language Gap
Back To Top