Andy Edmonds

The use of crowdsourcing platforms like Amazon Mechan-ical Turk for evaluating the relevance of search results has become an effective strategy that yields results quickly and inexpensively. One approach to ensure quality of worker judgments is to include an initial training period and sub-sequent sporadic insertion of predefined gold standard data (training data). Workers are notified or rejected when they err on the training data, and trust and quality ratings are ad-justed accordingly. In… Show more

The use of crowdsourcing platforms like Amazon Mechan-ical Turk for evaluating the relevance of search results has become an effective strategy that yields results quickly and inexpensively. One approach to ensure quality of worker judgments is to include an initial training period and sub-sequent sporadic insertion of predefined gold standard data (training data). Workers are notified or rejected when they err on the training data, and trust and quality ratings are ad-justed accordingly. In this paper, we assess how this type of dynamic learning environment can affect the workers' results in a search relevance evaluation task completed on Amazon Mechanical Turk. Specifically, we show how the distribution of training set answers impacts training of workers and aggregate quality of worker results. We conclude that in a relevance categorization task, a uniform distribution of la-bels across training data labels produces optimal peaks in 1) individual worker precision and 2) majority voting aggregate result accuracy. Show less

One of the most critical driving forces in the evolution of interfaces on the Internet has been the logging built into common Web servers and the decade-long deployment of analytics based upon this data source. Page-view logging has slowly moved to callback systems using client-side scripting to capture more aspects of the user experience. With the rise of JavaScript-based client-side interactivity and, more recently, asynchronous Javascript and XML (AJAX), server-side logging is less able… Show more

One of the most critical driving forces in the evolution of interfaces on the Internet has been the logging built into common Web servers and the decade-long deployment of analytics based upon this data source. Page-view logging has slowly moved to callback systems using client-side scripting to capture more aspects of the user experience. With the rise of JavaScript-based client-side interactivity and, more recently, asynchronous Javascript and XML (AJAX), server-side logging is less able to capture the user experience of Web sites and applications that are rising in complexity. We present a new technique for the in-page logging of interaction events that will help interaction designers make more informed design decisions based on how users are interacting with their systems. The potential benefit of our technique is demonstrated in a case study with a working system.
Show less

A reconciliation of hypertext principles with the evolution of hypermedia features on the web, very 2004!

The *definition* of personalization as being both individualization and contextualization.

Pitkow, James E., Schütze, Hinrich, Cass, Todd A., Cooley, Robert, Turnbull, Don, Edmonds, Andy, Adar, Eytan, Breuel, Thomas M. (2002): Personalized search. In Communications of the ACM, 45 (9) pp. 50-55. http://doi.acm.org/10.1145/567498.567526