Amy Pavel

Amy Pavel headshot

Email apavel@cs.cmu.edu
Twitter @amypavel
Publications Google Scholar

Carnegie Mellon University
HCII, Accessibility@CMU
Postdoctoral Fellow

Apple, Inc., AI/ML Research
Research Scientist

I am currently a postdoctoral fellow at Carnegie Mellon University (supervised by Jeff Bigham) and a Research Scientist at Apple. I recently graduated with my PhD from the department of Electrical Engineering and Computer Science at UC Berkeley. I was advised by professors Björn Hartmann at UC Berkeley and Maneesh Agrawala at Stanford, and my doctoral work was supported by an NDSEG fellowship. I've previously been supported by a department EECS Excellence Award and a Sandisk Gold Fellowship.

In the summer of 2018, I served as the lecturer for CS160, the UC Berkeley user interface design and development course. Here is the course website.

I'm on the job market this year! Materials: CV · Research · Teaching · Diversity

Research Highlights

Rescribe

UIST 2020

On one side, a set of extended audio description sentences with their corresponding frames, and in the other column a set of inline audio description sentences and the same frames. The shortened descriptions are as follows: `Shots of lavender in a farmers market' is shortened to `Shots of lavender', `Red flowers against a white house and blue sky' is shortened to `Red flowers', `a courtyard and a pool' is not shortened, `Gaby bikes along a path' is shortened to `Close up of french fries', and `Close up of tater tots and french fries' is shortened to `Close up of french fries'

PDF | Video | Talk

360° Video

UIST 2017

A thumbnail of a paper figure indicating three different editing techniques for 360 video: traditional, viewpoint-oriented cuts and active reorientation.

PDF | Code | Video

SceneSkim

UIST 2015

PDF | Video

Video Digests

UIST 2014

A thumbnail of the Video Digests interface. Video thumbnails displayed alongside short summaries of the video content.

PDF | Demo | Code

Research Summary

My research leverages the connections between communication mediums (e.g., spoken audio, multiple video sources, text transcripts) to enable people to fully participate in and better work with digital communication. As a systems researcher in Human-Computer Interaction, I embed machine learning technologies (e.g., NLU, NLG) into new human interactions that I then deploy to test. Using my systems, remote content creators more effectively collaborate, video authors efficiently create accessible descriptions for blind users, and instructors help students to learn and retain key points. To inform future systems that capture what is important to domain experts and people with disabilities, I conduct and collaborate on in-depth qualitative (e.g., AAC communication, memes) and quantitative studies (e.g., 360° Video, VR Saliency). My long term research goal is to make communication more effective and accessible to all.

Conference Papers

A slide from a lecture where most text is in black but some of the text has been colored green to represent that it has been spoken by the presenter. There is an image with a squiggly circle brush on the slide, and it has a green border to indicate the presenter described the image.
Say It All: Feedback for Improving Non-Visual Presentation Accessibility
Yi-Hao Peng, JiWoong Jang, Jeffrey P. Bigham, Amy Pavel
The ACM Conference on Computer Human Interaction (Accepted to CHI 2021).
A YouTube search result augmented with accessibility information. A thumbnail of the video shows a woman in Rome. The rest of the video information reads as follows: Rome, Italy! Tips & Photo Spots by Ruby Keyvani. 5/7 Somewhat Accessible. 76% of the video is speech. The speech is descriptive but contains many visual references (3 per minute). Visual changes occur infrequently (5 shots per minute); few on-screen objects are described (20%).
What Makes a Video Accessible to Blind and Visually Impaired People?
Xingyu Liu, Patrick Carrington, Xiang "Anthony" Chen, Amy Pavel
The ACM Conference on Computer Human Interaction (Accepted to CHI 2021).
Small thumbnail image of the first table in the that contains examples of generated dialogue using semantic frames to control generation.
Controlling Dialogue Generation with Semantic Exemplars
Prakhar Gupta, Jeffrey P. Bigham, Yulia Tsvetkov, Amy Pavel
(Accepted to NAACL 2021).
A photograph taken at a workshop with a person using an AAC device, her close conversational partner, and puppeteers. All participants stand around a table full of craft supplies and are engaged in coversation.
Co-designing Socially Assistive Sidekicks for Motion-based AAC
Stephanie Valencia, Michal Luria, Amy Pavel, Jeffrey P. Bigham, Henny Admoni
The ACM Conference on Human Robot Interaction (Accepted to HRI 2021).
On one side, a set of extended audio description sentences with their corresponding frames, and in the other column a set of inline audio description sentences and the same frames. The shortened descriptions are as follows: `Shots of lavender in a farmers market' is shortened to `Shots of lavender', `Red flowers against a white house and blue sky' is shortened to `Red flowers', `a courtyard and a pool' is not shortened, `Gaby bikes along a path' is shortened to `Close up of french fries', and `Close up of tater tots and french fries' is shortened to `Close up of french fries'
Rescribe: Authoring and Automatically Editing Audio Descriptions
Amy Pavel, Gabriel Reyes, Jeffrey P. Bigham
ACM Symposium on User Interface Software and Technology (UIST 2020).

PDF | Video | Talk

Screenshot of a tweet by @CDCgov from April 1, 2020 3:55pm: Actions to reduce spread of the virus, such as social distancing, are key to #FlattenTheCurve. 2 of 3 (original tweet link: https://twitter.com/CDCgov/status/1245439600472084486) The tweet contains an image of the common public health infographic about “flattening the curve”, but the tweet did not include alt text for the image. The image shows an example of a common flatten the curve info-graphic. A tall peak indicates the height of the pandemic if left unchecked, and a shorter spread out curve depicts the effects of social distancing efforts.
Disability and the COVID-19 Pandemic: Using Twitter to Understand Accessibility during Rapid Societal Transition
Cole Gleason, Stephanie Valencia, Lynn Kirabo, Jason Wu, Anhong Guo, Elizabeth J. Carter, Jeffrey P. Bigham, Cynthia L. Bennett, Amy Pavel
ACM Conference on Computers and Accessibility (ASSETS 2020).

PDF

Two example prototypes for making AR apps accessible. A: Foundational Accessibility. Screenshot of a virtual chair with a voice over target around it, a speech bubble shows the app announcing 'Back of chair with blue cushion'. B: Scanning. Screenshot of AR grid overlaid on a coffee table. Speech bubbles show the app announcing 'Found a new horizontal surface' and 'Scanned 2 surfaces totaling 2.3 square meters'.
Making Mobile Augmented Reality Applications Accessible
Jaylin Herskovitz, Jason Wu, Samuel White, Amy Pavel, Gabriel Reyes, Anhong Guo, Jeffrey P. Bigham
ACM Conference on Computers and Accessibility (ASSETS 2020).

PDF | Video

A cartoon stylized version of a popular reaction GIF of Oprah Winfrey shrugging. She turns to look to the camera, glances to the side, stares at the camera, then shrugs with her palms up.
Making GIFs Accessible
Cole Gleason, Amy Pavel, Himalini Gururaj, Kris M. Kitani, Jeffrey P. Bigham
ACM Conference on Computers and Accessibility (ASSETS 2020).

Paper | Link to Thumbnail GIF

A thumbnail with a conversation between a person using an AAC device and two other people.
Conversational Agency in Augmentative and Alternative Communication
Stephanie Valencia, Amy Pavel, Jared Santa Maria, Seunga (Gloria) Yu, Jeffrey P. Bigham, Henny Admoni
The ACM Conference on Computer Human Interaction (CHI 2020).

PDF

A thumbnail with illegible examples of tweet images and their corresponding alt text.
Twitter A11y: A Browser Extension to Make Twitter Images Accessible
Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington, Kris M. Kitani, Jeffrey P. Bigham
The ACM Conference on Computer Human Interaction (CHI 2020).

PDF

A thumbnail of a paper figure showing three memes with descriptions.
Making Memes Accessible
Cole Gleason, Amy Pavel, Xingyu Liu, Patrick Carrington, Lydia Chilton, Jeffrey P. Bigham
ACM Conference on Computers and Accessibility (ASSETS 2019).

PDF | Time article

A thumbnail of a paper figure showing three memes with descriptions.
Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, Jeffrey P. Bigham
SIGDIAL Meeting on Discourse and Dialogue (SIGDIAL 2019).

PDF | Code

A thumbnail of a paper figure including nine images with heat maps indicating visual saliency in VR.
Saliency in VR: How do people explore virtual environments?
Vincent Sitzmann*, Ana Serrano*, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, Gordon Wetzstein
IEEE Transactions on Visualization and Computer Graphics (VR 2018).

Project Page | Code | Video

A thumbnail of a paper figure indicating three different editing techniques for 360 video: traditional, viewpoint-oriented cuts and active reorientation.
Shot Orientation Controls for Interactive Cinematography with 360 Video
Amy Pavel, Björn Hartmann, Maneesh Agrawala
ACM 30th Symposium on User Interface Software and Technology (UIST 2017).

PDF | Code | Video

A thumbnail of the VidCrit interface with a video on the left and text critiques of the video on the right.
VidCrit: Video-Based Asynchronous Video Review
Amy Pavel, Dan B Goldman, Björn Hartmann, Maneesh Agrawala
ACM 29th Symposium on User Interface Software and Technology (UIST 2016).

PDF | Video

SceneSkim: Searching and Browsing Movies Using Synchronized Captions, Scripts and Plot Summaries
Amy Pavel, Dan B Goldman, Björn Hartmann, Maneesh Agrawala
ACM 28th Symposium on User Interface Software and Technology (UIST 2015).

PDF | Video

A thumbnail indicating an overview of the CrowdCrit critique process.
Structuring, Aggregating, and Evaluating Crowdsourced Design Critique
Kurt Luther, Jari-lee Tolentino, Wei Wu, Amy Pavel, Maneesh Agrawala, Brian P Bailey, Björn Hartmann, Steven Dow
The 18th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2015).
A thumbnail of the Video Digests interface. Video thumbnails displayed alongside short summaries of the video content.
Video Digests: A Browsable, Skimmable Format for Informational Lecture Videos
Amy Pavel, Colorado Reed, Björn Hartmann, Maneesh Agrawala
ACM 27th Symposium on User Interface Software and Technology (UIST 2014).

PDF | Demo | Code

Thesis and Technical Reports

Title page of dissertation
Navigating Video Using Structured Text
Amy Pavel
PhD in Computer Science, University of California, Berkeley
Advisors: Bjoern Hartmann and Maneesh Agrawala
Additional committee members: Eric Paulos, Abigail De Kosnik
A thumbnail of the Sifter interface for browsing common sets of image editing commands.
Browsing and Analyzing the Command-Level Structure of Large Collections of Image Manipulation Tutorials
Amy Pavel, Floraine Berthouzoz, Björn Hartmann, Maneesh Agrawala
Berkeley EECS Technical Report and Poster Presentation (TR 2013, TECHCON 2013).

Posters and workshops

A thumbnail showing a line graph of the precision-at-one of an algorithm going up as the number of noteworthy sentences considered rises. After a certain point, the number of noteworthy sentences decreases the precision at one -- indicating that lower quality noteworthy sentences add noise rather than value to the prediction.
Extracting Structured Data from Doctor-Patient Conversations By Predicting Noteworthy Utterances
Kundan Krishna, Amy Pavel, Benjamin Schloss, Jeffrey P. Bigham, Zachary Lipton
AAAI Workshop on Health Intelligence (W3PHIAI 2020 Workshop Paper).
A thumbnail with an illegible system diagram of Twitter A11y.
Twitter A11y: A Browser Extension to Make Twitter Images Accessible
Christina Low*, Emma McCamey*, Cole Gleason, Patrick Carrington, Jeffrey P. Bigham, Amy Pavel
The 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2019 Poster).
A thumbnail indicating an overview of the CrowdCrit critique process.
CrowdCrit: crowdsourcing and aggregating visual design critique
Kurt Luther, Wei Wu, Amy Pavel, Jari-lee Tolentino, Maneesh Agrawala, Björn Hartmann, Steven Dow
The 18th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2014 Demo).

Work

Postdoctoral Fellow — Carnegie Mellon University

Advisor: Jeffrey P. Bigham January 2019 - Present

Research Scientist — Apple

Group: Accessibility/ML Research
July 2019 - Present

Graduate student researcher — UC Berkeley

Advisors: Björn Hartmann, Maneesh Agrawala Committee: Eric Paulos, Björn Hartmann, Maneesh Agrawala, Abigail de Kosnik
August 2013 - December 2018

Undergraduate research assistant — UC Berkeley

Advisors: Maneesh Agrawala, Björn Hartmann, Floraine Berthouzoz
May 2011 - May 2013

Research intern — Adobe Research, Seattle

Advisor: Dan Goldman
May 2014 - August 2014, May 2015 - August 2015

Teaching

Instructor — UC Berkeley

CS 160: User interface design and development
Summer 2018

Graduate Student Instructor (TA) — UC Berkeley

CS 160: User interface design and development
Summer 2017

Project Advisor — UC Berkeley

NWMEDIA 190: Making Sense of Cultural Data
Fall 2017

Instructor (Workshop) — UC Berkeley

CS Kickstart, intro CS for incoming freshmen women
Summer 2012