Amy Pavel

Amy Pavel headshot

Assistant Professor
University of Texas at Austin
Computer Science

Email apavel@cs.utexas.edu

Office GDC 3.704

Twitter @amypavel

Publications Google Scholar

Curriculum Vitae PDF

People
Ph.D. Students: Mina Huh, Karim Benharrak, Ananya Gubbi Mohanbabu, Meng Chen, Yi-Hao Peng (co-advised with Jeffrey P. Bigham)
Masters and Undergraduates: Aadit Barua, Akhil Iyer, Katie Clark, Sumaya Al-Bedaiwi, Ujjaini Das, Jerry He, Sarah Zheng
Recent Alumni: Doeun Lee, Pranav Venkatesh, Tess Van Daele, Yuning Zhang, Daniel Killough, Jalyn Derry, Aochen Jiao, Soumili Kole, Chitrank Gupta

I am an Assistant Professor in the Department of Computer Science at The University of Texas at Austin. Before I joined UT in 2022, I was a postdoc at Carnegie Mellon University (supervised by Jeff Bigham) and a Research Scientist at Apple. I received my PhD from the department of Electrical Engineering and Computer Science at UC Berkeley, advised by professors Björn Hartmann at UC Berkeley and Maneesh Agrawala at Stanford.

I regularly teach a Computer Science class covers the design and development of user interfaces (Introduction to Human-Computer Interaction). Prior versions of this class include CS160 at UC Berkeley in Summer 2018 and CS378 at UT Austin in Spring 2022.

Research Highlights

GenAssist

UIST 2023 – Best Paper Award

A thumbnail depicting four generated image options of a young chef cooking dinner and their descriptions.

PDF | Project Page

CrossA11y

UIST 2022 – Best Paper Award

A thumbnail depicting CrossA11y's computational approach that compares video frames with the corresponding audio.

PDF | Project Page | Video

Rescribe

UIST 2020

On one side, a set of extended audio description sentences with their corresponding frames, and in the other column a set of inline audio description sentences and the same frames. The shortened descriptions are as follows: `Shots of lavender in a farmers market' is shortened to `Shots of lavender', `Red flowers against a white house and blue sky' is shortened to `Red flowers', `a courtyard and a pool' is not shortened, `Gaby bikes along a path' is shortened to `Close up of french fries', and `Close up of tater tots and french fries' is shortened to `Close up of french fries'

PDF | Video | Talk

Long-Form VQA

COLM 2024 – Oral Spotlight

A thumbnail of longform visual question answers across humans, GPT-4V and Gemini. Each answer sentence is labeled with a discourse role (e.g., confirmation, answer) and information source (e.g., image quality, image content).

Project Page

Research Summary

As a systems researcher in Human-Computer Interaction and Accessibility, I embed machine learning technologies (e.g., Natural Language Processing) into new human interactions that I then deploy to test. Using my systems, remote content creators more effectively collaborate, video authors efficiently create accessible descriptions for blind users, and instructors help students to learn and retain key points. To inform future systems that capture what is important to domain experts and people with disabilities, I also conduct and collaborate on in-depth qualitative (e.g., AAC communication, memes) and quantitative studies (e.g., 360° Video, VR Saliency). My research goal is to make communication effective and accessible.

Research Papers

A thumbnail of longform visual question answers across humans, GPT-4V and Gemini. Each answer sentence is labeled with a discourse role (e.g., confirmation, answer) and information source (e.g., image quality, image content).
Long-Form Answers to Visual Questions from Blind and Low Vision People

Mina Huh, Fangyuan Xu, Yi-Hao Peng, Chongyan Chen, Hansika Murugu, Danna Gurari, Eunsol Choi, Amy Pavel

COLM 2024

Oral Spotlight

A thumbnail of an interface for browsing potentially harmful video segments.
Design Considerations for Photosensitivity Warnings in Visual Media

Laura South, Caglar Yildirim, Amy Pavel, Michelle A. Borkin

ASSETS 2024

A thumbnail of a couch listing image with a context free description that does not contain much information about the couch, and a context-aware description that focuses on the couch.
Context-Aware Image Descriptions for Web Accessibility

Ananya Gubbi Mohanbabu, Amy Pavel

ASSETS 2024

A thumbnail of DreamStruct's UI generation that starts with a UI description, then generates HTML UI code, then generates images for the UI and outputs the final user interface.
DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation

Yi-Hao Peng, Faria Huq, Yue Jiang, Jason Wu, Amanda Xin Yue Li, Jeffrey P. Bigham, Amy Pavel

ECCV 2024

A thumbnail of 5 websites before and after using DesignChecker. The after websites have improved aesthetically according to design pricinples (e.g., color harmony, alignment).
DesignChecker: Visual Design Support for Blind and Low Vision Web Developers

Mina Huh, Amy Pavel

UIST 2024

A thumbnail of a table of potential accessibility barriers in VR.
Barriers to Photosensitive Accessibility in Virtual Reality

Laura South, Caglar Yildirim, Amy Pavel, Michelle A. Borkin

CHI 2024

Honorable Mention Award

A thumbnail of the ShortScribe pipeline. The pipeline takes the video as input, transcribes it and describes the keyframes, then uses GPT-4 to generate multiple types of descriptions.
ShortScribe: Making Short-Form Videos Accessible with Hierarchical Video Summaries

Tess Van Daele, Akhil Iyer, Yuning Zhang, Jalyn Derry, Mina Huh, Amy Pavel

CHI 2024

A thumbnail of the COMPA interface that features views for the AAC user and their conversational partners.
COMPA: Using Conversation Context to Achieve Common Ground in AAC

Stephanie Valencia, Jessica Huynh, Emma Y Jiang, Yufei Wu, Teresa Wan, Zixuan Zheng, Henny Admoni, Jeffrey P. Bigham, Amy Pavel

CHI 2024

GeoLatent: A Geometric Approach to Latent Space Design for Deformable Shape Generators

Haitao Yang, Bo Sun, Liyan Chen, Amy Pavel, Qixing Huang

SIGGRAPH ASIA 2023

A thumbnail depicting four generated image options of a young chef cooking dinner and their descriptions.
GenAssist: Making Image Generation Accessible

Mina Huh, Yi-Hao Peng, Amy Pavel

UIST 2023

Best Paper Award

A thumbnail of a livestream paired with a set of descriptions with timestamps.
Exploring Community-Driven Descriptions for Making Livestreams Accessible

Daniel Killough, Amy Pavel

ASSETS 2023

A thumbnail of the audio visual script that includes a transcript of the video coupled with detected errors in the video footage (e.g., camera blur).
AVscript: Accessible Video Editing with Audio-Visual Scripts

Mina Huh, Saelyne Yang, Yi-Hao Peng, Xiang "Anthony" Chen, Young-Ho Kim, Amy Pavel

CHI 2023

A thumbnail of SlideSpecs featuring numbered slides and audience feedback on the slides.
SlideSpecs: Automatic and Interactive Presentation Feedback Collation

Jeremy Warner, Amy Pavel, Tonya Nguyen, Maneesh Agrawala, Björn Hartmann

IUI 2023

A thumbnail depicting CrossA11y's computational approach that compares video frames with the corresponding audio.
CrossA11y: Identifying Video Accessibility Issues via Cross-modal Grounding

Xingyu Liu, Ruolin Wang, Dingzeyu Li, Xiang "Anthony" Chen, Amy Pavel

UIST 2022

Best Paper Award

A thumbnail of Diffscriber's interface for reviewing slide changes.
Diffscriber: Describing Visual Design Changes to Support Mixed-Ability Collaborative Presentation Authoring

Yi-Hao Peng, Jason Wu, Jeffrey P. Bigham, Amy Pavel

UIST 2022

Three images of Tech Help Desk including the Tech Help Desk classroom, the Tech Help Desk sign on the classroom door and the building entrance.
Tech Help Desk: Support for Local Entrepreneurs Addressing the Long Tail of Computing Challenges

Yasmine Kotturi, Herman T Johnson, Michael Skirpan, Sarah E Fox, Jeffrey P. Bigham, Amy Pavel

CHI 2022

A thumbnail image of a chart of alt text coverage that appears in the paper.
Toward supporting quality alt text in computing publications

Candace Williams, Lilian de Greef, Ed Harris III, Amy Pavel, Cynthia L. Bennett

W4A 2022

A thumnbail image of the tutorial lens interface.
TutorialLens: authoring Interactive augmented reality tutorials through narration and demonstration

Junhan Kong, Dena Sabha, Jeffrey P. Bigham, Amy Pavel, Anhong Guo

SUI 2021

A thumnail image of SlideCho's interface for surfacing information in slide videos.
Slidecho: Flexible Non-Visual Exploration of Presentation Videos

Yi-Hao Peng, Jeffrey P. Bigham, Amy Pavel

ASSETS 2021

A thumnail image of a semantic exemplar with three corresponding examples.
Controlling Dialogue Generation with Semantic Exemplars.

Prakhar Gupta, Jeffrey P. Bigham, Yulia Tsvetkov, Amy Pavel

NAACL 2021

A slide from a lecture where most text is in black but some of the text has been colored green to represent that it has been spoken by the presenter. There is an image with a squiggly circle brush on the slide, and it has a green border to indicate the presenter described the image.
Say It All: Feedback for Improving Non-Visual Presentation Accessibility

Yi-Hao Peng, JiWoong Jang, Jeffrey P. Bigham, Amy Pavel

CHI 2021

A YouTube search result augmented with accessibility information. A thumbnail of the video shows a woman in Rome. The rest of the video information reads as follows: Rome, Italy! Tips & Photo Spots by Ruby Keyvani. 5/7 Somewhat Accessible. 76% of the video is speech. The speech is descriptive but contains many visual references (3 per minute). Visual changes occur infrequently (5 shots per minute); few on-screen objects are described (20%).
What Makes a Video Non-Visually Accessible?

Xingyu Liu, Patrick Carrington, Xiang "Anthony" Chen, Amy Pavel

CHI 2021

A photograph taken at a workshop with a person using an AAC device, her close conversational partner, and puppeteers. All participants stand around a table full of craft supplies and are engaged in coversation.
Co-designing Socially Assistive Sidekicks for Motion-based AAC

Stephanie Valencia, Michal Luria, Amy Pavel, Jeffrey P. Bigham, Henny Admoni

HRI 2021

On one side, a set of extended audio description sentences with their corresponding frames, and in the other column a set of inline audio description sentences and the same frames. The shortened descriptions are as follows: `Shots of lavender in a farmers market' is shortened to `Shots of lavender', `Red flowers against a white house and blue sky' is shortened to `Red flowers', `a courtyard and a pool' is not shortened, `Gaby bikes along a path' is shortened to `Close up of french fries', and `Close up of tater tots and french fries' is shortened to `Close up of french fries'
Rescribe: Authoring and Automatically Editing Audio Descriptions.

Amy Pavel, Gabriel Reyes, Jeffrey P. Bigham

UIST 2020

Two example prototypes for making AR apps accessible. A: Foundational Accessibility. Screenshot of a virtual chair with a voice over target around it, a speech bubble shows the app announcing 'Back of chair with blue cushion'. B: Scanning. Screenshot of AR grid overlaid on a coffee table. Speech bubbles show the app announcing 'Found a new horizontal surface' and 'Scanned 2 surfaces totaling 2.3 square meters'.
Making Mobile Augmented Reality Applications Accessible.

Jaylin Herskovitz, Jason Wu, Samuel White, Amy Pavel, Gabriel Reyes, Anhong Guo, Jeffrey P. Bigham

ASSETS 2020

A cartoon stylized version of a popular reaction GIF of Oprah Winfrey shrugging. She turns to look to the camera, glances to the side, stares at the camera, then shrugs with her palms up.
Making GIFs Accessible.

Cole Gleason, Amy Pavel, Himalini Gururaj, Kris M. Kitani, Jeffrey P. Bigham

ASSETS 2020

Screenshot of a tweet by @CDCgov from April 1, 2020 3:55pm: Actions to reduce spread of the virus, such as social distancing, are key to #FlattenTheCurve. 2 of 3 (original tweet link: https://twitter.com/CDCgov/status/1245439600472084486) The tweet contains an image of the common public health infographic about “flattening the curve”, but the tweet did not include alt text for the image. The image shows an example of a common flatten the curve info-graphic. A tall peak indicates the height of the pandemic if left unchecked, and a shorter spread out curve depicts the effects of social distancing efforts.
Disability and the COVID-19 Pandemic: Using Twitter to Understand Accessibility during Rapid Societal Transition.

Cole Gleason, Stephanie Valencia, Lynn Kirabo, Jason Wu, Anhong Guo, Elizabeth J. Carter, Jeffrey P. Bigham, Cynthia L. Bennett, Amy Pavel

ASSETS 2020

A thumbnail with illegible examples of tweet images and their corresponding alt text.
Twitter A11y: A Browser Extension to Make Twitter Images Accessible.

Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington, Kris M. Kitani, Jeffrey P. Bigham

CHI 2020

Honorable Mention Award

A thumbnail with a conversation between a person using an AAC device and two other people.
Conversational Agency in Augmentative and Alternative Communication.

Stephanie Valencia, Amy Pavel, Jared Santa Maria, Seunga (Gloria) Yu, Jeffrey P. Bigham, Henny Admoni

CHI 2020

Honorable Mention Award

A thumbnail of a paper figure showing three memes with descriptions.
Making Memes Accessible.

Cole Gleason, Amy Pavel, Xingyu Liu, Patrick Carrington, Lydia Chilton, Jeffrey P. Bigham

ASSETS 2019

A thumbnail of a paper figure showing three memes with descriptions.
Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References.

Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, Jeffrey P. Bigham

SIGDIAL 2019

A thumbnail of a paper figure including nine images with heat maps indicating visual saliency in VR.
Saliency in VR: How do people explore virtual environments?

Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, Gordon Wetzstein

IEEE VR 2018

A thumbnail of a paper figure indicating three different editing techniques for 360 video: traditional, viewpoint-oriented cuts and active reorientation.
Shot Orientation Controls for Interactive Cinematography with 360 video.

Amy Pavel, Björn Hartmann, Maneesh Agrawala

UIST 2017

A thumbnail of the VidCrit interface with a video on the left and text critiques of the video on the right.
Vidcrit: Video-based Asynchronous Video Review.

Amy Pavel, Dan B Goldman, Björn Hartmann, Maneesh Agrawala

UIST 2016

SceneSkim: Searching and Browsing Movies Using Synchronized Captions, Scripts and Plot Summaries.

Amy Pavel, Dan B Goldman, Björn Hartmann, Maneesh Agrawala

UIST 2015

A thumbnail indicating an overview of the CrowdCrit critique process.
Structuring, Aggregating, and Evaluating Crowdsourced Design Critique.

Kurt Luther, Jari-lee Tolentino, Wei Wu, Amy Pavel, Brian P Bailey, Maneesh Agrawala, Björn Hartmann, Steven Dow

CSCW 2015

A thumbnail of the Video Digests interface. Video thumbnails displayed alongside short summaries of the video content.
Video Digests: A Browsable, Skimmable Format for Informational Lecture Videos.

Amy Pavel, Colorado Reed, Björn Hartmann, Maneesh Agrawala

UIST 2014

Thesis and Technical Reports

Navigating Video Using Structured Text

Amy Pavel

PhD in Computer Science, University of California, Berkeley

Advisors: Bjoern Hartmann and Maneesh Agrawala

Additional committee members: Eric Paulos, Abigail De Kosnik

Browsing and Analyzing Command Structure of Large Collections of Image Manipulation Tutorials.

Amy Pavel, Floraine Berthouzoz, Björn Hartmann, Maneesh Agrawala

UC Berkeley Technical Report, EECS-2013-167

Posters and Workshops

A thumbnail screenshot of a forum people use to report flashing lights.
Exploratory Thematic Analysis of Crowdsourced Photosensitivity Warnings

Laura South, Caglar Yildirim, Amy Pavel, Michelle A. Borkin

CHI 2023 (Extended Abstract)

A thumbnail with an illegible system diagram of Twitter A11y.
Twitter A11y: A Browser Extension to Make Twitter Images Accessible.

Christina Low, Emma McCamey, Cole Gleason, Amy Pavel, Emma McCamey, Patrick Carrington, Jeffrey P. Bigham

ASSETS 2019

A thumbnail showing a line graph of the precision-at-one of an algorithm going up as the number of noteworthy sentences considered rises. After a certain point, the number of noteworthy sentences decreases the precision at one -- indicating that lower quality noteworthy sentences add noise rather than value to the prediction.
Extracting Structured Data from Doctor-Patient Conversations By Predicting Noteworthy Utterances.

Kundan Krishna, Amy Pavel, Benjamin Schloss, Jeffrey P. Bigham, Zachary Lipton

W3PHIAI 2020 Workshop Paper

A thumbnail indicating an overview of the CrowdCrit critique process.
CrowdCrit: Crowdsourcing and Aggregating Visual Design Critique.

Kurt Luther, Amy Pavel, Wei Wu, Jari-lee Tolentino, Maneesh Agrawala, Björn Hartmann, Steven Dow

CSCW 2014

A thumbnail of the Sifter interface for browsing common sets of image editing commands.
Sifter: Analyzing and Exploring Large Collections of Web-Based Image Manipulation Tutorials.

Amy Pavel, Floraine Berthouzoz, Björn Hartmann, Maneesh Agrawala

TECHCON 2012

Work

Assistant Professor — University of Texas at Austin

Department of Computer Science

January 2022 —

Visiting Faculty Researcher — Google

Google Research

Remote, 20% appointment

October 2024 —

Research Scientist (50% time) — Apple Inc

AI/ML

Machine Intelligence Accessibility Group

July 2019 — January 2022

Postdoctoral Fellow (50% time) — Carnegie Mellon University

HCII

Supervised by Professor Jeffrey P. Bigham

January 2019 — October 2021

Graduate Researcher — UC Berkeley

Visual Computing Lab

Advised by Professors Björn Hartmann and Maneesh Agrawala

September 2013 — January 2019

Research Intern — Adobe

Creative Technologies Lab

Advised by Principal Scientist Dan Goldman

Summer 2014, Summer 2015

Undergraduate Researcher — UC Berkeley

BiD Lab, Visual Computing Lab

Advised by Professors Björn Hartmann and Maneesh Agrawala

June 2011 — September 2013

Teaching

Instructor — UT Austin

CS 395T: Human-Computer Interaction Research

Fall 2024

Instructor — UT Austin

CS 378: Introduction to Human-Computer Interaction

Spring 2024

Instructor — UT Austin

CS 395T: Human-Computer Interaction Research

Fall 2023

Instructor — UT Austin

CS 378: Introduction to Human-Computer Interaction

Spring 2023

Instructor — UT Austin

CS 378: Introduction to Human-Computer Interaction

Spring 2022

Instructor — UC Berkeley

CS 160: User interface design and development

Summer 2018

Graduate student instructor — UC Berkeley

CS 160: User interface design and development

Summer 2017

Student project advisor — UC Berkeley

NWMEDIA 190: Making Sense of Cultural Data

Fall 2017

Instructor — UC Berkeley

CS Kickstart, intro CS for incoming freshmen women

Summer 2012

Teacher — UC Berkeley

Berkeley Engineers and Mentors

2009 - 2010