Anna Maria Feit

Email:

feit@cs.uni-saarland.de

ABOUT ME

I am a researcher in Human-Computer Interaction and a full Professor at the Department of Computer Science at Saarland University, Germany where I lead the Computational Interaction group.

Previously, I was a postdoctoral researcher at the AIT Group of Prof. Otmar Hilliges at ETH Zurich. I received a Doctoral degree from Aalto University in Helsinki where I worked with Prof. Antti Oulasvirta in the User Interfaces group. Before that, I studied Computer Science at Saarland University, Germany.


My Research

My research interests are in the area of computational interaction. I use methods of optimization, machine learning, and mathematical modeling to understand and improve our interaction with computers. Among other forms of input, I am particularly interested in text input methods and have conducted several studies to understand how people type, in small lab settings and through large-scale online typing tests.

I have applied this approach in several projects, including the development of the new French keyboard standard. During an internship at Microsoft Research in Redmond, I worked with the MSR Enable Group and Meredith Ringel Morris on characterizing the accuracy and precision of eye tracking during practical use and implications for gaze-enabled applications.

IN THE PRESS

I enjoy communicating the results of my research to the general public. Whenever possible, we announce new findings in press releases which is frequently picked up by larger media, including BBC, NY Times, The Guardian, Washington Post, and many others. This is a selection of recent articles:

SELECTED PROJECTS

context-aware mixed-reality

Context-Aware Mixed Reality

See more
136M keystrokes

136M Keystrokes

See more
How we type

How we type

See more
Eye tracking quality

Eye tracking quality

See more
Mid-Air Text Entry

Mid-Air Text Entry

See more
PianoText

PianoText

See more

Datasets

How We Type Dataset

We released the data collected during the How We Type study. The dataset is very large. It contains the motion capture data (movement of hands and fingers during typing), the keystrokes, reference videos, and eye tracking data. The smaller parts of the dataset can be directly downloaded on the project page. Please contact me if you want other parts.

136M Keystrokes Dataset

We collected typing data of over 160,000 computer users participating in an online typing test. This dataset is available on the project page. It contains the keypress and -release events with details of each keystroke.

37K Mobile Typists Dataset

Data from over 37,000 mobile users completing a 15-sentence typing test on their mobile device. More info on the project page.

Code and Resources

Automatic Labeling of Motion Capture Markers (for Hand tracking)

As part of the How We Type project I developed Python scripts to automatically label motion capture markers based on a nearest neighbor approach. The scripts are free to use and can be found on GitHub


PUBLICATIONS AND TALKS

Dissertation

Assignment Problems for Optimizing Text Input
Aalto University publication series DOCTORAL DISSERTATIONS, 103/2018

Annas dissertation

This thesis won the SIGCHI Outstanding Dissertation Award in 2019.

My PhD thesis advances the state of the art in text-entry optimization by proposing novel objective functions that quantify the performance, ergonomics and learnability of a text input method. In addition, the thesis gives an introduction to user interface optimization and summarizes work in text entry optimization from the last 80 years. The assignment problem thereby serves as a unifying framework for formulating the novel objectives as well as those from prior work. I show the optimization of text input in three concrete cases: text input with multi-finger gestures in mid-air, text input on a long piano keyboard, and -- for a contribution to the official French keyboard standard -- input of special characters via a physical keyboard. While the work focused on text input, the assignment problem can be used to model other design problems in HCI (e.g., how best to assign commands to UI controls or distribute UI elements across several devices), for which the same problem formulations, optimization techniques, and even models could be applied.

Full papers

XRgonomics: Facilitating the Creation of Ergonomic 3D Interfaces
Conference paper, ACM CHI 2021.
PDF BibTeX Project page

João Marcelo Evangelista Belo, Anna Maria Feit, Tiare Feuchtner, Kaj Grønbæk

Arm discomfort is a common issue in Cross Reality applications involving prolonged mid-air interaction. Solving this problem is difficult because of the lack of tools and guidelines for 3D user interface design. Therefore, we propose a method to make existing ergonomic metrics available to creators during design by estimating the interaction cost at each reachable position in the user’s environment. We present XRgonomics, a toolkit to visualize the interaction cost and make it available at runtime, allowing creators to identify UI positions that optimize users’ comfort. Two scenarios show how the toolkit can support 3D UI design and dynamic adaptation of UIs based on spatial constraints. We present results from a walkthrough demonstration, which highlight the potential of XRgonomics to make ergonomics metrics accessible during the design and development of 3D UIs. Finally, we discuss how the toolkit may address design goals beyond ergonomics.

Detecting Relevance during Decision-Making from Eye Movements for UI Adaptation
Conference paper, ACM ETRA 2020.
PDF BibTeX Project page

Anna Maria Feit, Lukas Vordemann, Seonwook Park, Catharina Berube, Otmar Hilliges

This paper proposes an approach to detect information relevance during decision-making from eye movements in order to enable user interface adaptation. This is a challenging task because gaze behavior varies greatly across individual users and tasks and ground-truth data is difficult to obtain. Thus, prior work has mostly focused on simpler target-search tasks or on establishing general interest, where gaze behavior is less complex. From the literature, we identify six metrics that capture different aspects of the gaze behavior during decision-making and combine them in a voting scheme. We empirically show, that this accounts for the large variations in gaze behavior and out-performs standalone metrics. Importantly, it offers an intuitive way to control the amount of detected information, which is crucial for different UI adaptation schemes to succeed. We show the applicability of our approach by developing a room-search application that changes the visual saliency of content detected as relevant. In an empirical study, we show that it detects up to 97% of relevant elements with respect to user self-reporting, which allows us to meaningfully adapt the interface, as confirmed by participants. Our approach is fast, does not need any explicit user input and can be applied independent of task and user.

Context-Aware Online Adaption of Mixed Reality Interfaces
Conference paper, ACM UIST 2019.
PDF BibTeX Project page

David Lindlbauer, Anna Maria Feit, Otmar Hilliges

We present an optimization-based approach for Mixed Reality (MR) systems to automatically control when and where applications are shown, and how much information they display. Currently, content creators design applications, and users then manually adjust which applications are visible and how much information they show. This choice has to be adjusted every time users switch context, i.e., whenever they switch their task or environment. Since context switches happen many times a day, we believe that MR interfaces require automation to alleviate this problem. We propose a real-time approach to automate this process based on users' current cognitive load and knowledge about their task and environment. Our system adapts which applications are displayed, how much information they show, and where they are placed. We formulate this problem as a mix of rule-based decision making and combinatorial optimization which can be solved efficiently in real-time. We present a set of proof-of-concept applications showing that our approach is applicable in a wide range of scenarios. Finally, we present an evaluation with a dual task paradigm. Our approach resulted in similar task performance as a traditional UI and decreased secondary tasks interactions by 36%.

Honorable Mention AwardHow do People Type on Mobile Devices? Observations from a Study with 37,000 Volunteers
Conference paper, ACM MobileHCI 2019.
PDF BibTeX Project page Slides

Kseniia Palin, Anna Maria Feit, Sunjun Kim, Per Ola Kristensson, Antti Oulasvirta

This paper presents a large-scale dataset on mobile text entry collected via a web-based transcription task performed by 37,370 volunteers. The average typing speed was 36.2 WPM with 2.3 uncorrected errors. The scale of the data enables powerful statistical analyses on the correlation between typing performance and various factors, such as demographics, finger usage, and use of intelligent text entry techniques. We report effects of age and finger usage on performance that correspond to previous studies. We also find evidence of relationships between performance and use of intelligent text entry techniques: auto-correct usage correlates positively with entry rates, whereas word prediction usage has a negative correlation. To aid further work on modeling, machine learning and design improvements in mobile text entry, we make the code and dataset openly available.

Honorable Mention AwardObservations on Typing from 136 Million Keystrokes
Conference paper, ACM CHI 2018. Honourable Mention Award.
PDF, 2.8MB BibTeX Project page Slides

Vivek Dhakal, Anna Maria Feit, Per Ola Kristensson, Antti Oulasvirta

We report on typing behaviour and performance of 168,000 volunteers in an online study. The large dataset allows detailed statistical analyses of keystroking patterns, linking them to typing performance. Besides reporting distributions and confirming some earlier findings, we report two new findings. First, letter pairs that are typed by different hands or fingers are more predictive of typing speed than, for example, letter repetitions. Second, rollover-typing, wherein the next key is pressed before the previous one is released, is sur- prisingly prevalent. Notwithstanding considerable variation in typing patterns, unsupervised clustering using normalised inter-key intervals reveals that most users can be divided into eight groups of typists that differ in performance, accuracy, hand and finger usage, and rollover. The code and dataset are released for scientific use.

AdaM: Adapting Multi-User Interfaces for Collaborative Environments in Real-Time
Conference paper, ACM CHI 2018.
PDF, 4MB BibTeX Project page

Seonwook Park, Christoph Gebhardt, Roman Rädle, Anna Maria Feit, Hana Vrzakova, Niraj Dayama, Hui-Shyong Yeo, Clemens Klokmose, Aaron Quigley, Antti Oulasvirta, Otmar Hilliges

Developing cross-device multi-user interfaces (UIs) is a challenging problem. There are numerous ways in which content and interactivity can be distributed. However, good solutions must consider multiple users, their roles, their preferences and access rights, as well as device capabilities. Manual and rule-based solutions are tedious to create and do not scale to larger problems nor do they adapt to dynamic changes, such as users leaving or joining an activity. In this paper, we cast the problem of UI distribution as an assignment problem and propose to solve it using combinatorial optimization. We present a mixed integer programming formulation which allows real-time applications in dynamically changing collaborative settings. It optimizes the allocation of UI elements based on device capabilities, user roles, preferences, and access rights. We present a proof-of-concept designer-in-the-loop tool, allowing for quick solution exploration. Finally, we compare our approach to traditional paper prototyping in a lab study.

Physical Keyboards in Virtual Reality: Analysis of Typing Performance and Effects of Avatar Hands
Conference paper, ACM CHI 2018.
PDF, 8.2MB BibTeX

Pascal Knierim, Valentin Schwind, Anna Maria Feit, Florian Nieuwenhuizen, Niels Henze

Entering text is one of the most common tasks when interacting with computing systems. Virtual Reality (VR) presents a challenge as neither the user's hands nor the physical input devices are directly visible. Hence, conventional desktop peripherals are very slow, imprecise, and cumbersome. We developed a apparatus that tracks the user's hands, and a physical keyboard, and visualize them in VR. In a text input study with 32 participants, we investigated the achievable text entry speed and the effect of hand representations and transparency on typing performance, workload, and presence. With our {apparatus}, experienced typists benefited from seeing their hands, and reach almost outside-VR performance. Inexperienced typists profited from semi-transparent hands, which enabled them to type just 5.6 WPM slower than with a regular desktop setup. We conclude that optimizing the visualization of hands in VR is important, especially for inexperienced typists, to enable a high typing performance.

Selection-based Text Entry in Virtual Reality
Conference paper, ACM CHI 2018.
PDF, 2.5MB BibTeX

Marco Speicher, Anna Maria Feit, Pascal Ziegler, Antonio Krüger

In recent years, Virtual Reality (VR) and 3D User Interfaces (3DUI) have seen a drastic increase in popularity, especially in terms of consumer-ready hardware and software. While the technology for input as well as output devices is market ready, only a few solutions for text input exist, and empirical knowledge about performance and user preferences is lacking. In this paper, we study text entry in VR by selecting characters on a virtual keyboard. We discuss the design space for assessing selection-based text entry in VR. Then, we implement six methods that span different parts of the design space and evaluate their performance and user preferences. Our results show that pointing using tracked hand-held controllers outperforms all other methods. Other methods such as head pointing can be viable alternatives depending on available resources. We summarize our findings by formulating guidelines for choosing optimal virtual keyboard text entry methods in VR.

Computational Support for Functionality Selection in Interaction Design
Journal paper, ACM TOCHI, 2017
ACM DL Author-ize service PDF, 3.2MB BibTeX Project page Slides

Antti Oulasvirta, Anna Maria Feit, Perttu Lähteenlahti, Andreas Karrenbauer

Designing interactive technology entails several objectives, one of which is identifying and selecting appropriate functionality. Given candidate functionalities such as “print,” “bookmark,” and “share,” a designer has to choose which functionalities to include and which to leave out. Such choices critically affect the acceptability, productivity, usability, and experience of the design. However, designers may overlook reasonable designs because there is an exponential number of functionality sets and multiple factors to consider. This article is the first to formally define this problem and propose an algorithmic method to support designers to explore alternative functionality sets in early stage design. Based on interviews of professional designers, we mathematically define the task of identifying functionality sets that strike the best balance among four objectives: usefulness, satisfaction, ease of use, and profitability. We develop an integer linear programming solution that can efficiently solve very large instances (set size over 1,300) on a regular computer. Further, we build on techniques of robust optimization to search for diverse and surprising functionality designs. Empirical results from a controlled study and field deployment are encouraging. Most designers rated computationally created sets to be of the comparable or superior quality than their own. Designers reported gaining better understanding of available functionalities and the design space.

Honorable Mention AwardToward Everyday Gaze Input: Accuracy and Precision of Eye Tracking and Implications for Design
Conference paper, ACM CHI 2017. Honourable Mention Award.
ACM DL Author-ize service PDF, 7.8MB BibTeX Slides

Anna Maria Feit, Shane Williams, Arturo Toledo, Ann Paradiso, Harish Kulkarni, Shaun Kane, and Meredith Ringel Morris

For eye tracking to become a ubiquitous part of our everyday interaction with computers, we first need to understand its limitations outside rigorously controlled labs, and develop robust applications that can be used by a broad range of users and in various environments. Toward this end, we collected eye tracking data from 80 people in a calibration-style task, using two different trackers in two lighting conditions. We found that accuracy and precision can vary between users and targets more than six-fold, and report on differences between lighting, trackers, and screen regions. We show how such data can be used to determine appropriate target sizes and to optimize the parameters of commonly used filters. We conclude with design recommendations and examples how our findings and methodology can inform the design of error- aware adaptive applications.

How We Type: Movement Strategies and Performance in Everyday Typing
Conference paper, ACM CHI 2016
ACM DL Author-ize service PDF, 3.51MB BibTeX Slides Project page

Anna Maria Feit, Daryl Weir, and Antti Oulasvirta.

This paper revisits the present understanding of typing, which originates mostly from studies of trained typists using the ten-finger touch typing system. Our goal is to characterise the majority of present-day users who are untrained and employ diverse, self-taught techniques. In a transcription task, we compare self-taught typists and those that took a touch typing course. We report several differences in performance, gaze deployment and movement strategies. The most surprising finding is that self-taught typists can achieve performance levels comparable with touch typists, even when using fewer fingers. Motion capture data exposes 3 predictors of high performance: 1) unambiguous mapping (a letter is consistently pressed by the same finger), 2) active preparation of upcoming keystrokes, and 3) minimal global hand motion. We release an extensive dataset on everyday typing behavior.

Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry
Conference on Human Factors in Computing Systems, CHI 2015
ACM DL Author-ize service PDF, 3.51MB BibTeX Project page

Srinath Sridhar, Anna Maria Feit, Christian Theobalt, and Antti Oulasvirta.

This paper investigates an emerging input method enabled by progress in hand tracking: input by free motion of fingers. The method is expressive, potentially fast, and usable across many settings as it does not insist on physical contact or visual feedback. Our goal is to inform the design of high-performance input methods by providing detailed analysis of the performance and anatomical characteristics of finger motion. We conducted an experiment using a commercially available sensor to report on the speed, accuracy, individuation, movement ranges, and individual differences of each finger. Findings show differences of up to 50% in movement times and provide indices quantifying the individuation of single fingers. We apply our findings to text entry by computational optimization of multi-finger gestures in mid-air. To this end, we define a novel objective function that considers performance, anatomical factors, and learnability. First investigations of one optimization case show entry rates of 22 words per minute (WPM). We conclude with a critical discussion of the limitations posed by human factors and performance characteristics of existing markerless hand trackers.

Redesigning the Piano Keyboard for Text Entry
Conference paper, ACM DIS 2014
ACM DL Author-ize service PDF, 5.56 MB BibTeX Project page

Anna Maria Feit and Antti Oulasvirta.

Inspired by the high keying rates of skilled pianists, we study the design of piano keyboards for rapid text entry. We review the qualities of the piano as an input device, observing four design opportunities: 1) chords, 2) redundancy (more keys than letters in English), 3) the transfer of musical skill and 4) optional sound feedback. Although some have been utilized in previous text entry methods, our goal is to exploit all four in a single design. We present PianoText, a computationally designed mapping that assigns letter sequences of English to frequent note transitions of music. It allows fast text entry on any MIDI-enabled keyboard and was evaluated in two transcription typing studies. Both show an achievable rate of over 80 words per minute. This parallels the rates of expert Qwerty typists and doubles that of a previous piano-based design from the 19th century. We also design PianoText-Mini, which allows for comparable performance in a portable form factor. Informed by the studies, we estimate the upper bound of typing performance, draw implications to other text entry methods, and critically discuss outstanding design challenges.

Workshops, Posters, etc.

Computational Design of Input Methods
Doctoral Consortium, ACM CHI 2017

Designing a user interface or input method requires to evaluate and trade-off many criteria. The corresponding design spaces are huge, making it impossible to consider every potential design. Therefore, my work focuses on the use of computational methods for the design of input methods. I follow a modelling-optimization approach: understand and model the characteristics of the interaction, formulate the design space and develop (multi-) objective functions to evaluate designs, and develop algorithms to systematically search for the best design. In my projects I applied this approach to develop better text entry methods. Among others, I modelled the performance and anatomical constraints of the hand to computationally optimize multi-finger gestures for mid-air input, and studied how people type on physical keyboards, in order to understand and model the performance of two-hand typing.

Towards Multi-Objective Optimization for UI Design
Workshop Paper, ACM CHI 2015
PDF, 228KB BibTeX

Anna Maria Feit, Myroslav Bachynskyi and Srinath Sridhar.

In recent years computational optimization has been applied to the problem of finding good designs for user interfaces with huge design spaces. There, designers are struggling to integrate many different objectives into the design process, such as ergonomics, learnability or performance. However, most computationally designed interfaces are optimized with respect to only one objective. In this paper we argue that multi-objective optimization is needed to improve over manual designs. We identify 8 categories that cover design principles from UI design and usability engineering. We propose a multi-objective function in form of a linear combination of these factors and discuss benefits and pitfalls of multi-objective optimization.

Text is in the Air... Investigating Multi-Finger Gestures for Mid-Air Text Entry
ACM womEncourage 2015
PDF, 676KB

Anna Maria Feit, Srinath Sridhar, Christian Theobalt and Antti Oulasvirta.

Mid-air input, enabled by recent progress in computer vision based marker-less hand tracking, is an exciting candidate for text entry where direct touch input is not practicable or not available. In this paper we investigate the use of chord-like multi-finger gestures for entering text in mid-air. In contrast to previous methods, they require no extrinsic targets and can be performed eyes-free. Wesystematically explore the design space of hand gestures by computationally optimizing the letter-to-gesture mapping with respect to multiple objectives: performance, anatomical comfort, learnability and mnemonics. First investigations of one optimization case show entry rates of 22 words per minute. While this is promising, our study reveals several limitations of both, the mapping design as well as the available tracking methods. We discuss open challenges in mid-air input and conclude with recommendations for future work.

Invited Talks

Optimizing Special Character Entry: the Case of the French Keyboard Standard
Int. Conference on Operations Research OR, 2017
Slides

Anna Maria Feit, Antti Oulasvirta

We present the optimization of the new French standard for keyboard layouts. The typical French keyboard layout made it overly complicated or even impossible to type all French characters, punctuation marks, and other symbols. The goal of this new standard was to enable and facilitate the correct spelling of French, and allow the entry of symbols common in programming languages, mathematical expressions, and other European languages. Therefore, we had to assign over 115 characters to 148 slots on the keyboard, optimizing their placement with respect to each other and to the numbers (0-9) and non-accentuated letters (a-z) that were kept unchanged. In interaction with an expert committee, we identified four objectives: input performance and ergonomics, similarity to the prior keyboard, and coherent placement of similar symbols. We collected extensive performance data, language statistics, expert ratings on character similarities, and ergonomic scores to implement and solve a multi-objective quadratic assignment problem. In an iterative process, we computed and adapted a range of solutions in interaction with language experts. The resulting design unifies mathematical bounds with expert opinions and constraints. The new French keyboard layout is the first modern standard where computational optimization methods were used in interaction with domain experts to implement an optimal keyboard design.

Optimization of Text Input
Dagstuhl Seminar on Computational Interactivity, 2017

Designing a user interface or input method requires evaluating and trading-off many criteria. The corresponding design spaces are huge, making it impossible to manually build and test every potential design. For example, if we want to design a method to enter letters in mid-air via finger gestures, there are 10^33 possibilities to assign 27 characters to 32 hand gestures. Therefore, my work focuses on using optimization methods for the design of (text input) systems. The use of optimization methods allows us to efficiently and rigorously search very large design spaces, it gives quantitative guarantees on the goodness of a design and helps us to explicitly formulate and trade-off different criteria and constraints. Using optimization methods to design an input method or user interface requires 3 steps. The first step is how to formulate the design problem. This requires to explicitly state the decisions and constraints in a mathematical way and helps to understand the characteristics of the problem. Second, we need to formulate an objective function that can be used to evaluate and compare different designs. The input data and models we use for evaluation determines the outcome of the optimization. Third, in order to solve the optimization problem, we need computational search methods. This can be mathematical solvers which guarantee to find the optimum, or it can be approximation algorithms. I present several projects in which the focus is to develop more plausible, empirically valid formulations and objectives for more realistic optimization approaches. Among others, we modeled the performance and anatomical constraints of the hand to computationally optimize multi-finger gestures for mid-air input [1] and studied how people type on physical keyboards, in order to understand the performance of two-hand typing [2]. Most recently, we used Integer Programming to optimize the special character layout of the French keyboard to facilitate typing of correct French. Therefore, we reformulated the commonly used letter assignment problem to make it applicable to large real-world cases with over 120 to-be-mapped characters and quantified the performance, ergonomics, and ease-of-use of a keyboard layout. While my work focuses on text entry, the same problem formulations and optimization methods can be applied to the design of many other input methods and user interfaces.

THESIS OR JOB

I am always looking for outstanding students, PhD candidates or post-docs to join my research group at Saarland University.

If you are interested in the areas of Human-Computer Interaction, Optimization, Machine Learning, Psychology, Cognitive Science, or similar, reach out to me. I am always happy to discuss your ideas or propose a thesis topic to you.

CONTACT ME



Email address:
feit@cs.uni-saarland.de