Enhancing Mobile Interaction with a Flexible Sensing and Display Cover

FlexCase is a novel flip cover for smartphones, which brings flexible input and output capabilities to existing mobile phones. It combines an e-paper display with a pressure- and bend-sensitive input sensor to augment the capabilities of a phone. Due to the form factor, FlexCase can be easily transformed into several different configurations, each with different interaction possibilities. We can use FlexCase to perform a variety of touch, pressure, grip and bend gestures in a natural manner, much like interacting with a sheet of paper. The secondary e-paper display can act as a mechanism for providing user feedback and persisting content from the main display. In this paper, we explore the rich design space of FlexCase and present a number of different interaction techniques. Beyond, we highlight how touch and flex sensing can be combined to support a novel type of gestures, which we call Grip & Bend. We also describe the underlying technology and gesture sensing algorithms. Numerous applications apply the interaction techniques in real-world examples, including enhanced e-paper reading and interaction, a new copy-and-paste metaphor, high degree of freedom 3D and 2D manipulation, and the ability to transfer content and support input between displays in a natural and flexible manner.

Introduction

The smartphone has revolutionized computing, making interacting with digital information as simple as reaching into your pocket. Although the hardware continually improves, many of the interactions are still reduced to touching and sliding our fingers on a rigid piece of glass.

To remove these bounds on 2D input on the front of the display and to avoid occluding (often valuable) pixels on screen, researchers have explored interaction on the back, above, or around the device. Further, researchers have looked at using flexible sensors or even the combination of flexible display and input to bring more paper-like physical interactions to mobile devices.

Another trend extends the output space of smartphones by using electronic paper (e-paper) displays in conjunction with the phone’s regular screen. These e-paper displays
are very power-efficient as they only need to be powered while the screen content is being updated, and exhibit a paper-like reading experience, which can remove aspects of eye strain. However, none of the aforementioned systems support rich flexible and gestural interactions between the two screens or make use of the combined extended screen space.

Teaser_5

In this paper, we present FlexCase, a flexible interactive flip cover that augments the input and output capabilities of smartphones with novel paper-like and gestural interactions. As shown in Figure above, a cover housing both our novel sensor and e-paper display is attached to the phone without compromising the form factor. We utilize a custom-made flexible touch, pressure and bend sensor, combined with a flexible e-paper display that turns the phone cover into a versatile input and output device. The cover provides interactive mechanisms to move content between displays. It allows the secondary display to be used as a peripheral surface for input and output. It also enables rich physical paper-like interactions with the cover using touch, pressure and various flex gestures. The ability to simultaneously detect touch and complex deformation of the surface allows for gestures, which we call Grip & Bend, that provide a variety of novel paper-like interactions. We explore the design space for FlexCase in detail, demonstrating a variety of new applications.

While we consider the novel form factor and the resulting interaction techniques as the main contributions of this paper, we also present technical novelty with a sensor that combines multiple modes of input. The flexible input sensor is built on a touch and pressure-sensitive printed piezoelectric film, which recently demonstrated its ability to sense complex deformations. In contrast to this prior work that utilizes a sparse arrangement of sensors around the periphery, we use a matrix-like sensor layout to sense touch, pressure, and complex deformations with only a single sensor. Summarizing, the main contributions of this paper are:

  • A concept of a novel flip cover form factor that augments existing mobile interaction with flexible output and input, including touch and bend input.
  • Exploration of the design space for our system with its reconfigurable multi-display and multi-input form factor.
  • Novel interaction techniques, which arise from the combination of touch, pressure, grip, and bend sensing.
    e.g. bimanual and back-of-device interaction or the incorporation of different grips for internal mode-switching.
  • A custom input sensor, which enables bi-directional bending and touch on both sides; and a new-machine learning algorithm for leveraging grip sensing for richer bend input.
  • A set of application scenarios demonstrating novel interaction possibilities with our proposed design.