In this project we present the NiCE Formula Editor, which recognizes handwritten formulas to provide in situ computation and visualization. Our editor uses a novel hardware setup, suitable for use as a desktop or whiteboard that disambiguates pen and multi-touch input. In our interface, the pen writes and performs gestures, while touch manipulates. Multiple users can sketch mathematical expressions (e.g. formulas, vectors, matrices, etc.) with digital pens on the interactive surface. These expressions are analyzed and interpreted using the StarPad SDK’s support for mathematics recognition. Pen and touch gestures can be used to manipulate the written expressions. In addition, expressions can be selected and moved by touch or pen input. For example, a scribbling gesture can be used to delete symbols and a loop gesture can be used to select expressions. Also, the digital interpretation of the written expression can be manipulated through touch interaction. For example a user can rearrange terms in an equation or pinch equations to simplify them by combining the two touched terms together. Moreover, a written expression can be connected to visualization widgets simply by dragging the expression using a fingertip over an already visible widget and then dropping it by lifting the finger. A computer graphics teacher, for example, can drop various matrices onto the “2D Matrix Stack Viewer” widget to show how matrices transform a 2D object. Due to the diversity of mathematical problems, our formula editor supports a plug-in architecture for incorporating customized widgets. This flexible plug-in API provides developers with predefined interaction modules and access to “equation-solving functions” which are specially designed to support visualization.
Motivation behind NiCE Formula Editor
Interactive Whiteboards are becoming an increasingly important component of face-to-face meetings, particularly in educational settings. Still, non-interactive surfaces have remained the standard for mathematics-oriented discussions in part due to the lack of robust algorithms for recognizing handwritten mathematical expressions. Thus teachers, for example, are typically burdened with drawing visualizations by hand to reinforce abstract and complex mathematical problems. Our work expands upon the notion of mathematical sketching by supporting both pen and touch interactions on an interactive whiteboard to write expressions, to perform algebraic transformations, and to drive sophisticated 2D and 3D illustrations. By incorporating lightweight touch-based interactions, students can “play” with formulas and explore mathematics from a fundamentally interactive perspective that we believe many will find to be engaging and enjoyable.
Awards
Europrix Multimedia Award 2010:
Winner in the category “Content Tools & Interface Design”
RTT Emerging Technologies Contest 2010:
Winner in the category “Interfacing & Interaction”