Triggering commands on large interactive surfaces is less efficient than on desktop PCs. It requires either large physical movements to reach an interaction area (e.g., buttons) or additional operations to call context menus (e.g., dwell). There is a lack of efficient ways to trigger shortcuts. We introduce Kolibri – a pen-based gesture system that allows fast access of commands on interactive whiteboards. Users can draw tiny gestures (approx. 3 mm) anywhere on the surface to trigger commands without interfering with normal inking. This approach does neither require entering a gesture mode, nor dedicated gesture areas. The implementation relies on off-the-shelf hardware only. We tested the feasibility and explored the properties of this technique with several studies. The results from a controlled experiment show significant benefits of Kolibri comparing to an existing approach.
Modern mobile devices feature high resolution screens with pixel densities over 300 pixels per inch (ppi). Often relying on touch input, they have a rather low input resolution. This gap between the input- and output-resolution leads to a problem of input precision, which is well known as the fat finger problem. A lot of research has been conducted to better understand and improve input precision. For large interactive surfaces it is the opposite. Display resolutions of existing large interactive displays are low, ranging from around 30ppi for LCD or plasma screens to 15ppi for some projector-based setups. These surfaces mostly use pen input which can be captured at a very high resolution. Many manufacturers, such as Polyvision and Hitachi, achieve an input resolution that is up to 40× higher than the display resolution of a typical projector based whiteboard system. Interestingly, this resolution gap has not yet been explored.
Preliminary Study
In order to test the general feasibility of this concept and discover properties of Kolibri gestures, we conducted an exploratory study with 20 participants (2 left-handed) in the age from 19 to 27. The goal of this study was to test the participants’ general ability to perform recognizable tiny gestures which should help us choose practical gesture types and sizes. We also measured the time for performing each gesture as a basis for further studies.
The gesture sizes ranged from around 1mm to 7mm, the average size was 3.3mm (SD = 0.98). The overall recognition rate for all gestures was 89.3% percent and was higher than 95% for 5 of the gestures. This result is based on existing, unmodified gesture recognition software and standard gesture templates. Our other experiments showed that with adjusted software and custom templates, recognition rates can be improved further. We measured the time it took participants to perform the gestures, which is independent from the software. The gestures can be categorized in three groups according to the execution time: <500ms (fast), 500ms < 1000ms (medium), and >1000ms (slow). We noticed that all gestures in the fast category have not more than one corner; gestures in the medium category have either two or three corners; and finally, slow gestures have more than three corners. This is in line with the two-thirds power law which states that less complex shapes can be performed faster. According to this finding, we suggest to choose gestures with no more than one corner for use cases, where fast execution-times are crucial (e.g., often used shortcuts like undo/redo). However, simpler gestures might result in more false activations.
Distinguishing gestures from normal inking
One of the most important pre-requisites for Kolibri gestures to work well on an interactive whiteboard is to ensure that they do not interfere with regular inking. Based on the measurements from the preliminary study, we defined several thresholds to distinguish gestures from regular inking. To gather information on strokes created during normal use of whiteboards and to test this data against our thresholds, we collected all the ink-strokes from six different teams who performed collaborative brainstorming sessions. The participants used a custom made, freeform sketching application on an interactive whiteboard to create sketches and take handwritten notes. Each team consisted of three people and each session lasted about one hour during which the teams created and discussed different ideas.
We analyzed the resulting sketches and handwritten notes which consisted of a total of 4,283 strokes. We calculated a bounding square for each stroke to determine its size and also counted the number of Anoto points in each stroke as an indicator for stroke complexity.
The blue graph in the figure above shows the number of strokes for different stroke sizes. The line shows that for a large number of strokes the bounding square is only up to 1 mm large. These are the aforementioned dots, common for punctuation marks. For strokes with a bounding square between around 1mm and 7mm (our targeted size of Kolibri gestures) the number of strokes drops more than 50% before climbing again and reaching additional peaks at 15mm and 21mm. After this the number continuously goes down for larger sizes. This means that during regular inking, much less strokes are generated in the size range that is suitable for Kolibri gestures.
Kolibri Gesture Performance
In the next step, we investigated the performance of Kolibri for triggering commands. For this we conducted a controlled experiment to compare Kolibri Gestures against state of the art Dwelling + Marking Menu. The task was to connect bubbles using corresponding colors. The interaction techniques are used to switch ink colors after each bubble connection.
For quantitative analysis we measured the time for each phase of a color-switch task. The overall task completion time is the entire time between finishing the last bubble-connection and starting the next one after switching to the correct color (t(0) – t(3)). We also logged the time between finishing the last bubble-connection and starting to perform the technique (preparation time, t(0) – t(1)), the time it took to actually perform the technique (execution time, t(1) – t(2)) and finally also the time it took to return to the bubble and start drawing the next connection (return time, t(2) – t(3)).
We measured a significant difference in the overall task completion time. On average it was 2,030ms (SD = 260) for Kolibri and 2,275ms (SD = 227) for D+MM, which results in a 246 ms difference.
Summarizing, Kolibri is a novel interaction technique that allows users to trigger shortcuts on interactive whiteboards with current off-the-shelf hardware. It achieves robust recognition, as well as very low false positives. A controlled experiment shows that issuing shortcut commands with this technique is much faster than with Dwell + Marking Menu. The Kolibri technique utilizes a largely unexplored gap between display-resolution and input-resolution on whiteboard surfaces. In this paper we explored the feasibility and properties about how people draw tiny gestures on a vertical surface. This brings us new challenges for gesture design and recognition when it comes to very small sizes. Those can be taken as suggestions for the design and implementation of other small-scale gesture systems. Further exploring this design space gives us a new perspective on the possibilities of pen-input on large interactive surfaces and will lead to novel application areas.