LuminAR

LuminAR is a robotic mixed reality interface that evolves the concept of a desk lamp into a compact computing device transforming surfaces and objects into interactive augmented spaces.

Designed by Natan Linder, Master’s student at Fluid Interfaces Group, MIT Media Lab, LuminAR combines pico-projector technology with cameras and image processing software to create a gestural user interface projected on table-tops or walls. LuminAR is mounted on a robotic arm and is thus able to interact with the user in a wider range augmenting the entire environment of an office desk for example.

Predecessors

LuminAR advances original ideas such as DigitalDesk (Wellner, 1991), I/O Bulb and Luminous Room (Underkoffler/Ullmer/Ishii, 1999) and the Everywhere Display (Pingali, 2003) into a portable small form-factor design which fits into a traditional light bulb socket. LuminAR is similar to DockLamp (Kaplan, 2007) adding a robotic component.

Design principles

Linder radically rethinks the concept of simple everyday objects. His design of LuminAR deliberately hides technology, makes it portable, compact and easy to access. LuminAR fits into a traditional light bulb socket, is self-contained, networked and self-calibrating. Installation is simple and LuminAR is ready to use as stand-alone device at once. It’s real power for information handling and new interaction patterns is revealed when accessed from networked devices. Linder particularly chose technology that aids his principles, for example, a pico-projector that is laser-based and thus always in focus in a range between 200 and 2000mm.

LuminAR Design

Hardware design

During the three year project Linder designed different prototypes advancing the combination of hardware parts, tuning their arrangement, heat dissipation, cable handling and overall footprint. Here’s an overview of used hardware parts. A full list including detailed specifications is included in Linder’s Master’s thesis.

  • Microvision laser-based PicoProjector Evaluation Kit v2 (15 lumen, 848×480 WVGA resolution, 16:9 aspect ratio, always in focus between 200 – 2000mm)
  • Arduino Pro Mini 328 with motor driver to operate micro servomotors of the robotic arm
  • Dual-Core Atom CPU at 1.6 GHz on an Intel Machine-2-Machine reference board, equipped with 1GB memory, MicroSD slot, WiFi, Accelerometer, HDMI, USB
  • one fixed webcam (Sony Playstation 3 Eye Camera), aligned to the projector to ease semi-automated calibration of the display surface, used to capture interaction gestures, scans images and detect fiducials
  • a second inexpensive and titlable webcam (Microsoft LiveCam NX 6000, 2MPixel) to capture the tabletop or the user for videoconferencing and face tracking
  • PMDTec CamBoard time-of-flight depth sensor, 200×200 px resolution and 40cm operation range
  • Polulu Maestro Micro servomotor control board
  • Edison #27 bulb screw mount
  • custom made aluminium rig to mount projector, cameras, mainboard and power supplies
  • one small 5V and one 12V power supply

LuminAR Hardware

Software architecture

LuminAR is a complex networked system connecting multiple computing and I/O devices. All software is modularised and runs as independent servers communicating over proprietary and some standardised protocols. The Intel M2M board runs on Ubuntu Linux for Atom as underlying OS. The image processing backend uses OpenCV, reactiVision, Google protobufs, and OpenGL for face and gesture tracking, image analysis and UI rendering. A custom backend component deals with robotic interaction, body and motor control. A particularly nice design decision is to implement applications from the start in HTML5/JS. The custom Javascript client library implemented for tis purpose runs on top Chrome and communicates with the LuminAR backend over HTTP/WebSockets.

User interaction

One important feature in gesture controlled interfaces is the distinction of a purposeful interaction with an interface element from an unintended hand swipe. Touching the projected interface is the most natural alternative, so it would be nice to somehow detect touch events.

LuminAR gestures

LuminAR uses an overhead camera facing downwards to capture the projected user interface. From this perspective it is rather difficult to distinguish whether a user’s hand touches the surface or hoovers closely over it. Linder’s first design choice was measuring the distance offsets between fingertips and surface using a time-of-flight depth camera. Unfortunately the used hardware suffered from too much noise to reliably detect touch events. Constrained by deadlines, Linder found a quick and more robust solution in thresholding pixel saturation values in images from a normal webcam (low saturation for the projection surface, high for hands). Of course, such a solution is not able to detect distance and can approximate touch only.

Lessons learned

In his thesis, Linder summarises lessons on UX design principles learned from different user studies and applications. This is an important issue for designers and researchers I would like to highlight here. Linder’s findings apply to any camera-projector systems for ambient and mixed reality interfaces:

  • avoid white background, use black instead
  • design for dynamic scales
  • use non-rectilinear graphics to avoid handling complex skew and keystone correction
  • minimize shadows and occlusions to make the user not occlude the interface which happens more often on display edges
  • use big and legible fonts

Limitations and criticism

LuminAR is a great project. Linder and his team deserve all my respect for their technical achievement. There are some critical issues that need to be improved.

First, taking advertisement as a primary use case for this great technology is an aweful choice. C’mon guys! I had expected more caution in selecting good showcases for technology made at MIT.

Second, reaction time for gesture detection is almost one second so to me the technology is not yet ready for prime-time. Part of the problem which I experienced in my own work lies in universal image processing algorithms (found in OpenCV) which are not yet optimised for application or the embedded execution platform. This issue could be easy to fix.

Third, a dynamic geometric projection model is not yet included so LuminAR requires flat display surfaces and an almost perpendicular projection direction. This considerably limits LuminAR’s usefullness for all applications that require fonts, have grid-based interfaces or build on current widget designs (i.e. almost everything out there).

LuminAR video conferencing

Last, video conferencing is an application I consider completely broken on LuminAR. It’s the arrangement of projector and (tiltable) camera that makes it impossible to achieve eye-contact even with advanced image-based rendering techniques. The camera perspective is simply too far away from the projection surface to support any useful face-to-face interaction. I see, however, much use in capturing desktops and hand movements at high quality, for example to record, share and teach particular handcrafts or skills.

Credits

LuminAR was a large team effort and the entire project development with all the prototypes took about 3 years to complete. The project was developed at Fluid Interface group, MIT Media Lab.

Concept and Team Lead: Natan Linder
Industrial Design: Yoav Reches, Jason Robinson
Mechanical Engineering: Iliya Tsekov, Alax Xu, Craig Broady, Toomas Sepp
Electronics: Brian Wu, Michelle Wang, Fei Ni
Software: Natan Linder, David Yamnitsky, Gaia Carini, Houston King, Rebecca Kleinberger, David Jia, Alexander List, Kimberly Toy, Ben Weissmann
Robotics Software: Jesse Grey, Matt Berlin

The project was partially supported by a research grant from Intel and donations in kind from Best Buy, Microvision, and PMDTec.