Screen
This page is a draft. It may be incomplete, inaccurate, or written not well. It will definitely change in the future. Comments, corrections, and suggestions welcome.
This details my project to create a large multi-touch capable touch screen, as well as a nice home theatre display all in one.
Future Workday
Imagine yourself in an office, where your desk does not contain a computer. Instead, you have an entire wall glowing beside you, and you can simply lean over and touch it to interact with your computer. You begin your day by grabbing an incoming email and stretching it out to read it. You work throughout the day by drawing new design plans with your fingers or wireless light-pens. You leave your office and walk into the meeting room where you have another wall filled with a similar, but larger touch-screen, so all thirty people in the room can see your new designs in detail. Finally, you end your day by walking out of your office and dragging your hand across the screen to shut it down.
It almost works!
Overview
I am building a large touch screen capable of those interactions with technology based on the ideas Jeff Han brought to the world. I’m going to be using his FTIR technique for touch inputs, and augment this with wireless light-pens. I’m going to build an 8 foot wide, 4 foot tall screen (107 inches diagonally), engineered to use FTIR on a large scale. Because of the size, it’ll also be perfectly usable as a high-quality movie screen (or “HDTV” as the businesses want you to call it).
FTIR
Refraction is the passage of light rays bending through two mediums of different densities (different IOR, technically), like water and air. The light rays bend in when they go into a material with a higher density (IOR), and similarly bend out when they leave that material. However, when the angle of incidence inside the material is shallow enough, the light rays reflect inside instead of leaving. They cannot refract out, because there is no room to bend. That is Total Internal Reflection (TIR).
Frustrated Total Internal Reflection (FTIR) changes the rules of the bend. In my case, I have a webcam with an infrared pass filter located behind the piece of plexiglas with infrared light flooding through it. When I touch the plexiglas, the light tries to bend towards my fingers rather than into the air, which for light is much easier to do. The light bends out, hits my fingers, and causes bright points of Infrared light for recognition by the webcam.
A more technical and detailed description can be found on the Wikipedia article for Total Internal Reflection.
The only major downside to this technique is that you require a large space behind the surface for the lens of the camera to spread wide enough. This can be alleviated with expensive lenses or mirrors, plus a few good calibration techniques to deal with the image warping caused by those lenses.
Details
Visible-Light Test Bed
To help me understand how I needed to build the software, I built a simple test-bed with a pane of glass, a lamp, a book, and an old towel. I had the lamp aiming into the side of the glass, and the light that did not go into the glass was absorbed by the towel next to it. I could easily see the FTIR effect when I touched the glass surface. I mounted a webcam below the glass, and even though glass is not a good conductor of light, I wrote the tracking software using the test-bed.
Overall Construction
The screen frame (detailed below) holds all the neccessary parts for the FTIR effect and also holds the projection fabric. The IR camera and projector are placed behind the screen, away from the user. This keeps the user’s hands from blocking the projection, and makes the touch detection much more accurate.
Construction of the Frame
Designing the hardware was a bigger challenge than I anticipated. I had to design a wooden frame that would support fifty pounds of a quarter-inch thick sheet of plexiglas without distortion; would hold two strands of eleven infrared LEDs; and would suspend the rear-projection fabric an eighth-inch above the plexiglas, creating a pocket of air in between.
I settled on an H-frame design with multiple rabbets for holding the acrylic and for the buffer. I also cut a channel in the middle of the largest portion for the wiring, and drilled holes through for the LEDs to sit in.
Parts
These parts are needed for the screen:
My sheet, still wrapped in cardboard.
Acrylic. A big sheet of plastic to serve as the medium for the IR light.
Projection screen fabric. I got 4 yards of 55-inch wide Rosco Grey for it.
Webcam capable of IR. I’ve got this, and it picks up near-IR light beautifully—tested with multiple remote controls.
Near-IR pass filter for webcam, to make sure that the visible light that reflects back from the screen (and wherever else) doesn’t pollute the webcam’s image of the touches.
Near-IR LEDs, to light the inside of the plastic. I’m using 11 on each long (8ft) side of the screen.
Projector. I was able to get a broken projector at a local A/V rental place, and was able to fix it afterwards.
Power source for LEDs. I’m using an old IBM laptop power supply, coupled with a homegrown box for adjusting the amperage to be the right value for the LEDs.
Frame. Completed, see above.
Software
There are three main pieces of code I need to have for this project to work properly, although I could get away with joining two of them for testing.
First, the driver. I’ve recently made this work, implemented in plain C, and I’m tweaking it for a temporary visible light setup I have. The driver handles taking the raw image data from the camera and turning it into either a series of touches or a set of points with a little data on brightness and size. I have the latter working perfectly right now. I have a few thoughts on the implementation specifics.
Second, the window manager. I’m going to write this in Haskell using the HOpenGL bindings. This piece manages display to the screen and talking to the applications and driver. It ties everything together.
And third, the applications. These do the thinking. They talk to the window manager to get their events and to display things to the user. This is where we write things like video players, objects for music pipes, web browser interfaces, things to spawn pictures (file browsers, flickr search), etc.
