Two observations on John Underkloffer’s speech on 3D user interfaces

[elfs]

I’ve recently watched John Underkloffer’s presentation on 3D UIs, and how he helped create the presentation for the film Minority Report. You know the scene, the one where Tom Cruise is working his way through the UI with a series of hand gestures (although the one in Iron Man 2 is an upgrade). As I was watching the clip, I watched Underkloffer work through a prototype, wearing gloves (as Cruise did, but Downey did not), and he had to make all of these esoteric gestures to make it behave.

As he did so, I flashed on the way my wife has to use her iPhone, with these weird gestures she has to use to get it to behave. There’s an entire library of gestures, and even worse, those gestures can mean completely different things in different contexts– in different programs, or even in different modes of the same program.

Underkloffer talks about how the WIMP (Window, Icon, Menu, Pointer) interface was a miracle when it debut popularly in the Macintosh, but we haven’t really progressed far from there. He wants us to stretch beyond that interactive format.

The thing about the WIMP interface, and one of the reasons we haven’t progressed far from that original design, is that it’s absolutely minimal in what you need to know from the start to make the system behave. Point, click, read. You don’t need to memorize a whole slew of esoteric commands, as you did with DOS (or as we Linux people pride ourselves on doing). Well-written UIs have discoverability and affordance, with the written word and the icon as the primary cues as to what to do next.

Underkloffer’s demonstration shows a world where affordance and discoverability don’t exist; you have to know the gestures, or be shown them, before you can do anything. Maybe we’ll have the bandwidth per application to teach that, maybe not. But the 3D UI (and all gestural UIs, like those in tablets and phones) is a step back to the era when we had to know some esoteric and unfamiliar activity– a code word, a gesture– to get anything done.

Most people don’t love Emacs. I understand that.

On the other hand, I did love one bit about Underkloffer’s essay. Back in the summer of 1992, I had the good luck to accompany a student group to a presentation and dinner by Dr. Timothy Leary. At that dinner, Dr. Leary and I got into a rather heated discussion about virtual reality.

Leary’s contention was that virtual reality was never going be the stuff of home installations. It was too expensive, too complicated. We’d have to go to places, like we go to theaters, to get the full virtual reality experience. He was adamant; by 2010, there’d be these places in malls you’d go to have what sounded a lot like Huxley’s “feelies.”

I argued that we were already there. We had mucks at the time, which were the beginnings of a communal virtual experience. He was highly dismissive: after all, that was still text on a screen. The whole goggles-and-gloves things would never happen in the home. I argued that the problem was one of bandwidth, which had grown in leaps and bounds in the ten years since the earliest BBSes.

Underkloffer’s vision is that five years from now every object we buy will have spatial sensors in the bezel, and interaction with the real world is just a matter of time and effort, the development of software to meaningfully interpret our gestures and convert them into commands.

I look forward to that.

This entry was automatically cross-posted from Elf's technical journal, ElfSternberg.com

Comments

[shockwave77598.livejournal.com]

Today we have second life, which cannot be made stereographic because it uses OpenGL. But most programs that are written for DirectX can be made stereographic with lcd shutter glasses and the proper video card.

If Blue Mars goes stereo and permits UCC, they'll bury Second Life.

[elfs.livejournal.com]

UCC? The United Church of Christ? Uniform Commercial Code? Unsecured Credit Committee?

[shockwave77598.livejournal.com]

User created content. What made SL popular is not the graphics (sucky) or the reliability (way sucky) or the interface (are we sensing a pattern yet?). What launched it to prominence was that anybody could create anything they could imagine, own it, and sell it to other people. For all it's faults -- and there are plenty -- that simple little addition of the capitalist way turned what would have otherwise been a neat note in techhistory into an actual viable alternative world.

But for all the promise of a 3D interface, my experience inworld has been that expanding into 3D does nothing but clutter things up. Only so much can be visible at a time, and that is a 2D flat set of icons, buttons, etc. While I've experimented with a HUD that has multiple controls that come to the foreground or background as they are needed, in the end, that's nothing more than a Tabbed set of menus. Except for being able to wander through all the menus possible within a room, a 3D UI simply doesn't help the user do anything. In fact, walking around and around looking for the FILE menu is rather slow compared to simply selecting it from a menu with a mouse.

[gromm.livejournal.com]

and even worse, those gestures can mean completely different things in different contexts

But they look cool in the movies. Which is what it's really all about. Remember how computers were depicted in the movies in say, 1985? When noone to speak of actually owned one?

[voidrandom.livejournal.com]

The problem with gestural interfaces may be a bit worse than that, given the existence of patents in this area.

http://ignorethecode.net/blog/2010/05/25/gestures/

Thoughts from Don Norman:
http://jnd.org/dn.mss/gestural_interfaces_a_step_backwards_in_usability_6.html

[mouser.livejournal.com]

About that last one:

While I agree with a portion of it (I had to tell a friend that his interface he was working on had several key bits of difference from the iPhone he was developing on) I do find it amusing the author complains about standardization while he and the person he's talking about can't agree on CHI vs HCI (I assume both are "Human-Computer interface"...)

[(Anonymous)]

To use DirectX is essentially to tie yourself to Windows. By doing so you make it that much harder for Linux and Mac OS users to access your world. Second Life does indeed have a Mac client I believe. They're also working on open sourcing their client so in effect if someone wished to create a DirectX version they probably could with enough work.

Perhaps more relevantly in a couple generations maybe the iPhone or iPad will have the capability to run a slimmed down Second Life client. That would be an entirely different vector for the virtual world. Maybe some places could even have a specific second life location tied to them, so when you fire up SL on your mobile device the location aware part would take you right to the specific place in the virtual world.

[shockwave77598.livejournal.com]

SL is based on OpenGL so it can work across PCs, macs and Linux boxes. While I like open standards like this, unfortunately OpenGL has not aged well. Compared to DirectX, it's been neglected and still has terrible bugs that existed 10 years ago with no effort to fix them (transparency layering problem for instance). So I'd like to see it where the PC users can choose to use directX or openGL while everyone else uses OpenGL. If the open source OpenGL cannot fix bugs within a decade while DirectX leaves it in the dust performance wise, then sadly, the open source direction has failed.

[mouser.livejournal.com]

I'm thinking that the attack needs to come from the other direction. Simply put, you need to allow the user to customize their own interfaces and then apply them across other applications.

In my first application I teach it that to select an item in 3d space I touch it once. As I go from appA to appB it remembers that.

You, on the other hand, grab the object. It remembers that for you.

There should probably be a default standard of the "obvious" things - the LukeW page has some good "obvious" ones that once you know them (zoom in/out) you automatically try to apply them to other things. Don Norman's point (as voidrandom mentions) is that it's not seamless inside an application/shell, let alone across experiences.

I remember with Quake, I customized the keyboard for commands extensively. Unlike 99.9% of players, I use ESDF instead of WASD. It's simply more comfortable to me. I even used the keybind file in other ID games. I hated it when I realize other games weren't going to use the same interface.

Interfaces (touch, 3d, whatever) will change. The way that we "standardize", THAT'S what has to change.

[danlyke.livejournal.com]

I think the problem with customized interfaces is that they need to follow you, and we don't have a good way for that to happen yet. Just look at what happens to someone like me who customizes their window manager and their shell when we go to help a novice on another computer: There's a short learning curve just getting back to "normal" while we try to show them how to, say, select a menu item.

Maybe we'll find a way to customize computer UI in the same way that we customize car UI, where high end cars can remember a few user settings (seat and mirror positions), but I'm not optimistic.

I've long referred to "point and click" interfaces as "point and grunt", because I believe that the mouse (especially a single-button one) reduces our interaction with the computer to that level, but maybe that's what most people can tolerate. And the rest of us will continue to use the richer language structure of the command line...

[mouser.livejournal.com]

As developers, we could easily have interfaces follow us around. No one has made it happen. Yet. It will probably be the next Twitter or similar cloud service - if it can be made to happen...

What amuses me are the HP desktop computers with touch screens - they make my arms hurt just LOOKING at them! How long do they think people use that at one sitting?

[woggie.livejournal.com]

I keep wanting goggles, gloves, and multiple VR ginormous screens of whatever that you can just move around.

The tech seems to be available, but the idea of a VR screen or UI seems to be missing from the way researchers think about VR space.

And the hardware is all either really expensive or really narrowly focused on a particular application. Or both.