Future Interfaces: Our interactions beyond screens
42 min social networking, 35 min productivity, 12 min entertainment.
That’s my current screen time status for today — relatively, it’s not ‘that bad’, but it does bring to life the fact that I spend quite a bit of time looking at a screen — as these numbers don’t even include the time spent at my laptop. And it’s still early afternoon.
We’re at a point where the average person spends more than 11 hours using a screen — work or leisure, there’s some kind of a screen involved.
While the idea of interacting with screens for the majority of our day admittedly worries me, the possible consequences of this is a topic for another essay. But it does bring me to another question — what does the future interface look like? How much longer are we actually going to be using screens for, and what’s next?
I’ve nailed down a few thoughts from a bit of research and asking my peers, and will be looking at a range of interfaces, from voice, to smart touch surfaces, to our brains.
But first, what do I mean by interface? Interfaces can be extended to ‘everything designed into an information device with which a person may interact’ — in essence, it is any way through which a user interacts with an application or a website.
I believe that the easiest way to try and understand the future interface is to go back to understand what the purpose of an interface is. Essentially, it is an aid to help us complete a task (and a task can range from completing our work, to watching our favourite show). So what is a more efficient and slick way to complete this task? How can we shorten the gap between humans and computers?
Voice
Starting off with the most obvious one, voice-enabled technology is already extremely prevalent — many agree that it is generally easier to say things than type, and with voice controls, our interactions with computers can be greatly sped up. In terms of the future of voice technology, interactions will become more seamless, natural sounding (think Google Duplex), smarter, personalised, context sensitive, and accessible (in terms of both price and channels). In fact, many industry experts predict that nearly every application will integrate voice technology in some way in the next 5 years.
VR/AR/MR
Having worked in tech for 3 years now, Virtual/Augmented/Mixed reality feels a bit like old news. We are, however, moving beyond the standard PokemonGo phase and creating entirely new worlds at an unprecedented rate. Gear such as MagicLeap makes synthetic digital images that merges with real spaces — that is, it enhances our world with digital objects while still letting us interact with everything real that’s going on around us. While being completely frank, I wasn’t that impressed when trying it on as the digital images were limited and I find having to use physical controls a bit clunky (and I’d rather use haptic controllers such as Oculus’ gloves), it did open my eyes to the possibilities of what ‘pulling the web out of the screen’ looks like, while still staying within our surroundings and being aware of what’s happening around us.
Microsoft is releasing HoloLens 2 this year — while we do not know much about its features, it is expected to be a faster, lighter and end-consumer friendly version of its predecessor, the HoloLens, with an improved field of view. We’ll be able to put on a headset and view multiple screens and do some bonkers multitasking, while taking actions with only our hand gestures (no controls necessary).
Note: I did initially think of comparing HoloLens to Google Glass, which we can say was a bit ahead of its time, but while HoloLens is designed to project images in midair and on surrounding objects, Glass was designed to perform the functions of a smartphone — take photos and videos, perform internet searches, provide directions-but it didn’t offer much functionality that a smartphone doesn’t, and as a result consumers perceived Glass as a redundant, more expensive version of their handheld devices.
Virtual and augmented reality technology does open up an entire new world of ways to interface with our devices. But the idea of us all walking around with headsets on does seem a bit unrealistic and uncomfortable — these devices will likely stay in place for specific uses, rather than a standard to completely replace our phones and laptops.
Gestures
Gestural sensing — where a user’s hand, arm, and/or facial movement is tracked and translated into on-screen actions. Imagine being able to control your devices (no headsets or controls) just by flicking your hand or moving your eyes.
Apple has a patent that hints at a gesture-based interface for future Macs — for example, while on your Mac, you’ll be able to make the window you’re looking at active, so you don’t need to Command-Tab or click to switch windows.
There’s quite a few use cases out there already which use gestural sensing — BMW’s camera based gesture control system is a touch-less hand gesture interface that reduces the need for drivers to reach out to the dashboard control panel. Drone manufacturers like DJI are making photo-taking drones that can fly autonomously from the user’s hand and return without using remote control (you can just summon the drone back by waving hands). Nearly four years ago, Google released a video showcasing how we can use the intricacies of our hands to control the virtual world — e.g. moving our fingers a certain way to increase volume. Surgeons could use the technology to interact with computers mid-operation without having to touch non-sterile items such as keyboards. Lancaster University has created Matchpoint, which turns any object into a remote control — it connects body movements such as moving a cup of tea, rolling a toy car or rotating a spatula to control actions on your TV screen.
Gesture control feels like a more natural way to control our digital worlds, and will improve technology access for anyone who is able to physically move the lightest of objects (thus even an infant). It definitely feels like the most foreseeable future way to complete our tasks, reducing the need for physical screens.
Smart touch surfaces
Researchers at institutions such as Carnegie Mellon have been working on ways to turn pretty much any surface you can think of — from desks to human limbs to entire walls of your home — into smart touch surfaces.
Particularly as the “smart home” develops, this tech will allow us to control our surroundings by “weaving themselves into the fabric of everyday life until they are indistinguishable from it.”
And this technology isn’t so far from reality or overly expensive — any flat surface can be turned into a touchscreen using a $200 LIDAR (the technology is commonly used in self-driving cars and drones). A use case could be installing the sensor under a generic conference call speaker (say, a JABRA speaker), which could then allow office tables to become interactive surfaces. The table could tell how many people were gathered around it as well as allow attendees to touch the surface and move their own cursors around graphs on a PowerPoint presentation.
LightRing by Microsoft Research uses infrared tech to detect finger motion and a gyroscope to determine orientation, and it can turn any surface into an interface. You can tap, draw, flick and drag on a book, your knee, or the wall.
The team at Carnegie Mellon University also looked for ways of expanding the interactive zone around a smartwatch without making it physically bigger. Their Skin Buttons project uses miniature projectors to display interactive icons on the skin around the watch face. The projector parts cost less than $2 and can even increase battery life by shifting workload from the main display.
RoomAlive by Microsoft Research uses multiple depth cameras and spatially mapped projectors to overlay an interactive screen from which there can be no escape.
Given the accessibility we have to creating smart touch devices, I wouldn’t be surprised to see it used beyond gaming, into our smart homes and office spaces in the next couple of years.
Brain activity
The end game is the complete disappearance of interface altogether — or what is known as ‘Zero User Interface’. A Zero UI world is one where users do not have to understand how a machine works, but rather one where machines understand how users work and what they need.
Arnav Kapur, a student in MIT’s Media Lab, has developed a system, AlterEgo, to surf the internet with his mind — he is able to google anything just by thinking it, and hears the answer through vibrations transmitted through his skull and to his inner ear.
While AlterEgo is a wearable device, the next step to this would be embedded implants or chips — Elon Musk’s proposed Neuralink technology plans link our brains directly to computers and other electronic devices, via surgically inserted brain implants that allow your mind to interface with gadgets and programs. Musk believes that brain-to-machine interfaces are an important part of humanity’s future (preferably without the possible negative AI impacts who may ‘take over our brains’) — and Neuralink wants to find the best possible ways to make that mind connection possible.
While the closest we’ve gotten here are advanced bionics that can interpret brains signals and allow people to control limited movements for prosthetic arms and legs, surgery and brain implants are still going to be a big first step.
A single interface
Ok, so right now we’re still in a world where using screens is the status quo. But what if we don’t need to carry around multiple devices, which all have their own independent core systems, and instead you’d just keep 1 small core (imagine a USB stick) with all your information? I came across this thought on John Chiappone’s article, who has been looking for a single computer to replace all his devices and adapt to his every need.
He explains that our preferences, content and software are what really matter, not screens. He thus proposes a core device that acts as your digital duplicate to easily store and carry your info. It won’t itself be a smartphone but will be able to connect to one — or any other device.
Placing all of the power in one central device allows for a modular ecosystem that is less wasteful. One of the smartphone era’s biggest sins is being locked into the never-ending upgrade cycle for a slew of devices. But by centralizing user data and processing within the core, consumers would have far less to replace. They could upgrade their core and enjoy a boost in speed and capabilities across all devices it powers, prolonging their usefulness while reducing waste and spending.
I’d say you can think of this as a kind of VPN (Virtual Private Network), but better. We’d still need a few screens, but they would act as shells, rather than their own entities.
Final thoughts
Writing this while sitting at my laptop, I do wonder how exactly I would perform a similar task, or do something as simple as writing an email, without a screen. Would I dictate it? Would I think it? I like to see what I’m doing, would I be shown a hologram or turn my table into a smart touch device when I want to review and be able to make updates with some gestures?
And another question is — do we actually need this shift away from screens?
Windows Insider Program chief Dona Sarkar discussed her vision of mobile computing beyond smartphones, and emphasises that: “… as humans it is … unnatural for us to stare at a screen. Designing for a screen is kind of ridiculous. There’s only so many ways to navigate. It is important that people grow skills where they can build and use technology that is not behind a screen.”
Current need or not, we will find ourselves trying to close the interaction gap between humans and computers as much as possible. I trust that screens will stay an important part of this interaction, but will not be at the core. The Carnegie Melon FIG (Future Interfaces Group) makes the point that our current devices are too focused on specific actions and are too oblivious to what is going on around them — their goal is to add contextual awareness to devices, where the main interaction will be through gaze, gestures — basically allow for verbal & non-verbal communication, like a ‘good human assistant’ would do.
While there’s no hard and fast answer to what exactly our future interface will look like, it can be a fun exercise to stop yourself when interacting with your everyday screens (say a screen on your treadmill showing you your stats, or the screen at the tube station with a list of stops) and imagine another way you can consume all of this information. Who knows, you might end up helping define our future interactions with computers.
