Microsoft has a long tradition of spicing up relatively dull product announcements with compelling tech demos, and the Windows 10 announcement was no exception. The software giant used the opportunity to create a fair amount of buzz about the HoloLens, a futuristic headset that offers a glimpse into the future of Augmented Reality (AR). However, Microsoft also has a tradition of spectacular hardware flops, which peaked under the Ballmer regime. Remember the Kin phone? Neither do I.
The introduction of HoloLens probably won’t be such a flop for a number of reasons. First of all, the HoloLens still has a long way to go before it becomes a commercially viable device – it could be about a few quarters, or a couple of years. Secondly, the concept behind it is sound, and builds on a few promising emerging industry trends, such as wearable tech and Virtual Reality (VR) headsets. The HoloLens is trying to be somewhat different by bundling a lot of functionality into a single device, but in this Microsoft HoloLens review we will take a look at what’s already out there and what is in the works.
Since this is an engineering blog designed for VR professionals and other engineers, I won’t spend much time answering the question “What is HoloLens?” and explaining the difference between AR and VR. Augmented reality technology has a range of potential applications in various industries, but limited applications in entertainment. Virtual reality is more geared towards entertainment, although it has some professional applications as well.
Both technologies still have a lot of limitations, and numerous technical challenges have to be overcome in order to gain mass market appeal. This is a gradual process that will take years rather than months. The technology needed to create such products without breaking the bank is simply not ready, but it’s slowly getting there.
Let’s take a look at what’s out there and what’s missing.
Hardware Limitations – Google Glass vs. Oculus Rift vs. Microsoft HoloLens
Google Glass was announced in early 2012 and started shipping a year later at a cost of $1,500. The high price tag meant it was reserved for a very small niche – early adopters described as “explorers” by Google’s PR and marketing machine. The device offered limited AR functionality, and contained a small prism projector with a resolution of 640×360 pixels, powered by an outdated processor.
While it managed to captivate the public for a while, Google Glass can hardly be described as a success. App developers who were keen to jump on the bandwagon started losing interest, along with “explorers” who appeared to get over the fad in a matter of months. The latest rumours point to a new version of Google Glass with Intel silicon inside, so it might be a bit too early for an obituary. Either way, Google Glass was not a big success no matter how you look at it.
Oculus Rift is perhaps the most talked about VR system at the moment, but unlike Google Glass, it has yet to launch. Oculus VR has been working on the device for years, and in the process the company went through two generations of development kits. The consumer version is expected to launch sometime in 2015, with a revised spec. In March 2014, Oculus VR was bought by Facebook for more than $2bn in cash and Facebook stock.
Samsung’s Gear VR offers a different approach, as it utilises the Galaxy Note 4 phablet in lieu of a built-in screen, but it relies on some technology developed by Oculus. I find the modular concept interesting, as a similar approach could be employed with a range of mobile devices from different vendors that would allow users to effectively upgrade hardware every time they got a new phone. Qualcomm’s Vuforia platform boasts some promising features for mobile devices and potential AR/VR applications.
So, what’s missing? The simple answer might be processing power, but it’s a bit more complicated than that.
The problem with both concepts is that they are still ahead of their time, and the technology still needs to catch up. Microsoft’s HoloLens is bound to suffer from the same teething problems, but Microsoft’s concept is somewhat different, and therefore stands a chance of overcoming at least some of these issues.
Google Glass was designed as a lightweight wearable, which resulted in a number of compromises. The device featured a single display on a thick prism in front of the user’s right eye. The resolution was very limited given the field of view (FOV). For example, smartwatch displays tend to feature similar vertical resolutions for a device that takes up just a couple of degrees of the user’s field of view. Google Glass was based on an antiquated System on Chip (SoC) and had limited battery life.
Designing mobile devices is not easy, and always involves a number of trade-offs. Higher resolution displays require more GPU power, necessitating the use of bigger SoCs with more powerful GPUs working at a higher load, which then requires a larger battery and so on. It’s a fine balancing act, and an AR headset is simply too small to accommodate a large battery like those used in high-resolution tablets.
At first glance, Oculus Rift does not appear to suffer from similar shortcomings on the hardware front, since it does not have compromises for the sake of battery life and portability. It does not rely on an integrated SoC, and a 1080p display sounds desirable; but, in reality it’s not nearly enough for photorealism. The device has a very large FOV, and pixel density is still insufficient.
To overcome this problem, VR devices would have to use higher resolution 4K/UHD displays, or even 8K displays at some point in the future. The technology is almost there, but it does not come cheap, and is anything but portable.
If you want to run the latest AAA games on a 4K display with the highest possible detail settings, you need two high-end discrete graphics cards. For example, Nvidia and AMD cards based on flagship Maxwell and Hawaii generation GPUs. To eliminate frame tearing (using technologies similar to Nvidia’s G-Sync or AMD’s FreeSync) you need a bit more power, and to do proper 3D for both eyes you need even more GPU power.
The bottom line is: to power a 4K VR device using currently available technology, you would need at least two GPUs with a total of 12-14bn transistors in 28nm, consuming 350W to 500W of power, not counting the CPU and rest of the system. This is a conservative estimate, based on currently available GPU and CPUs – and let’s not even discuss the idea of powering two 4K screens, one per eye.
Nvidia’s latest mobile SoC, the Tegra K1 64-bit used in the Google Nexus 9, features 192 CUDA cores based on Kepler architecture, not the more efficient Maxwell. The company’s current flagship discrete graphics cards sport 2048 Maxwell CUDA cores running at higher clocks than Kepler cores in mobile Tegra SoCs.
Portable VR devices with photorealistic graphics are clearly unavailable for years to come, and even wired devices like Oculus Rift have a long way to go. The overall platform cost is another concern. Gaming PCs capable of pumping out playable frame rates at 1080p are relatively cheap, since mainstream GPUs are fast enough to do the job. But at 2160p you need four times the GPU muscle, backed by more memory and a faster CPU.
There is another way of tackling this problem, and I will go over it later.
So What Did Microsoft Get Right?
Remember Facebook’s Oculus Rift deal I mentioned earlier? Just a few days after it was announced, it emerged that Microsoft bought intellectual property (IP) assets related to augmented reality and wearable computers from the Osterhout Design Group (ODG). Some of the patents covered “see-through near-eye display glasses” with a partially transmitting optical element.
In other words, Microsoft bought the IP needed to create HoloLens; and the deal reportedly covered dozens of ODG patents, including a few dozen more patent applications in progress. Meanwhile, Oculus VR is said to have just a single patent, which vaguely describes “a virtual reality headset”.
Microsoft appears to be trying to get the best of both worlds – a wide FOV usually associated with VR devices, and a transparent display surface suitable for AR applications. The approach should allow HoloLens to utilize a lot less processing power than VR devices, while at the same time offering more functionality thanks to the wide FOV. Instead of trying to render photorealistic content, HoloLens could get away with a slightly lower resolution and image quality due to the limited opacity of displayed content. There is no need to create an illusion of reality, so there is a lot less hardware overhead involved. A lot of off-the-shelf technology could allow the HoloLens to reduce or eliminate aliasing and generate good looking composites, since the backdrop is already there.
This fact limits HoloLens’ appeal in the entertainment niche, as opposed to true VR headsets; but, it opens up a number of possibilities in other industries, ranging from engineering and healthcare to architecture and defense. HoloLens could be used to assist healthcare professionals, engineers, operators of industrial machinery, soldiers, and law enforcement.
However, HoloLens still has applications in the consumer space. Microsoft’s Phil Spencer said HoloLens needs to be a successful standalone product, adding that the company is already looking into ways of using it in unison with PCs and Xbox One consoles. The device could serve as a heads up display (HUD) for gamers, or even for fitness buffs in gyms.
Top of Form
Bottom of Form
HoloLens Hardware Conundrum
Microsoft has not revealed the exact hardware specifications, so we still don’t quite know what to expect. There is no word about display resolution, GPU GFLOPs, connectivity, or battery life. This leaves a lot of room for speculation, which the tech press is happy to fill with column inches and clickbait, but nothing is official yet.
Like I said, HoloLens should not require nearly as much GPU power as the Oculus Rift and similar VR products. However, this does not mean that Microsoft can get away with a cheap SoC, like the ones commonly used in mobile products. Microsoft currently uses a range of chips from different vendors – Qualcomm Snapdragon SoCs with integrated 4G/LTE for mobile phones, Intel chips for Surface Pro tablets (along with Nvidia SoCs on defunct Surface RT products), along with custom AMD APUs in the Xbox One.
Due to power considerations, the most obvious choice would be a Snapdragon SoC, similar to those used in Lumia phones. This does not mean that HoloLens would be as underpowered as Google Glass. HoloLens is a much larger device, with room for a bigger battery; and, the latest Snapdragon SoCs are vastly more powerful than the chipset used in Google Glass (which is significantly slower than chips used in smartwatches). Early benchmarks indicate that the Adreno 430 GPU used in Qualcomm’s upcoming flagship SoCs, like the Snapdragon 810, is a powerhouse capable of handling 4K resolutions and rendering relatively complex 3D content in 1080p.
It’s not just about sheer rendering performance. GPUs offer a lot of computing potential, and can be used for much more than gaming. Google used the Tegra K1 for Project Tango, which also deals with a number of technologies that could be very useful for AR or VR devices – automation, driverless cars, and so on. I already mentioned Vuforia, and there are other players in the GPU industry, but Nvidia has the advantage of using CUDA cores – it’s been a market leader in professional graphics and GPGPU compute markets for years.
However, we should not be locked into the “what’s out there” mindset. It will take a while before HoloLens goes on sale, and subsequent generations are bound to feature even more powerful hardware. Intel’s new 14nm Atoms are coming soon, while ARM-based 14nm and 16nm SoCs should appear a couple of quarters later. The new non-planar nodes will allow even more performance per watt, drastically improving overall performance without taking a toll on battery life.
Streaming As An Alternative
There is also an alternative which I mentioned earlier – cloud computing and streaming could be used to display complex, resource-intensive 3D content. The latest SoCs feature 802.11ac wireless and fast LTE modems, sufficient for high-resolution streaming. The downside to this approach, especially LTE, is lag.
If additional content is rendered locally, on a PC workstation or possibly even an Xbox One, lag should be limited, but remote cloud rendering could prove problematic. For example, Nvidia is trying to tackle this problem by setting up GRID servers at strategic locations, in an attempt to cover the biggest markets with low-lag game streaming. Just a few milliseconds of additional lag could compromise the user experience in an AR application.
A mobile SoC should be sufficient for most everyday tasks, such as Skype and some limited augmented reality applications. However, if an architect wants to walk into a construction site and see how the finished building will look using augmented reality, the HoloLens will have to be backed by more hardware; rendering complex scenes with hundreds of thousands or millions of polygons, advanced lighting effects, and so on.
The upside is that HoloLens could offer a lot of functionality out of the box, with a relatively powerful integrated GPU capable of handling a lot of everyday tasks, such as high resolution video streaming, browsing, and even casual gaming. On the other hand, professionals could employ 802.11ac or LTE to stream more complex content, rendered remotely.
Microsoft could practically use the same hardware platform for home users and professionals, with the latter employing local or cloud streaming for more advanced, resource-intensive tasks.
Is There A Use Case And A Market For HoloLens?
Microsoft showed off the HoloLens in a number of different scenarios. While the demos were quite interesting, they did not exactly spell out a realistic and commercially viable use case for the new device.
What I like about HoloLens is the fact that it is halfway between true wearables, like Google Glass, and wired VR solutions, like Oculus Rift. HoloLens does not have to be light and portable enough to wear on the street, but at the same time it does not have to be tethered to a computer or external power source – the best of both worlds. I also like the fact that Microsoft is choosing to lead rather than follow. HoloLens differs from existing concepts and products; it’s innovative, futuristic, and original – a breath of fresh air from Redmond.
However, this approach also raises a number of important questions about the use case for HoloLens, and the size of the market. It can’t replace a display like VR solutions, yet it can’t be used in everyday situations due to its sheer bulk and appearance. While you may see some commuters and athletes using smart glasses, you probably won’t see skiers or joggers wearing a HoloLens headset.
What could mainstream users do with HoloLens? What sort of software platforms and operating systems will be supported? What about professional applications? What about HoloLens cross-platform functionality, hardware specifications, retail price, and Bill of Material (BOM)?
A lot of questions still have to be answered, and it will probably take a while before Microsoft releases all the information.
Microsoft will have to target mainstream and professional markets at the same time, with the same hardware. Depending on the price and BOM, Microsoft could leverage its Xbox user base, as well as a segment of the PC gaming market, to bring HoloLens products to mainstream users. Marketing such a product won’t be easy if the price is too high, but the user base is there – and it is willing to spend a lot of money on new gadgets. A mainstream market approach would also help get more developers on board, thus expanding the ecosystem and creating new use cases.
But if HoloLens products are bound to be priced for the mainstream market, how will Microsoft go after the professional market, and make some money in the process?
Years ago I used to make a living in offline 3D graphics, and I can see a lot of potential in HoloLens. There are a lot of 3D/CAD users out there and many of them would agree. Does this mean that every designer will be able to pick up a HoloLens device priced for the mainstream market and use it for work? Possibly, but probably not.
There are other ways of marketing products in this space. I’ve been covering the GPU space for years, and in that time I’ve learned a thing or two about how the industry operates. Although high-end graphics cards for gamers get all the headlines, the real cash cows for Nvidia and AMD are professional graphics and compute solutions. They are the unsung heroes in this duopoly. The BOM for a consumer card and a professional card based on the same GPU is roughly the same, but professional cards cost a lot more, an order of magnitude more. They deliver huge margins, and generate a lot of revenue and profit, in spite of low overall volumes – you can check any Nvidia quarterly earning report for more info.
Microsoft could resort to a similar approach. HoloLens could use the same hardware for both markets, limit functionality on consumer models, and expand it on professional products through different licensing tiers.
Of course, this is all just speculation at this point – but that’s how this market works. Microsoft does not have to reinvent the wheel.
Story written by Nermin Hajdarbegovic – Technical Editor @ Toptal