There has been much talk about the high latency surrounding the Kinect, along with the heavy burden caused by the additional CPU usage that is needed to run the more complicated Kinect games. However this doesn’t have to be the case, and recently in an interview with CVG, Blitz Games and chief technical officer, Andrew Oliver revealed that they have developed a way around the limiting latency factor.
“There are various technologies involved. Some people are using a skeletal system, and it takes a little bit of time to calculate. It’s only a split second. We're actually using a different masking system, which can tighten things up. But this is all software-based, so where some people might see some little cracks, they're easily fixable by software. That is, the camera fundamentally works and gives you the input; game designers are running forward in a completely new area and learning this stuff. It's like any console. The first few games will look like nothing compared to second and third generation.”
He then elaborated on the question of how much lag can we expect to see using their approach, and how cleaver coding can almost eliminate it.
“It depends on what technology you're using. I have seen a few games with a bit of lag, but that is the software choice of the creators; they've programmed it a certain way, and they'll come up with new techniques. We will tighten and tighten it. There doesn't need to be a lag. We can get it down to maybe two frames behind, which is pretty insignificant; you won't notice. We're just learning new tricks. Ours is pretty tight.”
His comments make for interesting reading. Although we have always known that it was down to software in determining how the image is processed, we had no idea that the system in place was so flexible, that the developers can choose what data they want to use from the Kinect.
Ultimately, the way that Kinect works will always produce some lag, even if it is just a very small amount. There is no way around this. The device still needs to provide data to the Xbox 360 console in the form of a depth map, plus RGB camera image, whilst also performing basic set up routines with the sound sensor, before then transmitting it down the USB 2.0 cable to the machine, all of which results in a small amount of fixed lag.
In addition there is more lag added on top of this when the data gets processed by the 360 console. The amount of lag which takes place here depends entirely on how the developer chooses to interpret, and in turn use the data. They may choose not to use the RGB camera layer, instead simply relying on the depth map information, or they could just bypass any skeletal tracking altogether, thus saving on overall processing time but also resulting in lower latency but a more basic motion tracking system as a result.
Indeed, what we now know is that it’s the developer that decides which parts of the system to take advantage of. In effect they can choose to you all, or none of the above tracking methods according to what kind of experience they are looking to create. This means that they could produced a fairly basic game not too dissimilar to something you’d find on the Eye Toy, but with minimal lag. Or something which uses the full extent of the full body tracking available when using the depth map plus RGB camera image combined, resulting in a highly accurate and advanced experience, just at the expense of having noticeably more lag.
For project at Blitz Games they seem to be towing the line between the two. Although we can’t be sure as to how much complex data they are choosing to discard, we do know that to get the overall latency down to one or two frames, that you’ll effectively need to scrap most of the depth map information and forgo advanced skeletal tracking. Essentially, pairing back the image processing down to a bare minimum of what the Kinect can do, whilst also trying to maintain some of it’s more trademark features.
Having a less laggy system is always preferential with regards to any control system in gaming, whether that be a standard control pad or a motion-tracking camera. However, in this case finding a cleaver solution to the issue might also negate some of, or most of the additional features that the Kinect provides over other motion controllers.
You could argue, that what is the point in getting the lag down to one or two frames when you are having to cut back on many of the things which in essence makes Kinect so special. If you’re not going to be doing motion tracking, and just taking advantage of the standard RGB camera then why bother to use Kinect at all. Especially when a simpler solution such as the PS Eye could well be enough to handle the type of software you are trying to create.
So, assuming that Blitz Games aren’t using any kind of depth buffer at all for their title, then Ubisoft could easily convert the game over to the PS Eye with minimal issues. Effectively, if you’re only using a full-colour video stream to create image information for processing then you can do the same using other basic camera systems, thus defeating the point of using Kinect.
However, it is likely that the depth buffer is being used in some way with regards to determining specific object tracking with the final image, even if it isn’t as fully featured as the system used in Microsoft’s own titles. The depth buffer provides valuable information allowing you to single out certain objects for tracking without having to isolate them from an entire full colour image in order to pick out the specific parts in which to track. Using the depth buffer for this purpose would also cut down on the amount of processing that needs to be done. Again, highlighting another plus point for reducing latency with Kinect whilst still taking advantage of some of its advanced features.
All this is simply based on an assumption on what Blitz Games are using with their custom software solution. But without knowing exactly which parts of Kinect they are using, and to what extent, it’s a fairly moot point at this stage. Although it does beg the question of whether it is actually possible to produce a nearly lag-free experience when using any kind of advanced body tracking features with the device.
One thing we can say for sure though, is that titles featuring low latency are probably not using the skeletal tracking system, and that titles which have a noticeable amount of lag are. It’s really that simple, though not in terms of getting the low latency solution to really work effectively.
For Blitz Games solution they have taken to using the GPU of the Xbox 360 along with performing some less strenuous calculations on the CPU. So far most titles we’ve seen use the CPU to do most of the work, and according to Oliver doing it this way can be pretty slow in comparison.
“Well that's interesting, because obviously if you're trying to run your game and look at these huge depth buffers and colour buffers, that's a lot of processing. And it's actually processing that a general CPU is not very good at. So you can seriously loses half your processing if you were to do it that way. We've found that it's all down to shaders, but turning a depth buffer into a skeleton is pretty hardcore shader programming. What you tend to do is write all your algorithms, get it all working in C++ code, and then work out how to now write that in shaders.”
What he’s describing here is how the team at Blitz Games are effectively doing all the depth map processing and skeletal tracking on the GPU using customised shader routines.
“The GPU on the Xbox is very powerful but we've all only been using it for glossy special effects. A really good example of this is Kinectimals, as the most intensive thing that you can do on a GPU is fur rendering. So that GPU is doing all the fur rendering, and I can guarantee that it's also doing a lot of image processing too. It's brilliant that the Xbox has a really good GPU and can handle both these things, but actually writing that shader code to do image analysis is hardcore coding at its extreme!”
Like we mentioned earlier having the GPU handle the processing of all the image data from the Kinect can be incredibly difficult. Although the GPU definitely appears to be more suitable for such a task, it’s just that writing code to take advantage of that requires looking at the problem from outside the box. Now while this is pretty commonplace in PS3 development, on 360 it’s a far rarer occurrence. However it seems that for anyone looking to provide a different Kinect experience, that’s exactly what you will have to do.
Certainly, from the comments coming from Blitz Games’ Chief Technical Officer we can expect titles using the GPU to not only have lower latency, but also retain some of the advanced skeletal tracking feature specific to Kinect. Oliver describes how he and his team are already working within two frames of latency, which should translate roughly to around 132ms lag on top of what your HDTV is already adding to the signal.
Comparatively, top titles which run at 30fps, such as Halo3, are displaying 100ms of lag on top of your HDTV processing delay, and Sony’s own Killzone 2 is said to feature at least 150ms on top of that also. By contrast many first-party Kinect titles are operating at 200ms plus input lag.
When looking at these numbers it is clear that having around 132ms lag in a best case scenario is actually rather good, and is inline with what we expect from most ordinary games running at 30fps. Even Criterion Games’ Need For Speed: Hot Pursuit exhibits at least 100ms of lag at present, with the developers hoping to reduce that down to below the 90ms mark.
Playstation Move also operates within the same basic threshold, with the system capable of delivering games with only around 1 or 2 frames latency; that’s around 66 to 140ms lag on top of your HDTV. This means that through a custom approach to processing data, Kinect is more than capable of matching the Move’s higher numbers with regards to latency.
So, what has become apparent today is that games specifically created for Microsoft’s Kinect don’t necessarily need to have high levels of latency, with the overall complexity of the motion tracking method used signalling just how much lag will or won’t be present.
This also means that there will be a correlation between the kinds of games you can expect to have high latency on the Kinect to the ones that don’t. Titles which use the full potential of the device’s advanced body tracking capabilities, such as yoga or fitness games, are also the ones which will feature noticeably more lag. However, they will also be the ones to really demonstrate how different Kinect is compared to other motion control solutions.
Replicating your movements and translating them to your TV screen accurately is exactly why most people will be buying a Kinect, regardless of the latency. If the experience provides something far in excess of what was possible on the Wii, or on PS Move to the same degree, then it won’t really matter all that much.
What does matter however, is that developers have finally confirmed that they do indeed have a solution to the problem. Performing image analysis on the GPU is clearly said to be faster than MS’s own CPU solution. Plus, there’s always the choice of circumventing some of the full-body tracking in order to use a simpler system whilst still taking advantage of the device’s ability to generate a depth map.
In the end gamers will be sold on Kinect not simply by how much lag they encounter, but by the types of experiences on offer. And if those experiences do indeed manage to bring something fresh and innovative to the table, then adjusting to a little lag here and there isn’t likely to influence their decision to jump in. So, whilst the device might not be suitable for the types of ‘core’ gaming experiences we know and love, that’s not to say that other ones which can possibly cater to those, and casual users, won’t in turn come along and provide the best of both worlds with regards to low latency and advanced body tracking.
Ultimately, it’s going to take one or two years to really see the benefits of this technology, and like when learning new console hardware for the first time, cracks and small niggling issues are going to have to be worked out. Perhaps fittingly, this is also a statement echoed by Blitz Games, who believe that the kinds of experiences that you’re going to be getting in second or third generation of software will be far over and above what we are currently seeing today.
All that’s now left is for developers to really spurn their creativity by fully taking advantage of the various coding tricks and optimisations they’ve learned. And of course, for Microsoft’s marketing department to convince people that Kinect is really worth that £129 price of entry.
No comments:
Post a Comment