Posted on

Pose Recognition Unity SDK

This was another Unity Plugin SDK project for iOS and Android, but with particular emphasis on optimisation.

Nexus Studios were using Tensorflow in an AR dance game to track player’s movements and apply particle effects as they moved. For this to perform well, data needed to be sent to Tensorflow and the resulting pose data retrieved back as quickly as possible – otherwise the effects would be out of sync with the player movement.

This involved optimising the AR image pipeline, the C# to native interop, image manipulation within Tensorflow itself, and delivering the data back to my C# API for the game to use.

Posted on

MARS Lightgun Unity SDK

Another Unity native code Plugin project – this time across PC, PS4 and XBone! As well as developing the cross-platform C# plugin SDK, I had to also develop the underlying native console device drivers, delivering USB data from the MARS hub up to the C# SDK for parsing, resulting in Unity-friendly events and delegates.

Posted on

Headcaster

Headcaster was a next-generation messaging app for iOS and Android, allowing you to deliver animated, lip-synced messages via a variety of characters – ranging from cute kittens, through to the zombied reincarnation of Margaret Thatcher. Celebrities use the app too – notably Stephen Fry and Rio Ferdinand – to deliver podcasts via custom made caricatures of themselves.


We also developed a desktop version for broadcast purposes, for a series of short satirical pieces for BT Sport. This allowed for camera movements, green backgrounds and multiple characters.


We developed this over 18 months using Unity, with myself as CTO overseeing a team of six coders and 4 artists whilst developing native code plugins for video sharing. Challenges included building a patented lip sync system which worked across all languages, and building a server architecture able to withstand spikes in traffic from Rio tweeting a headcast to his 7 million followers.

Posted on

Playora

Playora is an iOS and Android app allowing you to play PC or Flash games on a phone/tablet, or stream content from a PC or phone/tablet to a Smart TV.


Challenges included low-latency video encoding, so games could be played without any noticable lag, and DLL injection coding, effectively hijacking a game or app’s visual and audio output from within the app itself, rather than simply recording the screen.