Flying a drone has gotten pretty easy. Add the camera, though, and it quickly gets tricky. Perceptiv, a startup that rose out of the robotics engineering department at Canada’s Waterloo University, thinks they’ve got the solution. They’re introducing SHIFT, an add-on that uses computer vision technology to control camera tracking automatically.
SHIFT aims to transform your drone into something that works more like a flying dolly. Here’s how it works: The tracking software on the market today relies on following a GPS signal, which is accurate to within a few meters. Computer vision, however, can visually recognize and memorize the subject you tell it to focus on, even a moving one, and can keep the camera pinpointed with a much higher degree of accuracy than GPS. This means that computer vision has the capacity to make shot framing automatic: If it works, you wouldn’t need to worry about controlling camera aim and tilt; you’d only need to fly the copter.
3DR already automates a lot of the flying and filming experience for our users. Our Follow Me technology, for instance, lets you go completely hands-free: The copter follows you, keeping the camera centered on you to capture your every move. But Neil Mathew, co-founder of Perceptiv, raised a couple of pain points.
The first is the relative inconsistency of GPS tracking, which means the camera can sometimes drift as you move. The second, he says, is that “a lot of people actually like the creative control of flying, framing their shots in a particular manner. They want to retain that autonomy.”*
Debate that conclusion if you wish, but the Perceptiv team came by it honestly. While at Waterloo they used drones to film their robotics projects in action, but grew frustrated. “Flying was pretty easy,” Neil said, “but add the camerawork and it was like you needed two people mindmelded to do this thing. We had backgrounds in computer vision, and so we naturally wondered if that would allow us to control the camera better.”
SHIFT has two components, a processor that sits on top of the drone and plugs into the autopilot and a tracking sensor that plugs into the camera. You use your remote controller as usual to fly the drone, but when paired with the SHIFT phone app the two devices talk to the drone together. The app gives you a live camera feed on your phone; tap on any subject you want to follow and SHIFT will remember that subject, learning it better over time. The app communicates with the processor via WiFi, and Neil said they’ve achieved ranges of 200–300 meters.
Originally SHIFT was only compatible with DJI drones and IRIS+ (and presumably other Pixhawk-powered systems), but due to demand the Perceptiv team has had to scramble to make the technology compatible with a spate of other devices. With DJI they had to spoof the RC signal, but when it came to IRIS+ Neil said, “The Pixhawk is just awesome. It’s easy to plug in, and it’s open so it gives you control.”
To be fair, a number of companies are in the race to develop highly-functioning computer vision in drones, which could power real-time 3D mapping of the world around you and expedite the development of near-perfect sense-and-avoid technology — the latter still being sort of a holy grail for the drone industry. Among SHIFT’s competition are 3DR partners Qualcomm and Intel, with whom we demonstrated a computer vision version of a Follow Me flight last fall. The difference with Perceptiv, however, is that in SHIFT they’ve developed a market-ready product — well, not quite: SHIFT ($599) is currently on pre-order and will ship sometime in the fall.
But Perceptiv is realistic. There are still a number of hurdles to overcome in making a consumer computer vision product that can reliably negotiate the infinite unpredictable and mercurial conditions in the real world. “And the industry isn’t quite there,” Neil said. “Right now production is cost-prohibitive. Not many people are going to pay more than a drone for something to put on a drone.”
Still, to us, Perceptiv’s future looks awfully bright. Does computer vision work with shades?
To learn more about SHIFT check out the video below, or visit their website, which has a list of supported devices.
*Hope the Perceptiv folks forgive me for compulsively pointing out here that Follow Me allows you to customize and change camera angles, even mid-flight, for a dynamic filming experience. And with the copter following you, you’re actually still in control, in a sense piloting it with your movement.