Announcing Programmable Video DataTracks API

October 26, 2017 0 Comments

Announcing Programmable Video DataTracks API

 

 


  • Share and synchronize real-time data among Room Participants.

  • Build shared whiteboarding, collaboration features, and augmented reality apps.

  • Available in Programmable Video’s Peer-to-Peer and Group Rooms today.

  • Media Sync API to synchronize messages with real-time audio and video streams—coming soon.

  • Join us at Kranky Geek on October 27 for a live demo. 

If you’ve built an app with our Programmable Video SDKs, you are familiar with the concept of Tracks. Tracks represent an individual stream of audio from a microphone or video from a camera, shared by a Participant in one of your Programmable Video Rooms.

Today we’ve added another API that should help you make your Programmable Video apps just a bit richer. DataTracks—a simple API for publishing real-time data among Room Participants lets you build shared whiteboarding, collaboration features, augmented reality apps, and more. And coming soon, the Media Sync API will let you synchronize these messages with real-time audio and video streams.

DataTracks for Everything

Whiteboarding. Let’s say you want to let your users whiteboard together during a live audio/video call. First, you would connect your users to a Programmable Video Room so they can see and hear one another. Next, for any user who wants to draw on the whiteboard, have your app publish a DataTrack to the Room. As the user clicks and drags, send DataTrack messages to the other Room participants with the coordinates of their annotations for the app to display.

You can see a simple example of this by installing the Draw with Twilio sample app. The participants simply need to enter in their own name, the name of the common room and hit connect. See below: 

Whiteboarding

Multi-User Augmented Reality. You can also use DataTracks for something way more exotic like, shared augmented reality. Let’s say for instance you’re building an app for interior designers. You want a customer to be able to share their camera with an interior designer, and have the interior designer recommend a new room layout by tapping on the screen to place furniture and other items virtually in the customer’s physical space (using something like Apple’s ARKit or Google’s ARCore for the virtual objects).

In this example, the customer can capture audio and video from their phone and share it with the designer. On the designer’s side, you’re going to capture audio so they can speak back to the customer, and you’ll use a DataTrack to send their touch events and coordinates back to the customer’s device. Then you’ll use ARKit/ARCore on the user’s device to drop in a virtual object—say a couch, a houseplant, or a painting. Want to see this use case in action? Check out Jen Aprahamian’s blog post for all the details.

ARKit Twilio DataTrack App

Media Sync APIs to bridge Augmented Reality with real-time media

“Wait, shared augmented reality…” you say? That’s right! We’re convinced that the widespread availability of APIs for augmented reality is going to make multi-user AR a core use case for real-time video in mobile apps.

With that in mind, we’ll also be adding Media Sync APIs to the Programmable Video SDKs to make it easier to synchronize real-time media streams with external application metadata, like the state of AR objects and user interactions, starting with support for synchronization of video frames with external data in Peer-to-Peer topologies.

If you’re building an AR application, and want to learn more about our Media Sync APIs, make sure to sign up here.

Can’t wait to get started?

DataTracks are available in our Android, iOS, and JavaScript SDKs today.

Under the hood

Peer-to-Peer Room. In a Peer-to-Peer Room, DataTracks work just like audio and video tracks: Each client that publishes a DataTrack will set up a DataTrack connection with each of the other Participants in a Room.

As you can see below, first a DataTrack gets created (as shown by the blue dotted lines) between the participant sending the message and the remote participants. Once the DataTrack has been created, the participant is able to send messages to the other participants (as indicated by the squares and circles).

Peer to Peer

Whenever a Participant calls #send on their LocalDataTrack, that Participant will actually send a message n times, where n is the number of remote Participants who have subscribed to that DataTrack.

Group Room. In a Group Room, the topology is a bit different. As you can see below, once the DataTrack is established, the message goes to from one participant up to Twilio’s cloud, then out to n remote participants.

GroupRooms

Each client that publishes a DataTrack, sets up a single DataChannel connection to Twilio’s cloud. When that Participant calls #send on its LocalDataTrack, it sends the message up to Twilio’s cloud just once, and Twilio handles the fan out of the message to all the remote Participants who have subscribed.

DataTracks in Group Rooms is supported in our JavaScript SDK today; we’ll be shipping updates to our iOS and Android SDKs that add DataTrack support in Group Rooms very soon. All our Video SDKs support DataTrack in Peer-to-Peer Rooms today.

DataTrack messages are capped at 16kb in size. Read more about the DataTrack API in the docs.

DataTrack Pricing

In Peer-to-Peer Rooms, there’s no additional cost to add DataTracks to your Rooms. Of course, the messages you send via the DataTrack API will consume some bandwidth—so if your Room session is relayed using TURN, there will be some additional bandwidth consumed.

In Group Rooms, you’ll be charged $10.00 for every 1,000,000 messages you send through the Group Room, plus the Media Routed cost for any additional bandwidth consumed by the messages. Note that you pay for the number of messages sent (not the number of messages distributed): If Alice sends one message to Bob, Charlie, and Dave, you pay for only one message, not three.

We can’t wait to see what you build with DataTracks!


Tag cloud