Our Road to Native Videos on the Web

February 07, 2019 0 Comments

Our Road to Native Videos on the Web

 

 

Author visualization of the process

Videos are an integral part of the Kitchen Stories experience.

They provide a compelling medium to give inspiration to foodies around the world as well as tips for people only now getting into cooking. So, naturally, we like to have them as integrated into our page as possible.

Historically we have used an unoptimized Vimeo implementation and never got around to refactor this. Meanwhile the web platform and the state of video in the platform have evolved rapidly. Things change, and roadmaps get travelled. During a sprint in December, we finally got around to rebuild our approach to loading videos.

This article will walk you through the state we have been in before and the steps we took to build a better and more performant video experience.

I am going to provide code samples where possible. Additonally there is an embedded CodePen at the end of the article which shows a demo implementation.

We are using Vimeo as a video provider. Having an external service provides a level of comfort for our content team that we could not possibly recreate. Also, Vimeo takes over a lot of the heavy lifting regarding file conversion et cetera. So discontinuing to use it was out of question.

The current implementation, however, has grown over time and as with any software product, this leads to bloat.

Relying on iFrames is a great out-of-the-box experience. However, we have full control over how we render elements on our page. We are box-builders ourselves. After all, a website is made out of boxes.

Videos are files. Files come in formats. One of them has to play in our box. My colleague Aaron Hedges gave some insights into the pros and cons of some of these. With this guidance, we decided on HLS.

HLS stands for HTTP Live Streaming. It’s an adaptive bitrate streaming protocol developed by Apple. One of those sentences to casually drop at any party. Äh. Back on track: HLS allows you to specify a playlist with multiple video sources in different resolutions. Based on available bandwidth these video sources can be switched and allow adaptive playback.

The format can not only be used for streaming but also playback.

Tomo Krajina wrote an excellent introduction to HLS over at the Toptal blog if you want to get a deeper insight into the implementation.

While this is all great I can hear you say «but what about browser support?» It’s, well, patchy. At the time of writing HLS is natively supported by Edge, Safari, and most mobile browsers. Check Can I Use for up-to-date data.

Don’t we care about other browsers? Of course, we do. How we mediated the lacking support is explained in the section Loading a Library Only When Needed.

To recap, we had the following building blocks: Using Vimeo as the host of our videos, HLS as our format of choice for playing these videos and a video implementation that relies on native elements, no iFrames and as little scripting as possible.

Vimeo’s Pro tier allows you to get direct access to your uploaded videos, bypassing the regular embed. Vimeo delivers HLS, DASH as well as multiple MP4 sources.

The complete flow is as follows: Our editors upload the videos to Vimeo and copy the ID into our Content Management System. We have an Express server running which is used to retrieve the ID from Ultron, our API. Using this we get the HLS manifest from Vimeo’s API.

We got all the data we need. Time for rendering. We have wrapped the video element in a div to ensure the aspect ratio. This div is of another use when we start implementing the custom poster image.

View of the initial markup structure

Okay, so now we have a video rendered on screen. With a format that is not universally supported, which makes Firefox rather sad.

Screenshot, taken from Firefox while trying to play an HLS video. The User Interface displays a message saying «No video with supported format and MIME type found»

Let’s make Firefox (as well as Chrome, IE and the rest of the bunch) happy, then.

Performancewise the move to native video has its reasoning first and foremost in the fact that you don’t need a library.

Things get a bit more complicated when using HLS. As shown above, it has its advantages but isn’t supported everywhere.

There are two different workarounds: We can supply a different media type as a fallback source, e.g. .mp4. Providing an MP4 fallback would have cost us on one of our core premises: Improving the performance of the video experience. The format is not adaptive. We would have to care of detecting bandwidth changes, setting sources dynamically, ideally also if the network condition changes during playback and so forth. Stuff that is built into HLS from the ground up.

Fortunately, there is the MediaSource Extensions API. It makes it possible to teach browsers to play media they can’t play natively. Or, more precisely, it allows browsers to handle multi-track video streaming playback.

We decided that the advantages of using HLS outweigh the performance hit we get from patching support. This can, of course, vary from project to project.

Our decision for a library was hls.js as it is actively maintained and at a reasonable size.

What is reasonable you may ask? For us, it was the size of Vimeo’s player script, about 160 kB gzipped. We use the light version of hls.js as we currently don’t have closed captions or multiple audio tracks in our videos, which makes for 59 kB. That’s 100 kB of JS our user don’t have to load and parse. Great.

Okay. Details. First, we detect if a browser can play HLS video. David Walsh wrote an article on video format detection. We simplified it a bit as we have a pretty narrow use case.

Now we can play the video if HLS is supported. If not, we load the library. The loading method resolves a Promise on load. This way the video only starts once hls.js has been loaded.

Using this approach, we load circa 3 kB of gzipped JavaScript in supporting browsers to handle the complete play cycle. Unsupporting browsers have to load a 59 kB (gzipped) script to patch support. A fraction of what we needed to play videos with the Vimeo embed.

The parsing itself looks like this:

We create an instance of HLS, load the source and wait for the MANIFEST_PARSED event. In the event handler, we are setting the parsed flag to true – which avoids parsing the manifest again – and play the video.

The last part is rather straight forward. We hide the preview image — why we need to do this is the topic of the next part — and call the play method of the video element.

The HTML video element has a — in theory — handy attribute called poster. «In theory?» you may ask. The spec defines this attribute as an «address of an image file that the user agent can show while no video data is available». Note the singular.

While developing for a fragmented market like the web this a problem: We can’t specify one image to cater for all our users.

Another thing to note is the part that the user agent should show the poster image «while no data is available». This is great but leaves with you a UI that might feel broken once the video has started loading or is finished.

We decided to ditch the poster attribute and instead build our own. We wrapped the video element in a container and put an image in the same one.

Because we want the playback experience to be as seamless as possible, the image is given pointer-events: none. It ensures that all clicks in the container will have the video as a target and act as a signal to play.

To signify that we have hidden a video we also add an after element with a play icon. On click, we hide the image with a CSS transition.

To hide the image all we need to do is removing the hidden class from the image:

Remember what I said about the native poster image only being visible if no data is present? We can do better. When the video ends, we fade the custom image back in.

Video on mobile is a double-edged sword. They are nice, sure, but they also consume a considerable amount of bandwidth. Having videos set to autoplay, therefore, comes with constraints in mobile browsers. In short: you are only allowed to play muted videos as long as there isn’t a user interaction.

We knew this before and assumed that we would not be able to have a video overlay with autoplaying videos. Luckily, assumptions can be wrong.

Apple calls these interaction gestures (you can find an outline of what makes a gesture in this article on the WebKit blog).

The gesture does not need to have the video as its target. A click on a button or any other DOM element suffices. We have touched on handling inline videos before when talking about the poster image integration. The second way we use videos is inside an overlay. Naturally this makes it impossible to click on the video to play it.

Firefox does not register click events on videos if the controls attribute is present.

As we have the faked poster image with a play icon on it we don’t need to show the controls immediately. We render the video tag without it, handle user interaction as shown above, and add the controls back in once the user clicks on the video.

One last tweak targeted mobile browsers. Videos are part of our User Interface, and we want them to play like that. The default behaviour is to play a video in an overlay. We used the playsinline attribute to disable this behaviour.

Going through all these steps was a whole lot of work. Was it worth it? We think so.

Native video elements under our control are seamlessly integrated into the operating system — think picture-in-picture player, controls through the multimedia keys of the keyboard.

Most mobile browsers don’t need to load a large batch of JavaScript, to play the video, while browsers without native support get an alternative that is functionally equivalent. Moreover, the script is only loaded if a user tries to watch a video. Just reading? No problem.

We were able to dramatically cut down on the amount of data loaded on some of our video pages; the places where multiple kind-of-but-not-really-the-same-implementations met.

If you want to play around, here is a fully functional CodePen:

Hey, you made it to the end. Seems like you like frontend development. We do, too. And we are hiring. If you have experience with Vue, CSS and strive to create great user experiences, we would love to hear from you.

Image credit: Title collage made out of photos from Simone Hutsch and Gabriel Petry.


Tag cloud