Faster Page-Loads by Prefetching Links During Idle Time

January 14, 2019 0 Comments

Faster Page-Loads by Prefetching Links During Idle Time

 

 

Photo by Steven Arenas from Pexels

The first time I encountered prefetching (or so I thought) was in my UCBrowser for Nokia Symbian E71. Whenever I browsed to a site that has next>> link. Clicking the next>> kinda loads very fast!!

Like it has been there all the while. I don't know how the site or UCBrowser did it, but it was lightning fast and awesome!!! I didn't have to wait for my dreadfully slow network to load the next page.

So when I came across the term prefetching, It occurred to me what happened those days in my browser. The site or the browser was actually fetching the webpage in the next>> link and caching it, so when clicked on it(the next>> link), it loads from the cache rather than requesting network access.

Although not used in this post, I’d also recommend using Bit to build your applications with components faster. Organize your JS and UI components in a reusable collection, and sync them across your apps to build faster. Try it.

Prefetching is the process whereby the browser fetches the resources of a link and stores it in its local cache. When the user eventually requests for the page via the link, the browser serves the user the cached page. This speeds up both loading and rendering of the webpage.

How is prefetching achieved?

It is very simple. We can task a browser to prefetch a resource by using the link tag with rel="prefetch" attribute.

<link rel="prefetch">

Resources which can be prefetched include:

  • Web pages
  • JS files
  • CSS files
  • Media files (Audio, Image, Video, Documents, and Web fonts)
  • DNS lookup

Types of prefetching

We have:

  • link prefetching
  • DNS prefetching

Link Prefetching This lets the browser fetch the resource and store it in its cache. Link prefetching includes is used in :

  • Web pages
  • JS files
  • CSS files
  • Media files (Audio, Image, Video, Documents, and Web fonts)

Here, prefetching is done like this:

<link rel="prefetch" href="/your/webpage/link">
<link rel="prefetch" href="/your/js/file/link">
<link rel="prefetch" href="/your/css/file/link">
<link rel="prefetch" href="/your/audio/file/link">
<link rel="prefetch" href="/your/audio/file/link">
<link rel="prefetch" href="/your/video/file/link">
<link rel="prefetch" href="/your/image/file/link">
<link rel="prefetch" href="/your/document/file/link">
<link rel="prefetch" href="/your/webfont/file/link">

The link tag alone doesn't prefetch the resource. The browser looks for rel="prefetch" attribute in the link tag: <link rel="prefetch">

To specify the resource to be prefetched you need to include the href attribute with the URL path of the resource: <link rel="prefetch" href="/src/pages/pagetobeprefetched.html">

DNS Prefetching This performs DNS lookup during idle time or in the background before a resource is requested.

What is DNS lookup? It is the process by which a domain name is resolved to an IP address. Domain names are transferred as unicode codepoints which are transferred as zeroes and ones as well. When you want to access a webpage via its domain name (either by entering the URL address in your address bar or by clicking on its hyperlink), the IP address is first looked up in a central system, then the returned IP address is what your web browser uses to communicate with the machine. This is important because IP addresses are a stream of 1s and 0s, unlike computers it will be difficult for us to remember strings of 1s and 0s. You need IP addresses in order to communicate with a machine via TCP. So the binary digits were converted to letters so it would help us remember them. These letters are stored as a map in a central system mapping each URL address to its equivalent binary stream

DNS Records in a central system repository
URL address  | IP address
----------------------------
google.com | 101010100101010101
facebook.com | 010101010101010010
twitter,com | 101010101010101010
cnn.com | 101110000000111000

This a table mapping different websites URL to its corresponding IP address digits. When we request for cnn.com, the IP address 101110000000111000 is sent to us. So with this IP address, we can communicate with the computer on the internet.

DNS prefetching does the lookup in the background for every href link. So whenever we click the link, we don’t go through the DNS lookup because it has already been done beforehand. This prevents the delay before the transfer of data begins(latency) whenever we click a link or load a resource outside our domain.

DNS prefetching is achieved by adding "dns-prefetch" to the rel attribute of the link tag.

<link rel="dns-prefetch" href="https://cdn.jquery.js";
<link rel="dns-prefetch" href="https://www.cnn.com/africa/";
<link rel="dns-prefetch" href="https://pexels.com/tag/man
in_red.jpg"

End of theory for now. I will demonstrate how we can achieve faster page-loads using pre-fetching. There are several libraries which you can plug in to achieve the same thing. The most useful ones I have found are:

quicklink by Addy Osmani

Guess.js attempts to use predictive prefetching, we will come to that later. quicklink(the newest kid on block) prefetches links during idle time when the link comes within the viewport. According to its README.md:

  • Detects links within the viewport (using Intersection Observer)
  • Waits until the browser is idle (using requestIdleCallback)
  • Checks if the user isn’t on a slow connection (using navigator.connection.effectiveType) or has data-saver enabled (using navigator.connection.saveData)
  • Prefetches URLs to the links (using <link rel=prefetch> or XHR). Provides some control over the request priority (can switch to fetch() if supported).

We will implement all the characteristics (quicklink) above in our bid to learn how it(prefetching) works. the quicklink source code is very simple to understand, I did on the first glance :)

Note: Most code I’ll show here to demonstrate prefetching where lifted from the quicklink source. So if you didn’t understand the source this post might serve as a guide to know what and why each feature was implemented in the source. Most non-trivial kinds of stuff were left out, I only demoed the main parts. If you haven’t read the source code or know what prefetching is all about, this post will be a solid guide for you to kick-start your journey into it.

First, let’s tick off some points:

  1. We will use the Intersection Observer to detect links whenever it's visible.
  2. We will use requestIdleCallback to run our prefetching.
  3. We won’t go about wasting a user’s data when he is on a slow network or in a save data mode. So we use navigator.connection.effectiveType or navigator.connection.saveData to check for both. If he is on a 3G or 4G network super!!! we launch our attack.
  4. We can prefetch either through <link rel="prefetch"> which we have seen or through XHR(XMLHttpRequest).

First, we start by creating our project folder and initializing a Node environment in it:

mkdir pref && cd pref/
npm init -y

We will need an http-server to serve our pages.

npm i http-server -D

Next, we create a folder demo/ with an HTML file index.html

mkdir demo && touch demo/index.html

Open up the index.html in your fav editor and pour in the following contents.

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Basic demo</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" type="text/css" media="screen" href="/test/main.css" />
</head>
<body>
<div class="screen">
<h1>Basic demo</h1>
<a href="/test/1.html">Link 1</a> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip
ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
<a href="/test/2.html">Link 2</a>
</div>
<div class="screen">
Lorem ipsum dolor sit amet, consectetur adipisicing elit. Eos, quos?
<a href="/test/3.html">Link 3</a>
<section id="stuff">
<a href="/test/main.css">CSS</a>
</section>
</div>
<div class="screen">
<a href="/test/4.html">Link 4</a>
</div>
</body>
</html>

Notice the following <a href="" links. We see some links to 1.html, 2.html, 3.html and 4.html all in a test folder. If we click on <a href="/test/1.html"></a> our browser will navigate us to 1.html page, likewise <a href="/test/2.html"></a>, <a href="/test/3.html"></a> and <a href="/test/4.html"></a>. Pages 2.html, 3.html and 4.html will be loaded respectively.

Those are the links will be prefetching.

Before proceeding let’s do something. Create a test folder and add the following commands to create the following html files

mkdir test 
touch test/1.html
touch test/2.html
touch test/3.html
touch test/4.html
touch test/main.css
How our project folder will look like

Add the following to test/main.css:

body {
font-family: Roboto, Arial, sans-serif;
}
.screen {
height: 100vh;
}

Populate the files with content of your choice. OK.

Let’s write some code!!! First, add a script tag:

/...
<div class="screen">
<a href="/test/4.html">Link 4</a>
</div>
<script>
    </script>
</body>
</html>

Intersection Observer

To observe when the links come into view. The Intersection Observer API provides a way for us to asynchronously observe changes in the visibility of our elements or the relative visibility of two elements in relation to each other. We want if any of the links come into view in relation to the document, the webpage it refers is prefetched.

We define the Intersection Observer.

<div class="screen">
<a href="/test/4.html">Link 4</a>
</div>
<script>
const observer = new IntersectionObserver(entries => {
entries.forEach(entry => {
if (entry.isIntersecting) {
const url = entry.target.href;
prefetcher(url)
}
});
});
</script>

The callback entries=> {...} is called when the a link(s) enter or exit the document viewport. The entries argument is the array of the link tags that either entered or exited.

So above, we loop through them. for each entry we check if the link a is intersecting with the parent viewport, if it is we extract the page link from its href attribute using entry.target.href. Then, the extracted page link is prefetched by calling the prefetcher function.

Right now, our code will fail because we have not defined the prefetcher function.

To see our demo, we can comment out the prefetcher statement and add

observer.observe(document.querySelectorAll('a'));

after our IntersectionObserver declaration.

Also, let’s add console.log to see our entires displayed on our console.

Our little code so far should look like this:

<script>
const observer = new IntersectionObserver(entries => {
entries.forEach(entry => {
if (entry.isIntersecting) {
const url = entry.target.href;
log(url)
//prefetcher(url)
}
});
});
observer.observe(document.querySelectorAll('a'));
</script>

To see it play out:

  • add "start": "http-server ./demo" in the scripts section of your package.json file.
  • Run npm run start at the root of your project.
  • Open your browser and navigate to 127.0.0.1:8080.
  • Scroll up and down; move your mouse above the page in random circles.

OK, I believe you have played enough with it.

Like we said earlier, we will call the observer when the main thread is idle. JS being single-threaded lags when a time-consuming operation is run. We should defer own time-consuming op, to be executed when the main-thread is less busy or idle. Our prefetching will be kinda a long op so we use he requestIdleCallback to run it when the main thread is idle.

The requestIdleCallback API takes a callback and a object with several options on how to run:

requestIdleCallback(()=>{...},{...})

we will run the requestIdleCallback function when our webpage loads. To do that we add this:

<script>
// ...
(function(options) {
requestIdleCallback(() => {
Array.from(document.querySelectorAll('a'), link => {
observer.observe(link);
});
}, {
timeout: 23
});
})()
</script>

We aggregate all the links in an array and feed each to the link=>{...} callback. The callback function calls the observe function in the IntersectionObserver observer instance with the link passed. Next, an object with timeout set to 23ms is also passed to requestIdleCallback, this instructs the function to execute the callback with 23ms. so we should take care to make sure our callback runs within the timeout range.

Let’s do some cleanup,

<script>
const observer = new IntersectionObserver(entries => {
entries.forEach(entry => {
if (entry.isIntersecting) {
const url = entry.target.href;
//log(url)
//prefetcher(url)
}
});
});
//observer.observe(document.querySelectorAll('a'));
//...
</script>

Prefetching

Now, we will add our prefetching function. We can prefetch a webpage using XMLHttpRequest or .

Using XMLHttpRequest requires sending a request to the link. The browser will cache the response, so when the link is clicked, the browser checks in the cache first before attempting an http request. Here is the code:

<script>
function xhrPrefetchStrategy(url) {
return new Promise((resolve, reject) => {
const req = new XMLHttpRequest();
                req.open(GET, url, req.withCredentials = true);
                req.onload = () => {
(req.status === 200) ? resolve(): reject();
};
                req.send();
});
}
</script>

Here, the xhrPrefetchStrategy function returns a Promise, it resolves when there is a server response and rejects when there is none.

Using rel="prefetch", we just programmatically create a link element using document.createElement append the attributes: rel="prefetch" and the href link and finally we append the link element to the head element. Here is the code:

<script>
function linkPrefetchStrategy(url) {
return new Promise((resolve, reject) => {
const link = document.createElement(link);
link.rel = prefetch;
link.href = url;
        link.onload = resolve;
link.onerror = reject;
        document.head.appendChild(link);
});
};
</script>

Also, linkPrefetchStrategy function returns a Promise. It resolves when the link element has been loaded to the current page and rejects when there is a loading error.

Before we choose our prefetch strategy we must know if the browser supports the <link rel="prefetch">, if it supports link rel="prefetch" we use the linkPrefetchStrategy, if not we use xhrPrefetchStrategy.

Why do we have to use link[rel=”prefetch”] over XHR? This is because link[rel="prefetch"] is a hint to the browser that a resource is useful for future navigation. Because it's non-critical, browsers like Chrome fetch these with a low network priority. sync XHR has a high priority and async fetch/XHR has medium priority. Thanks to @addyosmani for this answer on Twitter.

We create a support function that checks if the browser supports prefetching with rel='prefetch':

<script>
function support(feature) {
const link = document.createElement('link');
return (link.relList || {}).supports && link.relList.supports(feature);
}
const supportedPrefetchStrategy = support('prefetch') ? linkPrefetchStrategy : xhrPrefetchStrategy;
</script>

The supportedPrefetchStrategy will hold the strategy the browser supports. Now, we create a prefetcher function, that will call the prefetch strategy.

<script>
function prefetcher(url) {
url = new URL(url, location.href)
return (supportedPrefetchStrategy)(url).then(() => {
console.log(${url} prefetched)
},()=>{
console.log(${url} not prefetched)
});
};
</script>

We construct a new URL object using the document’s script location as the base for relative URLs. We call the supportedPrefetchStrategy function. Remember our both prefetch strategies returns a Promise, so we use the .then() function call to catch them. We create the first callback function to run when the prefetch strategy succeeds(resolves) and the second callback to run when the prefetch strategy fails(rejects). In our console, we either see /test/1.html prefetched or /test/1.html not prefetched.

Now we uncomment the //prefetcher(url) in our IntersectioObserver instance.:

<script>
const observer = new IntersectionObserver(entries => {
entries.forEach(entry => {
if (entry.isIntersecting) {
const url = entry.target.href;
//log(url)
prefetcher(url)
}
});
});
//observer.observe(document.querySelectorAll('a'));
//...
</script>

Data Saver and 2G Network

Our app now works fine, but we need to be data-considerate. We won’t prefetch when the user is on a 2G network or has data saver on.

We will the navigator.connection object to check for data saver setting and 2G network. We will modify our prefetcher function to check for both network and data saver mode. If either of the two is true we return, no prefetching is done.

<script>
function prefetcher(url) {
url = new URL(url, location.href)
if(conn = navigator.connection) {
if(conn.effectiveType.includes('2g') || conn.saveData) return
}
return (supportedPrefetchStrategy)(url).then(() => {
console.log(${url} prefetched)
},()=>{
console.log(${url} not prefetched)
});
};
</script>

The navigator.connection object returns a NetworkInformation containing information about the browser's connection.conn.effectiveType returns the connection type, so check if the returned string contains '2g'. conn.saveData will return true if the data saver mode in on or false if the data saver mode is off.

Now, let’s view our app in its entirety:

In case, you missed us along the way, you can paste this code in the index.html file we created earlier and launch the http-server: npm run start. Navigate to 127.0.0.1:8080 in your Chrome browser(guess everyone has a Chrome browser in their machine these days) and you will see all we have done play out.

To actually check if the prefetching works, you can turn off your network/close the server Ctrl + C for Windows users and try to navigate to a link within the viewport now.

quicklink can only work in vanilla JS/HTML apps because they have absolute route direction between a URL and the content. It won't work in JS frameworks because they manage their own routing. Their routing and navigation systems doesn't hit the server or initiate a full page load.

In React, we map our app routes like this:

<Route path='about' component={About} />
<Route path='contact' component={Contact}>

In Angular:

const routes:Routes = [{
path:'/faq',
loadChildren: './faq/faq.module#FAQModule'
}]

When we want to link to a page, we do this:

<!-- Angular routing -->
<a routerLink='/fag'>FAQ</a>
// React routing
<Link to='/about'>About</Link>

quicklink will not be able to find the faq bundle and preload it. It will not find the About component when the /about link is clicked on the React app.

Looking at these restrictions, Minko Gechev provided a solution for the Angular framework to effectively use quicklink, he created a library ngx-quicklink, which enables Angular developers to take advantage of quicklink and it’s powerful and brilliant strategies it provides.

In the nearest future, we will explore in-depth how ngx-quicklink works, how it was created and how it leverages quicklink to help create a performant app.

In our next post, we will look at predictive prefetching using GA. We will model our demo after Minko Gechev’s Guess.js. Stay tuned.

If you have any question regarding this or anything I should add, correct or remove, feel free to comment, email or DM me.

Thanks !!!


Tag cloud