Synching PouchdDB with remote DB

September 16, 2016 0 Comments couchdb, browser, pouchdb, docker

Synching PouchdDB with remote DB

How to run a database locally that syncs with central one.

If we want to control our own data, we must know how to run our own database. To make this more attractive, let us pick a database that knows how to sync with our mobile or web application. Then we can get an advanced feature - offline data support almost without any extra work.

I will show how to run a Couch database locally and in a server (using Dokku), and then how to write a simple web application using VueJS that works with a local in-browser Pouch database. The Pouch database keeps the local browser in sync with the remote Couch database.

A basic knowledge of Docker is required to follow this blog post.

Running local CouchDB

Let us try out CouchDB locally inside a Docker container using the frodenas/couchdb image. To be able to easily replicate work, I saved the shell command into a file

This gave me a running DB named "c1" in about 15 seconds. Most of this time was spent by the docker downloading the base image binaries.

Right away you should be able to see empty database

Let us connect to this database using the JavaScript PouchDB library.

This gives us the same information as did the curl command.

Note: if the Docker is restarted, the new container with the CouchDB will be stopped. You can simple run docker restart 7dd6baed4d70 to get it running again.

Populating data from command line

Since the database is empty, let us insert a couple of fake todos.

We should get 3 documents in the database after running this code.

We can see them by running db.allDocs() method and printing the returned rows list

You can also see this data by fetching it from the CouchDB directly using url curl http://localhost:5984/c1/_all_docs?include_docs=true or by opening the url http://localhost:5984/_utils/database.html?c1 in the browser.

PouchDB just uses the CouchDB as its data store.

Running PouchDB in the browser

CORS: controlling who can call the database

Note: this step is optional, as we are going to "hide" the database behind a dedicated server later on.

Before we can access our CouchDB from the browser, we need to enable cross domain requests. There is a tiny Node utility package to do this, which needs the user name and the password we have used when we created the CouchDB.

You can check the status of the CORS by going to the database web interface at http://localhost:5984/_utils/config.html, the selecting "login" and using the username and password from the database setup step. You should see the following settings enabled

You can limit the domains by editing the "origins" value and specifying a list of domains that can access the database directly.

Using the CouchDB from the page

Now let us create a page and fetch the documents just like we have done before.

If you open this index.html in the browser you should see the 3 items printed in the console. PouchDB works fine in the local Node environment, as well as in the browser.

Small Todo web application

Let us make a tiny web application to show the fetched Todos and be able to add new ones. I will use Vue.js for this, but I will not implement the full example. If you want a full example, see Vue examples page.

All we will do is fetch the items from the PouchDB and then put them into a table.

This gives us the list in the browser, but it is static - if there is an update to the data in the database, then new items do not appear in the web application; the user must reload the page to fetch the data again.

Data replication

Let us setup two way data replication - if the CouchDB gets new data, we want to the data to propagate to the PouchDB in the browser, and vice versa. The PouchDB has a good replication documentation page. In our case we want to keep the browser application in sync with the CouchDB, and even handle the case when the browser loses network connection (offline mode).

Instead of working directly with the remove CouchDB we will work against a local replica

The web application works as before, but now we can stop the CouchDB Docker container but continue working like before in the browser.

The data still loads in the page, but shows the failed network sync requests if we have XHR logging enabled in the DevTools

If we start the container, the errors stop.

What happens if we create new data items in the CouchDB database? Let us use our Node program to add 3 more Todo items and observe the PouchDB event callbacks in the browser. The browser console shows 3 new events

The change property includes the actual new data object. While we could use the change directly to update the vue.data.todos list, we might as well just refetch the data, since this is now a local browser data fetch! Here is the relevant web application code

Here is the fetch in action; as the new items are inserted into the CouchDB, the browser PouchDB gets the updates and refreshes the list.

Run CouchDB in the cloud

I am a big fan of Dokku and will use it to run CouchDB on a personal DigitalOcean droplet. I have already described how to setup a Dokku instance, so I will skip the general introduction.

I will install the CouchDB plugin from the list of Dokku Community plugins. Since I do not have an application (CouchDB has built-in web server), I will just create the service and will expose it to the web.

The CouchDB is now running available at the direct IP address <ip4.com>:6690 which is not the best practice of course, but would work for now. Let us insert 3 todo records - just change the database url in the index.js. We can even use the domain name instead of IP4 address.

We can see the items using the CouchDB built-in web browser, just open in the browser url http://<domain.com>:6690/_utils/database.html?c1

Limit direct access via CORS

Note: again, you can safely skip this step as we are going to proxy calls to the database through a server.

We need to enable our browser client to call the CouchDB interface across the domain boundary. Login into the Dokku host and lookup the CouchDB password.

The DSN field has the long random password generated for us when we created a CouchDB service instance. The username is the database name "c1".

Now go back to the CouchDB web configuration interface and change the "cors|credentials" value to "true". The at the bottom click "Add a new section" and add the following sections one by one (copy from local config created by add-cors-to-couchdb utility).

If you are running the PouchDB only from a certain domain, you should enter a specific value instead of a wild card. Finally, change 'httpd' section value 'enable_cors' to "true".

The PouchDB running locally now should sync with the CouchDB running in the cloud!

Restricting database access

Exposing the database to the whole world is less than optimal. Plus we need to serve the web page somehow. We can solve both problems by writing a tiny static HTML page server + proxy to redirect the database requests. Here is a server code that is using Express framework. The server redirects any request to /db/ to a local CouchDB running at the standard port 5984.

The index.html page can be moved into folder /public and just go back to the server to get the data

The application is working locally, let us deploy it.

Log into your Dokku box and create new application

Stop exposing Couch database directly to the outside world. Instead link it to the new pouch-couch app.

The command shows long unique URL string we should be using to access the Couch database from our proxy application. We should use this variable name in the server.js if available.

Add simple "Procfile" to the repo to let the hosting environment know we need a web process

Deploy the application to the Dokku server by pushing it as a Git remote

If we browse to http://pouch-couch.<domain name>.com we will see an empty HTML page (there are no tasks yet), and if we browser to http://pouch-couch.<domain name>.com/db/ we can see the CouchDB database system information.

We should quickly protect the application from any interception by installing SSL. We can also protect the requests to the database in the server code if we wanted to.

Result

We got an isolated database only accessed via the Node server. The database and the server are running in two connected Docker containers on a single DigitalOcean "droplet" server. The web client (or multiple ones) works with local database in the browser, and the PouchDB syncs the data automatically.

Multiple clients can connect to the same url at the same time and have the data stay in sync, even when going offline and reconnecting. The PouchDB synchronization does the heavy lifting, while the web app only works with the local data source.

Additional information

Synching PouchdDB with remote DB


Tag cloud