I have learned a lot by reading Chris Whong‘s techblog about how he created the fun app NYC Taxis: A Day in the Life. One particularly interesting tidbit he described was how to extract the points and polylines from the JSON output of Google directions. He does a nice job of describing the details and I recommend you read his discussion.
For some upcoming work, I was interested in experimenting with extracting the Google directions. A couple of things to keep in mind if you plan on reading more below: first, this is bit of a hack, done quickly to get a sense of what is possible — a final implementation would not likely include using R, for example. Secondly, you will likely want to read Google’s terms of service to find out what is allowed and not allowed. That said, here are the steps:
1) Retrieve directions in JSON format
Getting directions in JSON format is a simple call to Google Maps API. The call looks something like this:
https://maps.googleapis.com/maps/api/directions/json?origin="XXX"&destination="YYY"where XXX is your origin and YYY is your destination and they can be addresses or coordinates. For this example, we will get directions from one of my favorite places for coffee (Gimme Coffee) to a craft beer bar called Proletariat.
https://maps.googleapis.com/maps/api/directions/json?origin="228 Mott, New York, NY"&destination="102 St Marks Pl, New York, NY"In this case we will simply put the URL into a browser but programatically you might use Python or JavaScript depending on your needs.
2) Extract the encoded polyline
If you take a look at the output you will see that each of the steps have a start_location and an end_location with coordinates. They also include an encoded string for the polylines. Chris Whong's techblog includes a lot of nice discussion about these encoded strings so I'll mostly skip over the details. But what is important for this example is the overview_polyline which looks like this:
"overview_polyline" : {
"points" : "}qpwFxdsbM`D|@z@wClA_EiCw@qCs@_@K}EsAaIuBoAYf@eBpF{Pv@eCaC}AkJiGuBuAyBwA~@wC"You will want to copy the piece that begins with }qpwFxd and ends with }qpwFxd for the next step.
3) Decode the overview_polyline using MapBox's polyline utility
Again, I learned about this useful utility via Chris Whong's techblog. MapBox has created a node.js library, called polyline, to decode strings from Google. Even if you don't know what node.js is or are not comfortable with JavaScript, implementing this step is not that difficult. You will, of course, need to install node.js on your local machine. Then, to install polyline, you simply type:
npm install polylineand then, using the example, from the polyline GitHub page you can open your command window, type node and then use the following code to decode the string:
var polyline = require('polyline');
// returns an array of lat, lon pairs
polyline.decode('_p~iF~ps|U_ulLnnqC_mqNvxq`@') One tiny wrinkle in our case is that we want the result in a text file that looks something like this:
Here is where the hack begins. Given that I'm not a node expert I'm going to pipe the result to a non-formatted text file and then use R to parse. This is not recommended practice -- be warned. I am using the node library fs (file system) to do the piping. To install use npm install fs. Then I use the following:
var fs = require("fs")
var polyline = require('polyline')
var dat = polyline.decode('}qpwFxdsbM`D|@z@wClA_EiCw@qCs@_@K}EsAaIuBoAYf@eBpF{Pv@eCaC}AkJiGuBuAyBwA~@wC')
fs.writeFile("datafile.txt", dat)4) Reformat the output
Since the previous step produced a long series of mostly-unformatted text we need to refashion the data into something we can use. Again, you would probably be wise to spend more time on the previous step and pipe the data to another format. In my case I'm using R to reformat and add a few bits.
I want to add a "time" component and a route ID (time and route ID are not needed for this example but adding them more closely approximates the kind of data I receive from clients). Here is the function for reading in and reformatting the raw data:
reformatData<-function(path, id){
dat<-scan(path, what="character")
#With more time I would have used regular expressions rather than
#this terrible alternative.
dat<-unlist(strsplit(dat, ',4'))
dat[-1]<-paste("4", dat[-1], sep="")
dat<-data.frame(matrix(unlist(strsplit(dat, ",")), ncol=2, byrow=T))
dat$X1<-as.numeric(as.character(dat$X1))
dat$X2<-as.numeric(as.character(dat$X2))
dat$time<-1:nrow(dat)
dat$id<-id
names(dat)[1:2]<-c("latitude", "longitude")
return(dat)
}I then run the function and look at the output
result<-reformatData("D:/junk/datafile.txt", id="route1")
head(result)
# latitude longitude time id
#1 40.72239 -73.99517 1 route1
#2 40.72158 -73.99548 2 route1
#3 40.72128 -73.99472 3 route1
#4 40.72089 -73.99376 4 route1
#5 40.72158 -73.99348 5 route1
write.csv(result, "newdatafile.csv", row.names=FALSE)
5) Use QGIS to create a point or line GeoJSON
I opened the CSV in QGIS. For points, I simply right click on the point layer and save as GeoJSON. For the lines I first use the Points2One plugin and then I right click the created line and, again, save as GeoJSON.
6) Put the points and lines on a map
In this case, I'm adding the points and lines to a Leaflet map with MapBox tiles and it looks like this:
We put the final HTML/JS code for this map on GitHub including the GeoJSON files. here is what the HTML file looks like:
Python implementation of the decode function, in case anyone needs it: https://github.com/mgd722/decode-google-maps-polyline
thanks. was looking for exactly this sort of thing to plot an animated line in mapbox. incidentally, i think mapbox has come up with a package for easy encode/decode. no more data munging! https://github.com/mapbox/polyline