ggplot2When you’re plotting a lot of data overplotting can sometimes obscure important patterns. In situations like this it can be useful at the exploratory data analysis phase to ‘jitter’ the data so that underlying data can be viewed making it easier to see patterns. In addition, jittering can be a way of ‘anonymizing’ spatial data […]
Continue reading →dplyr's function inner_join() is much faster than merge()Using the merge() function in R on big tables can be time consuming. Luckily the join functions in the new package dplyr are much faster. The package offers four different joins: inner_join (similar to merge with all.x=F and all.y=F) left_join (similar to merge with all.x=T and all.y=F) semi_join (not really an equivalent in merge() unless […]
Continue reading →Although many data science-related projects can be completed with a single software tool we often find that decisions about what tool to use for a project involve weighing a combination of what tool would be “best” for the job, what tools we're most familiar with and whether we already have scripts we can use. As […]
Continue reading →leafletR packageIn a recent post (which you can find here) we identified the first publish date for all spatial packages listed in the Analysis of Spatial Data Task View on the R website. The most recent of these, published in March 2014, is the leafletR package by Christian Graul. We were surprised and impressed that, if […]
Continue reading →rChartsData Driven Documents, or D3 for short, is an incredible JavaScript library for creating interactive data visualization on the web. Earlier this year, for example, we illustrated the power of D3 by interactively linking maps and charts in this visualization. D3, however, can be challenging to work with, especially if you don't have experience with […]
Continue reading →dplyrThe R package dplyr, written by Hadley Wickham is only a few months old but has already become an important part of our data analysis/manipulation workflow, replacing functions that we have used for years. There are several reasons why dplyr is such a valuable tool but most important from my perspective are the following: Speed. […]
Continue reading →We use PostgreSQL/PostGIS to manage a lot of our tabular and geographic data from the US Census. In terms of workflow we will either download a shapefile manually from ftp://ftp.census.gov/geo/tiger/ or, if we’re dealing with more than one file (block groups or blocks for example), we will do this from within R (using the download.file() […]
Continue reading →ggmap PackageOne of the great pieces of the new ggmap package is the geocoding functionality. Other R functions can be used to geocode but they fail to provide detailed output like geocode accuracy which is often critical. You need to know if the lat/long in the output refers to a rooftop location or a city center, […]
Continue reading → data.tableFor a recent project we have been working with a relatively large database of all historical air pollution data for New York City going all way back to the 1950s. Including both daily and hourly measurements the database includes about 10 million records and 20 variables. Using traditional R functions working with a database of […]
Continue reading →