PostgreSQL allows you to easily dump tables for backup purposes into plain text files using code such as: pg_dump mytable > mytable_backup.sql But this process is not very flexible and can create very large output files. As an alternative, PostgreSQL has a nice little feature that allows users to export to what is called a […]
Continue reading →gdalAlthough we still use ESRI products such as ArcGIS for geographic analysis, more and more we find ourselves turning to open source software like PostGIS and QGIS. In many cases, though, even PostGIS and QGIS are more software than necessary for a given task and we prefer to skip the ‘middleman’ in favor of the […]
Continue reading →For a recent project I was conducting some fairly dry quality assurance/quality control. In particular, I was comparing whether the population-weighted centroids of ZIP codes (ZCTA, to be specific) we computed in PostGIS matched those computed using QGIS. Given that I was creating temporary, internal maps, to make things more fun, I used the OpenLayers […]
Continue reading →AngularJS is a powerful tool for creating single page web applications, particularly data driven applications. One of the most powerful features of Angular is its two-way data binding, the automatic synchronization of data between the view and the application itself. In this example, we demonstrate the two-way data binding using a table and LeafletJS map […]
Continue reading →The Python library BeautifulSoup is an incredible tool for pulling out information from a webpage. You can use it not only to extract tables and lists but you can also use to to pull out very specific elements like a paragraph with a green font color. To briefly illustrate this functionality and in honor of […]
Continue reading →For a recent project we were provided with a GIS shapefile of developments (primarily houses) in a suburban neighborhood in Ithaca, NY. We created static maps to show change through time with color coded dots for different time windows but we thought that the dataset was a perfect opportunity to show off CartoDB’s time series […]
Continue reading →ggplot2When you’re plotting a lot of data overplotting can sometimes obscure important patterns. In situations like this it can be useful at the exploratory data analysis phase to ‘jitter’ the data so that underlying data can be viewed making it easier to see patterns. In addition, jittering can be a way of ‘anonymizing’ spatial data […]
Continue reading →CartoDB is a great service for generating online maps and is particularly useful if you’re working with big data that might not render quickly using APIs such as Leaflet and Google Maps. For details on CartoDB we did a write-up of the nice functionality here. In working with CartoDB layers, though, one of the challenges […]
Continue reading →dplyr's function inner_join() is much faster than merge()Using the merge() function in R on big tables can be time consuming. Luckily the join functions in the new package dplyr are much faster. The package offers four different joins: inner_join (similar to merge with all.x=F and all.y=F) left_join (similar to merge with all.x=T and all.y=F) semi_join (not really an equivalent in merge() unless […]
Continue reading →Although many data science-related projects can be completed with a single software tool we often find that decisions about what tool to use for a project involve weighing a combination of what tool would be “best” for the job, what tools we're most familiar with and whether we already have scripts we can use. As […]
Continue reading →