Overview

pyopendata is a Python utility to offer an unified API to read world various data sources, and output pandas.DataFrame.

This is an unstable release and API is forced to be changed.

Installation

Use pip.

pip install pyopendata

Basic Usage

This section explains how to retrieve data from website which uses CKAN API.You can create DataStore instance to access CKAN website by passing CKAN URL to DataStore class.

In this example, we’re going to retrieve the ‘California Unemployment Statistics’ data from data.gov. The target URL is:

We can read abov URL as:

  • CKAN API URL: https://catalog.data.gov/
  • Package ID: california-unemployment-statistics
  • Resource ID: ffd05307-4528-4d15-a370-c16222119227
In [1]: import pyopendata as pyod

In [2]: store = pyod.DataStore('http://catalog.data.gov/')

In [3]: store
Out[3]: CKANStore (http://catalog.data.gov)

DataStore.serch performs search by keyword. Results will be the list of packages. You can select a target package by slicing.

In [4]: packages = store.search('Unemployment Statistics')

In [5]: packages
Out[5]: 
[annual-survey-of-school-system-finances (1 resource),
 current-population-survey (1 resource),
 federal-aid-to-states (1 resource),
 dataferrett (1 resource),
 unemployment-rate (3 resources),
 local-area-unemployment-statistics (2 resources),
 mass-layoff-statistics (1 resource),
 consolidated-federal-funds-report (1 resource),
 annual-survey-of-state-government-finances (1 resource),
 foreign-labor-statistics (1 resource)]

In [6]: packages[0]
Out[6]: annual-survey-of-school-system-finances (1 resource)

Otherwise, specify the package name to be retrieved.

In [7]: package = store.get('california-unemployment-statistics')

In [8]: package
Out[8]: california-unemployment-statistics (4 resources)

A package has resources (files) which contains actual data. You use get method to retrieve the resource.

In [9]: resource = package.get('ffd05307-4528-4d15-a370-c16222119227')

In [10]: resource
Out[10]: 
Resource ID: ffd05307-4528-4d15-a370-c16222119227
Resource Name: Comma Separated Values File
Resource URL: https://data.lacity.org/api/views/5zrb-xqhf/rows.csv?accessType=DOWNLOAD
Format: CSV, Size: None

Once you get the resource, use read method to read data as pandas DataFrame.

Important

The target file must be the correct format which can be parsed by pandas IO functions.

In [11]: df = resource.read()

In [12]: df.head()
Out[12]: 
   Year Period                Area   Unemployment Rate  Labor Force  \
0  2013    Jan          California               10.4%     18556500   
1  2013    Jan  Los Angeles County               10.9%      4891500   
2  2013    Jan    Los Angeles City                 12%      1915600   
3  2013    Feb          California  9.699999999999999%     18648300   
4  2013    Feb  Los Angeles County               10.3%      4924000   

   Employment  Unemployment Adjusted Preliminary  
0    16631900       1924600  Not Adj  Not Prelim  
1     4357800        533800  Not Adj  Not Prelim  
2     1684800        230800  Not Adj  Not Prelim  
3    16835900       1812400  Not Adj  Not Prelim  
4     4418000        506000  Not Adj  Not Prelim  

Or you can get raw data by specifying raw=True.

In [13]: raw = resource.read(raw=True)

In [14]: raw[:100]
Out[14]: 'Year,Period,Area,Unemployment Rate,Labor Force,Employment,Unemployment,Adjusted,Preliminary\n2013,Jan'