Overview¶
pyopendata is a Python utility to offer an unified API to read world various data sources, and output pandas.DataFrame.
This is an unstable release and API is forced to be changed.
Basic Usage¶
This section explains how to retrieve data from website which uses CKAN API.You can create DataStore instance to access CKAN website by passing CKAN URL to DataStore class.
In this example, we’re going to retrieve the ‘California Unemployment Statistics’ data from data.gov. The target URL is:
We can read abov URL as:
- CKAN API URL: https://catalog.data.gov/
- Package ID: california-unemployment-statistics
- Resource ID: ffd05307-4528-4d15-a370-c16222119227
In [1]: import pyopendata as pyod
In [2]: store = pyod.DataStore('http://catalog.data.gov/')
In [3]: store
Out[3]: CKANStore (http://catalog.data.gov)
DataStore.serch performs search by keyword. Results will be the list of packages. You can select a target package by slicing.
In [4]: packages = store.search('Unemployment Statistics')
In [5]: packages
Out[5]:
[annual-survey-of-school-system-finances (1 resource),
current-population-survey (1 resource),
federal-aid-to-states (1 resource),
dataferrett (1 resource),
unemployment-rate (3 resources),
local-area-unemployment-statistics (2 resources),
mass-layoff-statistics (1 resource),
consolidated-federal-funds-report (1 resource),
annual-survey-of-state-government-finances (1 resource),
foreign-labor-statistics (1 resource)]
In [6]: packages[0]
Out[6]: annual-survey-of-school-system-finances (1 resource)
Otherwise, specify the package name to be retrieved.
In [7]: package = store.get('california-unemployment-statistics')
In [8]: package
Out[8]: california-unemployment-statistics (4 resources)
A package has resources (files) which contains actual data. You use get method to retrieve the resource.
In [9]: resource = package.get('ffd05307-4528-4d15-a370-c16222119227')
In [10]: resource
Out[10]:
Resource ID: ffd05307-4528-4d15-a370-c16222119227
Resource Name: Comma Separated Values File
Resource URL: https://data.lacity.org/api/views/5zrb-xqhf/rows.csv?accessType=DOWNLOAD
Format: CSV, Size: None
Once you get the resource, use read method to read data as pandas DataFrame.
Important
The target file must be the correct format which can be parsed by pandas IO functions.
In [11]: df = resource.read()
In [12]: df.head()
Out[12]:
Year Period Area Unemployment Rate Labor Force \
0 2013 Jan California 10.4% 18556500
1 2013 Jan Los Angeles County 10.9% 4891500
2 2013 Jan Los Angeles City 12% 1915600
3 2013 Feb California 9.699999999999999% 18648300
4 2013 Feb Los Angeles County 10.3% 4924000
Employment Unemployment Adjusted Preliminary
0 16631900 1924600 Not Adj Not Prelim
1 4357800 533800 Not Adj Not Prelim
2 1684800 230800 Not Adj Not Prelim
3 16835900 1812400 Not Adj Not Prelim
4 4418000 506000 Not Adj Not Prelim
Or you can get raw data by specifying raw=True.
In [13]: raw = resource.read(raw=True)
In [14]: raw[:100]
Out[14]: 'Year,Period,Area,Unemployment Rate,Labor Force,Employment,Unemployment,Adjusted,Preliminary\n2013,Jan'