The Australian Government coat of Arms

Communities of practice

Communities of practice

A question and a bit of example Python code for accessing the ABS API

A few questions for the ABS

Firstly big support of the ABS API…data pipelines based on csv and xls files are too costly and no longer efficient enough for modern data processing practices.

  1. I note that the the 2016 GCP is not fully loaded only tables 43 to 59. Understandable given its Beta status. Is there a schudule for loading ABS data into the API? In particular, will the 2016 GCOP be fully laoded and is the a date for the loading the 2021 GCP tables?

  2. What date will the API drop it beta status? There must be at least an indicative date…perhaps with the first release of 2021 Census data?

Also thought it would be useful to share some simple code. Might help to get people going :wink:

I needed to extract the GCP table name for a selection of dataflows to see if a ML algorithm can classify them.

########################################################################################################################

Python 3.7 Script

Purpose: Simple example of connecting to the ABS API and receiving a result. Parse the result as an XML element tree

and print out derived information.

Dependencies: Access key for the ABS API module

Reference

ABS Data API (Beta) | api.gov.au

Author: Marcus Blake

Date: Feb 2021

########################################################################################################################

imports

import requests
import xml.etree.ElementTree as ET

2016 Census General Community Profile Tables

Note as of Feb 2021 the API does not have the full set of GCP tables loaded to the API

Only Table G43 to G59 exist

List of SA1 level GCP Tables

DataFlowList = [‘ABS_C16_G43_SA’,
‘ABS_C16_G44_SA’,
‘ABS_C16_G45_SA’,
‘ABS_C16_G46_SA’,
‘ABS_C16_G47_SA’,
‘ABS_C16_G48_SA’,
‘ABS_C16_G49_SA’,
‘ABS_C16_G50_SA’,
‘ABS_C16_G51_SA’,
‘ABS_C16_G52_SA’,
‘ABS_C16_G53_SA’,
‘ABS_C16_G56_SA’,
‘ABS_C16_G57_SA’,
‘ABS_C16_G58_SA’,
‘ABS_C16_G59_SA’]

for t in DataFlowList:
# Using guest account request the response for a dataflow
response = requests.get(“https://api.data.abs.gov.au/datastructure/ABS/” + t)

# This is where you should do some checking of the returned status code
# print(response.status_code)

# Parse the response text into a element tree using the fromstring function
e = ET.fromstring(response.text)

# Print the text within the Name element
# Note the whole SDMX http reference to the element needs to be included in the element reference for  the etree element tree to recognise the element
for n in e.iter('{http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common}Name'):
    print(n.text)
1 Like

Hi Marcus,

Great post! Thanks for sharing some example code and for the support of ABS APIs.
To answer your questions:

  • The Community Profile tables (G01 - G42) are coded T01 to T28 in the API. But there is really no strict mapping of the table numbers on the Census page to that in the API. Also not all the Tables that you see there are available in API. It is best to look at the dataflow names to know what Census data is contained in each dataflow. At this stage, there are no plan to release any more Census 2016 data in the API other than what’s currently in there.

  • We’re starting to look at what data to publish in the API for the 2021 Census and how to structure this data. If you have any feedback on this please send it through to api.data@abs.gov.au. We’re aiming to publish 2021 Census data in the API from mid 2022, aligning as closely as possible with the 2021 Census data release timeline.

  • We aim to take the beta tag off the API in 2021. The ABS Data API will definitely be Live before 2021 Census data is published.