Chapter 9 Generating file lists by plot

9.1 Pre-requisites:

  • if you have not already done so, you will need to 1) sign up for the beta user program and 2) sign up and be approved for access to the the sensor data portal in order to get the API key that will be used in this tutorial.

The terrautils python package has a new products module that aids in connecting plot boundaries stored within betydb with the file-based data products available from Globus.

  • if are using Rstudio and want to run the Python code chunks, the R package “reticulate” is required
  • use pip3 install terrautils to install the terrautils Python library

9.2 Getting started

After installing terrautils, you should be able to import the products module.

The get_sensor_list and get_file_listing functions both require the connection, url, and key parameters. The connection can be ‘None’. The url (called host in the code) should be something like https://terraref.org/clowder/. The key is a unique access key for the Clowder API.

9.3 Getting the sensor list

The first thing to get is the sensor name. This can be retrieved using the get_sensor_list function. This function returns the full record which may be useful in some cases but primarily includes sensor names that include a plot id number. The utility function unique_sensor_names accepts the sensor list and provides a list of names suitable for use in the get_file_listing function.

To use this tutorial you will need to sign up for Clowder, have your account approved, and then get an API key from the Clowder web interface.

Names will now contain a list of sensor names available in the Clowder geostreams API. The list of returned sensor names could be something like the following:

  • flirIrCamera Datasets
  • IR Surface Temperature
  • RGB GeoTIFFs Datasets
  • stereoTop Datasets
  • scanner3DTop Datasets
  • Thermal IR GeoTIFFs Datasets
  • …

9.4 Getting a list of files

The geostreams API can be used to get a list of datasets that overlap a specific plot boundary and, optionally, limited by a time range. Iterating over the datasets allows the paths to all the files to be extracted.

Datasets can be further filtered using the since and until parameters of get_file_listing with a date string.

9.5 Querying the API

The source files behind the data are available for downloading through the API. By executing a series of requests against the API it’s possible to determine the files of interest and then download them.

Each of the API URL’s have the same beginning (https://terraref.org/clowder/api), followed by the data needed for a specific request. As we step through the process you will be able to see how then end of the URL changes depending upon the request.

Below is what the API looks like as a URL. Try pasting it into your browser.

https://terraref.org/clowder/api/geostreams/sensors?sensor_name=MAC Field Scanner Season 1 Field Plot 101 W

This will return data for the requested plot including its id. This id (or identifier) can then be used for additional queries against the API.

In the examples below we will be using curl on the command line to make our API calls. Since the API is accessed through URLs, it’s possible to use the URLs in software programs or with a programming language to retrieve its data.

9.5.1 A Word of Caution

We are no longer using the python terrautils package, which is a python library that provides helper functions that simplify interactions with the Clowder API. One of the ways it makes the interface easier is by using function names that make sense in the scope of the project. The API and the Clowder database have different names and this is confusing since the same names are used for different parts of the database.

The names and meanings of variables in this section don’t necessarily match the ones in the section above and it may be easy to get them confused. The API queries the database directly and thereby reflects the database structure. This is the main reason for the naming differences between the API and the terraref client.

For example, the Clowder API’s use of the term SENSOR_NAME is equivalent to site_name above.

9.5.2 Finding plot ID

We can query the API to find the identifier associated with the name of a plot. For this example we use the variable name of SENSOR_DATA to indicate the name of the plot.

This creates a file named plot.json containing the JSON object returned by the API. The JSON object has an ‘id’ parameter. This ID parameter can be used to specify the correct data stream.

9.5.3 Finding stream ID within a plot

Using the sensor ID returned in the JSON from the previous call and the id of a sensor returned previously to get the stream id. The names of streams are are formatted as “ Datasets ()”.

A file named stream.json will be created containing the returned JSON object. This JSON object has an ‘id’ parameter that contains the stream ID. You can use this ID parameter to get the datasets, and then datapoints, of interest.

9.5.4 Listing Clowder dataset IDs for that plot & sensor stream

We now have a stream ID that we can use to list our datasets. The datasets in turn contain files of interest.

After the call succeeds, a file named datasets.json is created containing the returned JSON object. As part of the JSON object there are one or more properties fields containing source_dataset parameters.

The URL of each source_dataset can be used to view the dataset in Clowder.

The datasets can also be filtered by date. The following filters out datasets that are outside of the range of January 2, 2017 through June 20, 2017.

9.5.6 Retrieving the files

Given that a large number of files may be contained in a dataset, it may be desirable to automate the process of pulling down files to the local system.

For each file to be retrieved, the unique file ID is needed on the URL.

This call will cause the server to return the contents of the file identified in the URL. This file is then stored locally in *ir_geotiff_L1_ua-mac_2016-05-09__12-07-57-990.tif*.