RasterDataSources.jl

RasterDataSources downloads raster data for local use or for integration into other spatial data packages, like Rasters.jl. The collection is largely focussed on datasets relevant to ecology, but will have a lot of crossover with other sciences.

Currently sources include:

Please open an issue if you need more datasets added, or (even better) open a pull request following the form of the other datasets where possible.

Retrieving data

Usage is generally via the getraster method - which will download the raster data source if it isn't available locally, or simply return the path/s of the raster file/s:

julia> using RasterDataSources

julia> getraster(WorldClim{Climate}, :wind; month=1:12)
12-element Array{String,1}:
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_01.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_02.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_03.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_04.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_05.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_06.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_07.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_08.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_09.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_10.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_11.tif"
 "/home/user/Data/WorldClim/Climate/wind/wc2.1_10m_wind_12.tif"

Installation and setup

Install as usual with:

] add RasterDataSources

To download data you will need to specify a folder to put it in. You can do this by assigning the environment variable RASTERDATASOURCES_PATH:

ENV["RASTERDATASOURCES_PATH"] = "/home/user/Data/"

This can be put in your startup.jl file or the system environment.

RasterDataSources was based on code from the SimpleSDMDataSoures.jl package by Timothée Poisot.

getraster(T::Type, layers::Union{Tuple,Int,Symbol}; kw...)

Download raster layers layers from the data source T, returning a String for a single layer, or a NamedTuple for a Tuple of layers. layer values are usually values of Symbol, but can also be Int for BioClim datasets.

Keyword arguments depend on the specific data source. They may modify the return value, following a pattern:

• month keywords of AbstractArray will return a Vector{String} or Vector{<:NamedTuple}.
• date keywords of AbstractArray will return a Vector{String} or Vector{<:NamedTuple}.
• date keywords of Tuple{start,end} will take all the dates between the start and end dates, and also return Vector{String} or Vector{<:NamedTuple}.

Where date and month keywords coexist, Vector{Vector{String}} of Vector{Vector{NamedTuple}} is the result. date ranges are always the outer Vector, month the inner Vector with layer tuples as the inner NamedTuple. No other keywords can be Vector.

This schema may be added to in future for datasets with additional axes, but should not change for the existing RasterDataSource types.

RasterDataSource

Abstract supertype for raster data collections.

ALWB{Union{Deciles,Values},Union{Day,Month,Year}} <: RasterDataSource

Data from the Australian Landscape Water Balance (ALWB) data source.

See: www.bom.gov.au/water/landscape

The dataset contains NetCDF files. They have a time dimension so that multiple dates are stored in each file.

The available layers are: (:rain_day, :s0_pct, :ss_pct, :sd_pct, :sm_pct, :qtot, :etot, :e0, :ma_wet, :pen_pet, :fao_pet, :asce_pet, :msl_wet, :dd), available in daily, monthly and annual resolutions, and as Values or relative Deciles.

getraster for ALWB must use a date keyword to specify the date to download.

See the getraster docs for implementation details.

getraster(source::Type{<:ALWB{Union{Deciles,Values},Union{Day,Month,Year}}}, [layer]; date)

Download ALWB weather data from www.bom.gov.au/water/landscape as values or deciles with timesteps of Day, Month or Year.

Arguments

• layer: Symbol or Tuple of Symbol from (:rain_day, :s0_pct, :ss_pct, :sd_pct, :sm_pct, :qtot, :etot, :e0, :ma_wet, :pen_pet, :fao_pet, :asce_pet, :msl_wet, :dd). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• date: a DateTime, AbstractVector of DateTime or a Tuple of start and end dates. For multiple dates, a Vector of multiple filenames will be returned. ALWB is available with a daily, monthly, and yearly, timestep.

Example

This will return the file containing annual averages, including your date:

julia> getraster(ALWB{Values,Year}, :ss_pct; date=Date(2001, 2))
"/your/RASTERDATASOURCES_PATH/ALWB/values/month/ss_pct.nc"

Returns the filepath/s of the downloaded or pre-existing files.

AWAP <: RasterDataSource

Daily weather data from the Australian Water Availability Project, developed by CSIRO.

See: www.csiro.au/awap

The available layers are: (:solar, :rainfall, :vprpress09, :vprpress15, :tmin, :tmax).

getraster(source::Type{AWAP}, [layer]; date)

Download data from the AWAP weather dataset, from www.csiro.au/awap.

The AWAP dataset contains ASCII .grid files.

Arguments

• layer Symbol or Tuple of Symbol for layers in (:solar, :rainfall, :vprpress09, :vprpress15, :tmin, :tmax). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• date: a DateTime, AbstractVector of DateTime or a Tuple of start and end dates. For multiple dates, A Vector of multiple filenames will be returned. AWAP is available with a daily timestep.

Example

Download rainfall for the first month of 2001:

julia> getraster(AWAP, :rainfall; date=Date(2001, 1, 1):Day(1):Date(2001, 1, 31))

31-element Vector{String}:
 "/your/path/AWAP/rainfall/totals/20010101.grid"
 "/your/path/AWAP/rainfall/totals/20010102.grid"
 ...
 "/your/path/AWAP/rainfall/totals/20010131.grid"

Returns the filepath/s of the downloaded or pre-existing files.

CHELSA{Union{BioClim,<:Future}} <: RasterDataSource

Data from CHELSA, currently implements the current BioClim and Future{BioClim} variables, and Future{Climate}.

See: chelsa-climate.org for the dataset, and the getraster docs for implementation details.

getraster(source::Type{CHELSA{BioClim}}, [layer]) => Union{Tuple,String}

Download CHELSA BioClim data from chelsa-climate.org.

Arguments

• layer: Integer or tuple/range of Integer from (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19), or Symbols form (:bio1, :bio2, :bio3, :bio4, :bio5, :bio6, :bio7, :bio8, :bio9, :bio10, :bio11, :bio12, :bio13, :bio14, :bio15, :bio16, :bio17, :bio18, :bio19). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Returns the filepath/s of the downloaded or pre-existing files.

getraster(T::Type{CHELSA{Future{Climate}}}, [layer]; date, month) => String

Download CHELSA Climate data, choosing layers from: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19).

See the docs for Future for model choices.

Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• date: a Date or DateTime object, a Vector, or Tuple of start/end dates. Note that CHELSA CMIP5 only has two datasets, for the periods 2041-2060 and 2061-2080. Dates must fall within these ranges.
• month: the month of the year, from 1 to 12, or a array or range of months like 1:12.

EarthEnv{Union{HabitatHeterogeneity,LandCover}} <: RasterDataSource

Data from the EarthEnv including HabitatHeterogeneity and LandCover

See: www.earthenv.org

getraster(source::Type{EarthEnv{HabitatHeterogeneity}}, [layer]; res="25km")

Download EarthEnv habitat heterogeneity data.

Arguments

• layer: Symbol or Tuple of Symbol from (:cv, :evenness, :range, :shannon, :simpson, :std, :Contrast, :Correlation, :Dissimilarity, :Entropy, :Homogeneity, :Maximum, :Uniformity, :Variance). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• res: String chosen from ("1km", "5km", "25km"), defaulting to "25km".

Returns the filepath/s of the downloaded or pre-existing files.

getraster(T::Type{EarthEnv{LandCover}}, [layer]; discover=false) => Union{Tuple,String}

Download EarthEnv landcover data.

Arguments

• layer: Integer or tuple/range of Integer from (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12), or Symbols from (:needleleaf_trees, :evergreen_broadleaf_trees, :deciduous_broadleaf_trees, :other_trees, :shrubs, :herbaceous, :cultivated_and_managed, :regularly_flooded, :urban_builtup, :snow_ice, :barren, :open_water). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• discover::Bool: whether to download the dataset that integrates the DISCover model.

Returns the filepath/s of the downloaded or pre-existing files.

WorldClim{Union{BioClim,Climate,Weather}} <: RasterDataSource

Data from WorldClim datasets, either BioClim, Climate or Weather.

See: www.worldclim.org

getraster(T::Type{WorldClim{BioClim}}, [layer::Union{Tuple,AbstractVector,Integer}]; res::String="10m") => Union{Tuple,AbstractVector,String}

Download WorldClim BioClim data.

Arguments

• layer: Integer or tuple/range of Integer from (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19). or Symbols from (:bio1, :bio2, :bio3, :bio4, :bio5, :bio6, :bio7, :bio8, :bio9, :bio10, :bio11, :bio12, :bio13, :bio14, :bio15, :bio16, :bio17, :bio18, :bio19). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• res: String chosen from ("30s", "2.5m", "5m", "10m"), "10m" by default.

Returns the filepath/s of the downloaded or pre-existing files.

getraster(T::Type{WorldClim{Weather}}, [layer::Union{Tuple,Symbol}]; date) => Union{String,Tuple{String},Vector{String}}

Download WorldClim Weather data, for layer/s in: (:tmin, :tmax, :prec). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• date: a Date or DateTime object, a Vector of dates, or Tuple of start/end dates. WorldClim Weather is available with a daily timestep.

Returns the filepath/s of the downloaded or pre-existing files.

getraster(T::Type{WorldClim{Climate}}, [layer::Union{Tuple,Symbol}]; month, res::String="10m") => Vector{String}

Download WorldClim Climate data.

Arguments

• layer Symbol or Tuple of Symbol from (:tmin, :tmax, :tavg, :prec, :srad, :wind, :vapr). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• month: Integer or AbstractArray of Integer. Chosen from 1:12.
• res: String chosen from ("30s", "2.5m", "5m", "10m"), "10m" by default.

Returns the filepath/s of the downloaded or pre-existing files.

getraster(T::Type{WorldClim{Elevation}}, [layer::Union{Tuple,Symbol}]; res::String="10m") => Union{Tuple,AbstractVector,String}

Download WorldClim Elevation data.

Arguments

• layer: Symbol or Tuple of Symbol from (:elev,). Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Keywords

• res: String chosen from ("30s", "2.5m", "5m", "10m"), "10m" by default.

Returns the filepath/s of the downloaded or pre-existing files.

MODIS{ModisProduct} <: RasterDataSource

MODIS/VIIRS Land Product Database. Vegetation indices, surface reflectance, and more land cover data. Data from ModisProducts datasets.

See: modis.ornl.gov

getraster(T::Union{Type{<:ModisProduct}, Type{MODIS{X}}}, [layer::Union{Tuple,AbstractVector,Integer, Symbol}]; kwargs...) => Union{String, AbstractVector, NamedTuple}

Download MODIS data for a given ModisProduct as ASCII raster(s).

Arguments

• layer: Integer or tuple/range of Integer or Symbols. Without a layer argument, all layers will be downloaded, and a NamedTuple of paths returned.

Available layers for a given product can be looked up using RasterDataSources.layerkeys(T::Type{<:ModisProduct}).

Keywords

• lat and lon: Coordinates in decimal degrees of the approximate center of the raster. The MODIS API will try to match its pixel grid system as close as possible to those coordinates.

• km_ab and km_lr: Half-width and half-height of the raster in kilometers (kilometers above/below and left/right). Currently only Integer values are supported, up to 100.

• date: String, Date, DateTime, AbstractVector of dates or Tuple of a start and end date for the request. Strings should be in format YYYY-MM-DD but can be in similar formats as long as they are comprehensible by Dates.Date. The available date interval for MODIS is 16 days, reset every first of January.

Example

Download 250m NDVI in the western part of Britanny, France, from winter to late spring, 2002:

julia> getraster(MOD13Q1, :NDVI; lat = 48.25, lon = -4, km_ab = 50, km_lr = 50, date = (Date(2002,1,1), Date(2002,6,1)))

    10-element Vector{String}:
    "/your/path/MODIS/MOD13Q1/250m_16_days_NDVI/47.8313_-4.5899_2002-01-01.asc"
    ...
    "/your/path/MODIS/MOD13Q1/250m_16_days_NDVI/47.8313_-4.5899_2002-05-25.asc"

Will attempt to download several files, one for each date and layer combination, and returns the filepath/s of the downloaded or pre-existing files. Coordinates in the file names correspond to the lower-left corner of the raster.

ModisProduct <: RasterDataSet

Abstract supertype for MODIS/VIIRS products.

Usage

Some commonly used products are MOD13Q1 (250m resolution MODIS vegetation indices) and VNP13A1 (500m resolution VIIRS vegetation indices). Refer to the MODIS documentation for detailed product information.

layerkeys(T::Type{<:ModisProduct}) => Tuple

Tuple of Symbols corresponding to the available layers for a given product. May issue a request to MODIS server to get the layers list, or might just read this information if the correctly named file is available.

RasterDataSet

Abstract supertype for datasets that belong to a RasterDataSource.

BioClim <: RasterDataSet

BioClim datasets. Usually containing layers from 1:19. These can also be accessed with :bioX, e.g. :bio5.

They do not usually use month or date keywords, but may use date in past/future scenarios.

Currently implemented for WorldClim and CHELSA as WorldClim{BioClim}, CHELSA{BioClim} and CHELSA{Future{BioClim, args..}}.

See the getraster docs for implementation details.

Climate <: RasterDataSet

Climate datasets. These are usually months of the year, not specific dates, and use a month keyword in getraster. They also use date in past/future scenarios.

Currently implemented for WorldClim and CHELSA as WorldClim{Climate}, CHELSA{Climate} and CHELSA{Future{Climate, args..}}.

See the getraster docs for implementation details.

Weather <: RasterDataSet

Weather datasets. These are usually large time-series of specific dates, and use a date keyword in getraster.

Currently implemented for WorldClim and CHELSA as WorldClim{Weather}, and CHELSA{Weather}

See the getraster docs for implementation details.

Elevation <: RasterDataSet

Elevation datasets.

Currently implemented for WorldClim as WorldClim{Elevation}.

See the getraster docs for implementation details.

LandCover <: RasterDataSet

Land-cover datasets.

Currently implemented for EarthEnv as EarchEnv{LandCover}.

See the getraster docs for implementation details.

HabitatHeterogeneity <: RasterDataSet

Habitat heterogeneity datasets.

Currently implemented for EarchEnv as EarchEnv{HabitatHeterogeneity}.

See the getraster docs for implementation details.

Future{<:RasterDataSet,<:CMIPphase,<:ClimateModel,<:ClimateScenario}

Future climate datasets specified with a dataset, phase, model, and scenario.

Type Parameters

RasterDataSet

Currently BioClim and Climate are implemented for the CHELSA data source.

CMIPphase

Can be either CMIP5 or CMIP6.

ClimateModel

Climate models can be chosen from:

ACCESS1, BNUESM, CCSM4, CESM1BGC, CESM1CAM5, CMCCCMS, CMCCCM, CNRMCM5, CSIROMk3, CanESM2, FGOALS, FIOESM, GFDLCM3, GFDLESM2G, GFDLESM2M, GISSE2HCC, GISSE2H, GISSE2RCC, GISSE2R, HadGEM2AO, HadGEM2CC, IPSLCM5ALR, IPSLCM5AMR, MIROCESMCHEM, MIROCESM, MIROC5, MPIESMLR, MPIESMMR, MRICGCM3, MRIESM1, NorESM1M, BCCCSM1, Inmcm4, BCCCSM2MR, CNRMCM61, CNRMESM21, CanESM5, MIROCES2L, MIROC6 for CMIP5;

UKESM, MPIESMHR IPSLCM6ALR MRIESM2, GFDLESM4 for CMIP6.

ClimateScenario

CMIP5 Climate scenarios are all RepresentativeConcentrationPathway and can be chosen from: RCP26, RCP45, RCP60, RCP85

CMIP6 Climate scenarios are all SharedSocioeconomicPathway and can be chosen from: SSP126, SSP245, SSP370, SSP585

Example

using RasterDataSources
dataset = Future{BioClim, CMIP5, BNUESM, RCP45}
# output
Future{BioClim, CMIP5, BNUESM, RCP45}

Currently Future is only implented for CHELSA

datasource = CHELSA{Future{BioClim, CMIP5, BNUESM, RCP45}}

ClimateModel

Abstract supertype for climate models use in Future datasets.

CMIPphase

Abstract supertype for phases of the CMIP, the Coupled Model Intercomparison Project.

Subtypes are CMIP5 and CMIP6.

CMIP5 <: CMIPphase

The Coupled Model Intercomparison Project, Phase 5.

CMIP6 <: CMIPphase

The Coupled Model Intercomparison Project, Phase 6.

ClimateScenario

Abstract supertype for scenarios used in CMIPphase models.

RepresentativeConcentrationPathway

Abstract supertype for Representative Concentration Pathways (RCPs) for CMIP5.

Subtypes are: RCP26, RCP45, RCP60, RCP85

SharedSocioeconomicPathway

Abstract supertype for Shared Socio-economic Pathways (SSPs) for CMIP6.

Subtypes are: SSP126, SSP245, SSP370, SSP585

Values <: DataMode

Get the dataset as regular measured values.

Deciles <: DataMode

Get the dataset in relative deciles.

layerkeys(T::Type{<:ModisProduct}) => Tuple

Tuple of Symbols corresponding to the available layers for a given product. May issue a request to MODIS server to get the layers list, or might just read this information if the correctly named file is available.

rastername(source::Type, [layer]; kw...)

Returns the name of the file, without downloading it.

Arguments are the same as for getraster

Returns a String or multiple Strings.

rasterpath(source::Type, [layer]; kw...)

Returns the name of the file, without downloading it.

Arguments are the same as for getraster

Returns a String or multiple Strings.

rasterurl(source::Type, [layer]; kw...)

If the file has a single url, returns it without downloading.

Arguments are the same as for getraster.

Returns a URIs.jl URI or mulitiple URIs.

zipname(source::Type, [layer]; kw...)

If the url is a zipped file, returns its name.

Arguments are as the same for getraster where possible.

Returns a String or multiple Strings.

zippath(source::Type, [layer]; kw...)

If the url is a zipped file, returns its path when downloaded. (This may not exist after extraction with getraster)

Arguments are the same as for getraster where possible.

Returns a String or multiple Strings.

zipurl(source::Type, [layer]; kw...)

If the url is a zipped file, returns its zip path without downloading.

Arguments are the same as for getraster where possible.

Returns a URIs.jl URI or mulitiple URIs.

modis_request(T::Type{<:ModisProduct}, args...)

Lowest level function for requests to modis server. All arguments are assumed correct.

Arguments

• layer: String matching the "exact" layer name (i.e. as it is written in the MODIS dataset itself) for the given product. e.g. "250m_16_days_EVI".

• lat, lon, km_ab, km_lr in correct types

• from, to: Strings of astronomical dates for start and end dates of downloaded data, e.g. "A2002033" for "2002-02-02"

Returns a NamedTuple of information relevant to build a raster header, and a Vector of Dicts containing raster data, directly downloaded from MODIS. Those will almost always directly be passed to RasterDataSources.process_subset

process_subset(T::Type{<:ModisProduct}, subset::Vector{Any}, pars::NamedTuple)

Process a raw subset and argument parameters and create several raster files. Any already existing file is not overwritten.

For each band, a separate folder is created, containing a file for each of the required dates. This is inspired by the way WorldClim{Climate} treats the problem of possibly having to download several dates AND bands.

Can theoretically be used for MODIS data that does not directly come from RasterDataSources.modis_request, but caution is advised.

Returns the filepath/s of the created or pre-existing files.

product(T::Type{<:ModisProduct})

Extracts ModisProduct product name as a String

sinusoidal_to_latlon(x::Real, y::Real)

Convert x and y in sinusoidal projection to lat and lon in dec. degrees

The takes care of coordinate conversions. This is not ideal in terms of network use but guarantees that the coordinates are correct.