The Monitoring, Reporting, and Verification (MRV) pipeline is described at https://github.com/jjennewein/MDA_2023-24
This is a private repository, which provides a set of scripts for the Maryland Department of Agriculture (MDA) to support verification decisions for cover crop programs, using Harmonized Landsat Sentinel (HLS) time series imagery to detect termination dates of cover crops.
The source code is approximately 2,000 lines of R, written by Jyoti Jennewein. The goal was to convert the code to Python in order to be maintainable by the PSA development team. I began documenting this process in Confluence: Converting R Code to Python.
Unfortunately, the equivalent Python code takes about twice as long to run.
Furthermore, the Python code for dealing with rasters can be much more cumbersome than that for R. Compare:
| R | Python |
|---|---|
cloud_mask <- rast(bands[list_cloud_mask])
|
with rasterio.open(list_cloud_mask) as cloud_mask_src:
cloud_mask = cloud_mask_src.read(1).astype(float)
|
stk.mask <- mask(rast.stk, cloud_mask_merge)
|
stk_mask = [np.where(~np.isnan(cloud_mask_merge), band, np.nan) for band in rast_stk]
|
Also:
range(2, 10) or 2:10) behaves differently from R (3:10),
because Python uses 0-based indexing, and the upper bound is exclusive rather than inclusive.
Given the above, my recommendation is to keep the code in R, and to use Python or Node.js to provide any automation or APIs that the program needs.
We'll use the Tidyverse style guide for syntax as much as possible.
Notable changes from the original source code:
We'll use <- instead of = for assignment.
(https://style.tidyverse.org/syntax.html#assignment-1)
This change will not appear colored in the code comparisons below.
We'll use |> instead of %>% for pipes.
(https://style.tidyverse.org/pipes.html#magrittr)
|> was added to base R version 4.1.0 and probably wasn't available when the scripts were first written.
It is faster than the magrittr pipe %>%.
This change will not appear colored in the code comparisons below.
TENTATIVE: We'll use snake_case for variable names. (https://style.tidyverse.org/syntax.html#sec-objectnames)
Many variables in the original source use a dot (.) as a separator, such as rast.stk.
In other programming languages, a dot signifies an object property or method. Using snake_case will make it easier to translate into another language if necessary.
We'll load all libraries at the top of the source. (https://style.tidyverse.org/)
This was done for the most part in the original source, with the exception of 4.Bind_WIST_KillDown.R, which loaded lubridate on line 140 and zoo on line 151.
We'll use source('globals.R') to load global variables and libraries common to the R scripts.
In the original source, directories, filenames, Julian dates, etc. were hard-coded throughout the scripts, and they were updated each year. Having them in one place means only one file needs to be updated each year.
globals.R loads these libraries:
The above libraries will not need to be loaded in scripts that load globals.R.
Also note that tidyverse loads these other libraries, which will also be available to all scripts without loading them separately:
We'll use file.path() instead of setwd() to reference files,
and all directories will use relative instead of absolute file paths.
(https://www.tidyverse.org/blog/2017/12/workflow-vs-script/#whats-wrong-with-setwd)
Previously, directories, filenames, Julian dates, etc. were hard-coded throughout the scripts.
The thirteen directories have now been reduced to five directories with relative paths, which get created if missing:
base_dir): Stores the MDA shapefilestiles_dir): Stores the tiles data and subfoldersrds_dir): Stores R-specific data files for use in subsequent scriptscsv_dir): Stores CSV files
wist_dir): Stores WIST files
Links are to the first few rows of each file.
| Files | Global Variable | Scripts |
|---|---|---|
| MDAEnrollment2023-2024.shp | input_shapefile | globals.R |
| Tiles directories | tiles_dir | 1.Extract_HLS |
| WISE.Field_Imagery_Maryland_S30L30_HLS_v2.0_2023-24_through_201.csv | wise_file | 4.Bind_WIST_KillDown.R and File 5.Phenology_plots.R |
| WIST_first.Field_Imagery_Maryland_S30L30_HLS_v2.0_2023-24_through_201.csv | wist_file | 4.Bind_WIST_KillDown.R and File 5.Phenology_plots.R |
| FY24 SQL Spring Killdown Report 7.19.2024.csv | killdown_file | 4.Bind_WIST_KillDown.R |
| ND_report_WDH.csv | nd_report_file | 4.Bind_WIST_KillDown.R |
| ND_report_WDH_long.csv (??? missing "this is a pivot of the duplicated one and not a unique file") | 4.Bind_WIST_KillDown.R | |
| JPEG files (??? missing) | 4.Bind_WIST_KillDown.R |
Links are to the first few rows of each file.
| Files | Global Variable | Scripts |
|---|---|---|
| MDAEnrollment2023-2024_Buff30m.shp | shapefile | All scripts |
| L30.NDVI.output_MD_UMT17_UTM18_MD_2023-2024_through200.csv | l30_file | 2.Bind_HLS.R (output) and 3.Bind_L30_S30_for_WIST.R (input) |
| S30.NDVI.output_MD_UMT17_UTM18_MD_2023-2024_through201.csv | s30_file | 2.Bind_HLS.R (output) and 3.Bind_L30_S30_for_WIST.R (input) |
| Field_Imagery_Maryland_S30L30_HLS_v2.0_2023-2024_through201.csv | l30_s30_file | 3.Bind_L30_S30_for_WIST.R (output) and 5.Phenology_plots.R (input) |
| FY24 SQL Spring Killdown Report 7.19.2024. Duplicates.csv | killdown_duplicates_file | 4.Bind_WIST_KillDown.R |
| FY24 SQL Spring Killdown Report 7.19.2024. No Grower Reported Termination.csv | killdown_nogrower_file | 4.Bind_WIST_KillDown.R |
| FY24 SQL Spring Killdown Report 7.19.2024. HLS Termination.csv | killdown_hls_termination_file | 4.Bind_WIST_KillDown.R (output) and 5.Phenology_plots.R (input) |
| ND_report_updated.csv | nd_report_updated | 4.Bind_WIST_KillDown.R (output) and 5.Phenology_plots.R (input) |
| Wait for more imagery | 4.Bind_WIST_KillDown.R | |
| JPEG Plots | 5.Phenology_plots.R |
Matching code
| globals.R |
|---|
| Old: 1.Extract_HLS_L30 .R and 1.Extract_HLS_S30 .R | New: 1.Extract_HLS.R |
|---|
| Old: 2.Bind_HLS_L30.R and 2.Bind_HLS_S30.R | New: 2.Bind_HLS.R |
|---|
| Old: 3.Bind_L30_S30_for_WIST.R | New: 3.Bind_L30_S30_for_WIST.R |
|---|
| Old: 4.Bind_WIST_KillDown.R | New: 4.Bind_WIST_KillDown.R |
|---|
| Old: 5.Phenology_plots.R | New: 5.Phenology_plots.R |
|---|