Updating the GEDI subsetter for filtering data

[ParomitaBasak]

We’ve recently identified an issue with the GEDI subsetter related to erroneous data that wasn’t properly filtered by the automatic filters. This mainly involves orbits that likely encountered low-lying clouds, smoke, or fog, resulting in abnormally tall tree height and biomass readings. While the L4B product has been corrected, the issue persists in other GEDI products like L4A and L2A, which are commonly used in the MAAP platform.

Proposed Solution: We need to update the GEDI subsetter to either:

1. Apply the necessary fix automatically.
2. Enable users to apply this fix manually (the former is preferred).

This update will help ensure data accuracy across the GEDI products.

[chuckwondo]

@ParomitaBasak, please add full details of what you mean by "fix," otherwise we have no information here to allow somebody to determine what should be done, or even whether or how it might be done in the GEDI Subsetter.

[ParomitaBasak]

@chuckwondo could you please check the slack channel for more update on the issue! Thanks a lot in advance!

[chuckwondo]

@chuckwondo could you please check the slack channel for more update on the issue! Thanks a lot in advance!

The point of having you create the issue here is to gather all relevant details here rather than on Slack. Since I am unfamiliar with the issue, please transfer all relevant details to the issue here.

[ParomitaBasak]

@chuckwondo, Please find the following link for more relevant details on the filters that can be used to filter out the erroneous data. The MAAP GEDI subsetters need to integrate these filters so that the corrupted data can be removed while we are getting the subset.

https://docs.google.com/presentation/d/1Z05XiaZGEX0cOjGj-mPQoBiMEaHn-bUUa-aIGAnoxiE/edit#slide=id.g2b4a8dc9a1f_0_5

[chuckwondo]

Based upon the attached slides, I have a number of questions, as it is not clear to me what exactly needs to be done, so we'll need to schedule a meeting to get clarification. Since Alex and I are attending FOSS4G NA next week, this will likely have to be scheduled during the week of Sep. 16.

[chuckwondo]

Just to follow up, we (Laura, John, Paromita, Alex, and myself) held a call on Fri, Sep 27 to discuss how to approach this.

We agreed that since the existing collections are slated to be corrected within the next few months, we will not modify the GEDI Subsetter to deal with the filtering. Specifically, the filtering will be applied directly to the files, so there will be no need to manually apply filtering.

Instead, until the files are corrected, we will perform the filtering via the query input to the GEDI Subsetter. To assist, @chuckwondo will create a new branch (quality-filters) in the GEDI Subsetter repo and add a notebook with example query expressions reflecting what is given in the GEDI Quality Filtering slides.

Tangentially, Laura asked about making use of the new L4C collection. Paromita has confirmed that the GEDI Subsetter can make use of the new collection by simply setting the doi input value to "10.3334/ORNLDAAC/2338". As a convenience, @chuckwondo will update the GEDI Subsetter to allow users to specify "L4C" as the value of the doi input, instead of the full DOI.

[chuckwondo]

@ParomitaBasak, version 0.9.0 is now available, and it allows you to specify "L4C" for the doi, so you don't have to explicitly specify the official DOI.

[chuckwondo]

Yesterday (Oct. 21), Alex and I chatted further with John in Slack to get further clarity on how to proceed with computing the filters.

As a result, John clarified that we effectively want to compute a quality flag for L4A based on various data in both L2A and L4A. Based upon the slides linked in a prior comment, this will be a moderately heavy lift due to the GEDI Subsetter not being designed for cross-collection subsetting. Although there is an open issue to add support for this, the subsetter currently does not support it, supporting only subsetting one particular collection per job.

After consulting with Laura on the urgency of this issue, she indicated that Paromita has come up with a downstream workaround that should suffice, and that since this has turned out to not be a quick and easy lift, we can forego this work and focus on other things instead.

This has, however, provided good food for thought on how we might provide a convenient means for adding cross-collection subsetting capabilities to the subsetter (to address: MAAP-Project/gedi-subsetter#11).