RadioWinds provides methods for analyzing historical wind diversity trends from public archived sounding data and reanalysis forecasts (University of Wyoming Upper Air Sounding Archive and ECWMF Reanalysis Forecasts). The sounding and forecast data is not included with this software since it is publicly available, instead a script is included for batch downloading the soundings locally for faster and repeatable analysis. Additionally, several plotting examples are included to visualize historical trends by station monthly, annually, and decadal as well as the ability to do regional trends monthly or annually in the form of colored maps.
It is highly recommended to use WSL2 if using Windows. Several of these python libraries are untested on Windows, but have been tested to run with WSL Ubuntu 22.04 running on Windows 11. If still wanting to run directly on windows, see troubleshooting tips below.
For easy install on WSL and Ubuntu use:
pip install -r requirements.txt
For easy install on Windows use:
python3 -m pip install -r requirements-Windows.txt
Additionally, you will need to install Chrome (default) or Firefox websdriver
to use dfi_image package to generate colored probability tables
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt -y install ./google-chrome-stable_current_amd64.deb
or
sudo snap install firefox
Next install RadioWinds:
pip3 install -e .
Tested to work on Windows 11 with the following:
- Python Version 3.12
- Conda Version 23.7.4
TROUBLESHOOTING on Windows
- It's best to use a virtual conda enviroment.
conda create -n RadioWindsconda activate RadioWindspip install -r requirements.txt(You will get errors here because of cartopy and scipy-iris)conda install -c conda-forge irisconda install -c conda-forge iris-sample-dataconda install -c conda-forge cartopy
!IMPORTANT: In Siphon, for downloading individual soundings, download the latest version of
Wyoming.py from siphon Github, newer than the 0.9 release. This takes care of the occasional
height folding issue. We take care of it in SiphonMulti.py starting on line 131.
There are 3 main sources for acquiring sounding data
- https://weather.uwyo.edu/upperair/sounding.html
- https://ruc.noaa.gov/raobs/
- https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ncdc:C00975
We determined that RAOBs is missing wind data outside of the mandatory pressure levels both for ASCII and netcdf format.
It's also important to note IGRA2 and UofWy have some data discrepancies for the same soundings. Below is an explanation:
" There are three types of reports at levels below 100 hPa and for levels above 100 hPa. They are mandatory levels, significant temperature levels, and significant wind levels. The mandatory levels report pressure, temperature, humidity, and winds at defined pressure levels (1000, 925, 850, 700, ... hPa). Significant temperature levels report pressure, temperature, and humidity to fill in levels where an observation would be too different from an interpolation. Significant wind levels report heights and winds. These are merged into one report. My software, GEMPAK, interpolates the heights and winds at significant temperature levels and interpolates the height and winds at significant wind levels. IGRA reports these as missing.
In the last few years, these is a new data format called BUFR. It looks as if IGRA is using some of the data from this format. I provide this data at:
https://weather.uwyo.edu/upperair/bufrraob.shtml
I believe the US issues two different types of BUFR reports. IGRA and I may not be using the same reports.
As far as accuracy, my feeling is the inaccuracy of the measurement and variation of the atmosphere are much greater than the method used to present the data. The BUFR reports may use less smoothing that what is used to generate the old text style reports. "
FMH3 to be the defining document for radiosondes https://www.icams-portal.gov/resources/ofcm/fmh/FMH3/00-entire-FMH3.pdf
--Larry Oolman (Maintainer of Univerity of Wyoming Upper Air Sounding Dataset)
For this project we use UofWY because the dataset is easier to work with. You can download an individual sounding or up to a month by modifying the URL, whereas with IGRA2 you have to download a huge file of every sounding that station ever had and then parse through all the raw text to find the sounding you're looking for.
To run properly, and have the scripts be able to find other directories, always run from the main RadioWinds directory. For example:
python3 analysis/batchAnalysis.py Running from within the other directory might cause issues finding data sources and exporting plots and maps.
config.py The main configuration file for many of the scripts below.
utils.py Common repeated functions that are used across multiple scripts
analysis/SiphonMulti.py This script extends the Siphon Library to be able to download a month of data from UofWy rather than only one sounding at a time. This greatly speeds up the ability to bulk download soundings
analysis/AnnualWyomingDownload.py This script batch downloads soundings for an entire year locally and organizes the soundings by year/month/sounding.csv. To automate this process we use continent.csv in Radiosonde_String_Parsing/Cleaned/* for a lists of soundings to download from, and then do that for every year. We use multiprocessing to asyncronously download [multiple stations by month] simultaneously rather than [one station one month] at a time.
analysis/opposing_wind_wyoming.py This script analyzes the wind diversity for an individual sounding. It checks for 4 categories: Fail, Calm Winds, Opposings Winds, and Full Wind Diversity. if running this file standalone, it all produces windrose plots of the sounding.
analysis/batchAnalysis.py this script does opposing wind anaylsis for a year of soundings per station. Currently, we assume that the soundings were downloaded properly and in full by AnnualWyomingDownload.py. (If the script finished running everything should be downloaded for the parameters). We should add some error handling for incomplete downloads. The script then generates binary opposing wind charts organized by date and altitude level (500m increments) for each month. After all 12 months are analyzed and saved, a final annual probability chart is saved by taking the max probability from each altitude level per month.
analysis/ERA5.py This script creates a radiosonde-like dataframe for running opposing_wind_wyoming.py with ERA5 forecasts. Download a netcdf ERA5 forecast from ECWMF Copernicus CDS which includes the right timestamp, geographic region, pressure levels [300-10hPa], geopotential, u-wind, v-wind, and temperature.
To begin using RadioWinds first you need to have monthly radiosonde data locally downloaded by running AnnualWyomingDownload.py.
The main function gives an example of providing a list of station to download, from a region of interest ("continent").
The script should run as is with the current config.py variables.
The continent, start_year, end_year, and directories can be modified in config.py
Radiosonde_station_Info/CLEANED gives nice continents to chose as a starting list to download.
These continent csv's were generated by scraping the html from UofWy's Upper Sounding page and generating a csv of the station's info for that region
logging shows additional information text when downloading
parallelizing downloads each station's data for a year in it's on thread, greatly speeding up the download process.
Note: If downloading hangs, follow the instructions at the top AnnualWyomingDownload.py
Several saved data folders will appear as nan - 00000. This is because several stations don't have an FAA identifier. However the 5 digit code is an active WMO identifier.
After downloading, run checkRadiosondeDownloads.py to make sure everything downloaded properly. If everything downloaded properly, no red text should be output.
If there is error text, follow the instructions at the top of AnnualWyomingDownload.py.
opposing_wind_wyoming.py gives an example of determining wind diversity for an individual sounding and generates some nice plots.
I recommend running this first to get an idea of what batch analysis is going to do.
The configurable config parameters that this script relies on are:
- type : To analyze wind diversity via ALT or PRES (default is ALT for radiosonde, and PRES for ERA5)
- alt_step : The altitude step size for wind binning via ALT
- min_alt : the minimum altitude for wind binning via ALT
- max_alt : the maximum altitude for wind binning
- n_sectors : How many sectors to check for opposing or full winds is (Default is 16 for opposing winds and 8 for full winds, see figure below)
- speed_threshold : The minimum speed of winds to analyze, anything below this level is considered calm winds
- min_pressure : the minimum pressure for wind binning via PRES
- max_pressure : the maximum pressure for wind binning via PRES
batchAnalysis.py provides methods for batch analyzing multiple stations for wind diversity one month at a time,
and exporting the analyzed results for faster plotting by other scripts. The main function provides an example of how to analyze all the monthly and annual data
for an individual station in a given year using a list of stations for a continent. It should run as is if the sounding data for the given start_year and end_year range is downloaded.
Once the script is finished, run the script again. If all output text is green, then the monthly and annual data should be fully analyzed.
Recommended config settings:
- mode = "radiosonde" (era5 will under predict opposing winds due to low vertical resolution)
- type = "ALT" (pressure is binned into mandatory pressure levels, for ERA5 support, so the vertical resolution is limited and very spread out)
- logging = False
- parallelize = True
- monthly_export_color = False
- annual_export_color = False
Setting monthly_export_color or annual_export_color to True will generate colored dataframe tables like below. Generating colored tables, significantly increases processing time.
The station probability images will be saved in the analysis_folder within the station directory of interest
Another parameter in config.py is the mode, which can be "radiosonde" or "era5". The default is to use radiosonde data for wind diversity, because of the significantly higher vertical resolution.
But we also provide a method for performing the same analysis with ERA5 Reanalysis forecasts.
ERA5 forecasts are conveniently split into pressure levels.
If downloading a different range of pressure levels, you will need to update era5_pressure_levels in config.py.
The default era5_pressure_levels for wind diversity analysis for HAPS is between 10-300 hpa (~10-30 km).
era5_pressure_levels = np.asarray([300, 250, 225, 200, 175, 150, 125, 100, 70, 50, 30, 20, 10])
Download ERA5 Reanalyis forecasts from Copernicus. Below shows example parameters for downloading forecasts for wind diversity analysis in North America in 2024:
Note: ERA5 forecasts may need to be downloaded in batches depending on the region or time periods of interest. The Copernicus server does not support downloads over 10 gigs.
Option 1 is to do 2 different regions at a time. For instance North America, and then South America. This is what I've been doing
Option 2 is to partially download the region of interest then combine files.
Download a netcdf ERA5 forecast from https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels?tab=form which includes the right timestamp, geographic region, pressure levels [300-10hPa], geopotential, u-wind, v-wind, and temperature.
Pre-requisites: requires a sounding downloaded, or stored in the proper format.
Best way is to run AnnualWyomingDownload.py
plotting/plot3DWindrose.py: Default plots for Hilo, Hawaii (PHTO) in 2023
plot_wind_quadrants.pyskew-T.py[TODO]diurnal.py[TODO]
The following scripts generate plots and maps of monthly and annual wind diversity trends. These plots assume the proper data has been downloaded and analyzed for the desired year and continent (may be multiple years and continents)
plotting/hovmoller-full-wind-station.py
Default Run Pre-requisites: Produces an annual full wind directionality Hovmoller plot for a particular station (ex: SLC)
-
Download Radiosondes for
North_Americain2023
-
Mapping/map_stations.py: Produces a plot of all stations in the UofWy Dataset
-
Mapping/burst_map.py: Produces a map of annual burst averages for each station in the downloaded and burst analyzed regions
Default Run Pre-requisites:
-
Download Radiosondes for
North_Americain2023 -
Run
Analysis/batchAnalysis-Burst.pyforNorth_Americain2023with typeALT
-
Mapping/rainbow_probabilities.py: Produces Monthly Opposing Winds Probability Maps (ex. North_America in 2023)
Default Run Pre-requisites:
-
Download Radiosondes for
North_Americain2023 -
Run
Analysis/batchAnalysis.pyforNorth_Americaand in2023with typeALT
-
Mapping/difference_maps.py: Produces a map of OW probability differences between ERA5 and radiosondes
Default Run Pre-requisites:
-
Download Radiosondes for
North_Americain2023 -
Download ERA5 forecast for
North_Americain2023 -
Run (2)
Analysis/batchAnalysis.pyforNorth_Americain2023with typePRESfor both modeera5and moderadiosonde
-
plotting/era5scatterplot.py: Produces 2D and 3D scatter plots and overall trend lines for ERA5 vs Radiosdone OW analysis.
Default Run Pre-requisites:
- Download Radiosondes for
North_Americain2023 - Download ERA5 forecast for
North_Americain2023 - Run (2)
Analysis/batchAnalysis.pyforNorth_Americain2023with typePRESfor both modeera5and moderadiosonde
First, make sure the analysis files in step 3 have been run for the years and regions of interest before trying to run decadal analysis scripts. Some plots and maps require running an additional decadal analysis script.
Decadal Analysis scripts:
decadal_ow_analysis.pyanalyzeMonthlyMeans-Zonal-QBO.py
Decadal Plotting and Mapping Scripts:
decadal_variance_map.py
colormesh-CW.py
hovmoller-ow-probability-station.py
hovmoller_qbo.py
- While not the same, we assume geopotential height and geometric height are the same for this analysis. At stratospheric altitudes, the difference is usually 100-200m of difference.
- When binning winds, it can be good to go over the limit to include some measurements outside the desired range for more accurate results.
- For instance, when doing radiosonde-based batch analysis with a pressure of 125 hPa, anything above 125 won't be included even though 126hPa is much closer to the mandatory pressure level of 125 than the next highest level of 150. This causes lower opposing wind probabilities at the maximum and minimum altitudes.
- Pressure is not linear like altitude, so binning to the mandatory levels may not be the best method for radiosonde data. Because higher altitudes will have a lot more readings assigned to the pressure levels since they are more spaced out.
- Altitude is tough to compare between radiosonde and ERA5 forecasts because of the geopotential to height conversion. (The heights are not the same level for every location and time of year). Therefore pressure makes more sense to compare, however the pressure bins are huge altitude gaps, so that doesn't tell the full story either.
- Wind transition regions between 2 opposing wind altitudes levels have the most variability and calm winds. Many commercial balloon navigators try to avoid these regions because they're unpredictable
- The UofWy maps where we downloaded the continent stations lists from, changes from day to day, based on launches for that day. So there's a good chance our station lists are missing a few.
- Tristan Schuler - U.S. Naval Research Laboratory
- Craig Motell - NIWC Pacific
Hat Tip to Raven Aerostar, who did similar analysis in the past and presented at the 2021 Scientific Ballooning Technologies Workshop