Weather data reader

[thorade]

As a result of
ibpsa/modelica-ibpsa#82 (comment)
we do now have two Weather Data Readers.

The version from Buildings has been added to Annex60 and is therefore now included in BuildingSystems. This Reader reads TMY3 files, converts them to CombiTimeTable format and uses a Bus to distribute the info to components.

BuildingSystems has the netCDF reader from @jraedler , but an old version.
There exists a newer version by @jraedler
https://bitbucket.org/jraedler/ncdatareader2
and there also exists a fork by @tbeu that has been restructured a lot.
https://github.com/tbeu/netCDF-DataReader

The newer netCDF reader can also read HDF5, I believe.
http://www.hdfgroup.org/projects/netCDF-4/

Should the two readers be combined? What is the better weather format, TMY3 or HDF5? Can we support both, if possible with the identical interface to the end user?

[jraedler]

@tbeu made a lot of changes and a new release of ncDataReader2 that makes it much easier to
use with Modelica (including Windows DLLs):

https://github.com/tbeu/netCDF-DataReader

If you want to keep it as the file reader solution in the BuildingSystems library, one of us has to
test the new package and adapt the library. I can support this, but it will need some time.

[thorade]

There also exists one more HDF5 (or a special sub-dialect?) reading library:
https://bitbucket.org/3DS/simdevtools
https://bitbucket.org/dietmarw/simdevtools
http://simdevtools.org/

[tbeu]

Why should it be not possible to simply depend on a Modelica library that already can handle external files? We have several platform-generic, tool-independent, license-friendly FLOSS libraries available, that are also well maintained.

• ModelicaStandardTables (of the Modelida Standard Library) that can read MAT and TXT files (which you for some reason you call MOS file containing weather data)
• netCDF-DataReader that can read binary netDCF files (based on HDF5). Since @jraedler is the copyright holder the LGPL should be fine.
• ExternData that can read XML, JSON, MAT, Excel sheets and INI files (and also write MAT files). Read support of CSV files and also read support of arrays from Excel sheets is planned for v2.0.0. Other file formats (like EPW which is similar to CSV) can easily be added upon (pull) request.

Otherwise it's rather strange to me that every Building library implements its own weatherdata reader for that specific part.

[thorade]

The libraries started out separately and every library implemented everything themselves. The idea of Annex60 is exactly to reduce this duplication. Every part that is needed by all libraries could be moved from the extending libraries to the Annex60 core library. That is what happened with the weather data reader: It was copied from the Buildings library to the Annex60 library, and then during the next integration, it was copied from Annex60 to all extending libraries. This is why we now have two weather data readers, and this is why I filed this issue.
Technically, it is absolutely possible to use one single data reader and I believe we should develop one single good reader for all libraries. That means, it should be part of Annex60 or an external library that really fits our needs. As you said, there are various possible starting points and one of the weather data experts (this means: not me) would have to write down all requirements. Some requirements probably include TMY3 format support (for US and/or Energy+ users), netCDF format support, and some kind of interpolation and maybe looping data series.

[tbeu]

I am also not a weather data expert but checking the files you have, the spec and the Java conversion utility it does not look complicated. If I understand it correctly, currently the EPW files are converted to TXT files (which you call MOS) that the CombiTimeTable of the MSL can understand and which handles then the interpolation and/or periodic extrapolation.

One real improvement of the ModelicaStandardTables would be to decouple the file I/O from the core table business of interplation/extrapolation. Currently only MAT files and TXT files are supported. You ask for netCDF or HDF5 files. Many more formats are possible, e.g. in Modelica ticket #1153 Excel sheets were asked for. The idea is to have an replaceable package that holds the ctor and dtor of the external file object and one mandatory function to read the data. The goal of course is to have all external data only at the external side with no need to have it passed to the Modelica environment.

Sorry if I hijacked your issue for a ModelicaStandardTables issue. But since there is not much going on in Modelica ticket #1153 this "plugin concept" really looks like a valuable generalization. Tell me what you think. And let's see what the experts @mwetter and @MartinOtter think about it.

[mwetter]

@tbeu Your understanding about converting EPW files is correct; we simply change the syntax in order to use the CombiTable1Ds from the MSL.

Decoupling file I/O from interpolation and extrapolation is desirable in my view: In other words, if simply the file format changes from txt to mat (or hdf5), the output of the table reader should be identical. Hence, interpolation/extrapolation functions should ideally be reused.

I personally had not yet had a use case that requires hdf5, but @thorade or his co-workers may had. If hdf5 has indeed higher performance (compilation and simulation time) for very large tables, I think this could be of interest for example if annual measured data in intervals of seconds are available for a building.

[jraedler]

The fork of @tbeu has a lot of advantages for Modelica users, it should be preferred over my old code. The ncDataReader2 was developed for one-dimensional data sets (that depend on one variable like typical weather files) and is very efficient for smooth interpolation of large data sets and network operations, but it is not a general purpose file reader. At the moment I don't have any plans to improve or extend the code. netCDF 4 can use HDF5 as the storage engine, but it reads and writes its own structure and metadata in HDF5 files. As a result, only special HDF5 files can be read by netCDF and ncDataReader2. To implement a general HDF5 reader, the HDF5 API has to be used and not the netCDF API (like ncDataReader2 does).

…

-- Dipl.-Ing. Jörg Rädler CAD_LAB | Institut für Architektur und Städtebau | Fakultät Gestaltung Universität der Künste Berlin | +49 30 3185 2927 | [email protected]

[PMehrfeld]

As referenced by @thorade we are planning to add a model into the AixLib that makes use of the NcDataReader2 library and its functions.

Just a few words regarding the story behind that:
In some of my simulations I recognized problems that Modelica.Blocks.Sources.CombiTimeTable didn't deliver the desired values properly. For instance, only zeros were read in or in a one-year simulation with daily repetetions (repeat periodically) of the *.mat file the output of CombiTimeTable became zero after a couple of days. Therefore, I was looking for an alternative data reader.
So far NcDataReader2 works absolutely stable in my simulations.

@mwetter: Since NcDataReader2 will be used in 2 of 4 libraries, is there any interest in integrating it into the annex60 library?

@tbeu: What is your opinion about that, especially because @jraedler doesn't have any plans to continue development of NcDataReader2.

[tbeu]

@PMehrfeld

1. Since I am the maintainer of the CombiTimeTable of the MSL I am of course interested what the problem with periodic extrapolation is/was. Some problems were fixed in MSL 3.2.1+build.4, MSL 3.2.2+build.3 and current svn trunk, hence you might check it again with the updated MSL. If you have a minimal model, please let me know, such that I can check and file a Modelica ticket if applicable. In my opinion, this problem (of unknown nature) in CombiTimeTable does not yet justify a general change of the weather data reader.

2. As @thorade wrote above

   ... there are various possible starting points and one of the weather data experts (...) would have to write down all requirements.

   Up to now, I am not aware of these requirements.

3. In agreement with @jraedler I took over the maintenance of the NcDataReader2 Modelica library. I moved it from Bitbucket to GitHub (therefore, you won't recognize the fork origin in GitHub). The library is still actively maintained and I am going to relicense it under a more liberate license for easier integration in other MoLic2 packages.

[PMehrfeld]

@tbeu: I can provide the total model. But since it contains partially confidential models, which I cannot easily substitute by others to share the whole model here, I would suggest that you contact me via e-mail (see my profile).
I understand that an open-source minimal example would be the best, but unfortunately the problem so far only occured within complex and large simulation models.

addition: I tested it with MSL 3.2.2 build3 (2016-04-03)

[mwetter]

@tbeu @PMehrfeld
I would rather prefer to correct problems that @PMehrfeld apparently observed in the MSL table than working around it with another data reader in the Annex 60 library. If there are indeed performance gains with another data reader (e.g., netCDF or ncDataReader2), it would be nice to be able to use them from the MSL so that other application domains can also use it, and tool providers can package binaries if needed. Annex60 could then call it from the MSL, rather than implementing it directly.

[tbeu]

@mwetter We had this discussion at a previous MAP-Lib web meeting that it is not the aim of the MA to include anymore third-party C code dependencies in the MSL. There were some objections regarding maintenance effort and platform independence. Well, I am not sure how fixed this statement is, but at least it holds for now.

I compared the performance of TXT-file, MAT-file and netCDF-file based reading in modelica/ModelicaStandardLibrary#1682 (comment). Test tables were a 112156x2 matrix which are also attached there. For me, performance is not a real issue.

[tbeu]

@PMehrfeld Please try to minimize the model as much as possible and describe the expected vs. the observed behaviour.

[tbeu]

FYI: It turned out that @PMehrfeld's observation with the CombiTimeTable is the same as reported in modelica/ModelicaStandardLibrary#2080.

[tbeu]

I recently started some effort with the Modelica Table Additions - a drop-in extension of the Modelica Standard Tables but with additional support of external formats (currently CSV and JSON). It would then be possible to directly support EPW files without the need of the pre-processing step using ConvertWeatherData.jar.
@mwetter Let me know what you think of it.

[mwetter]

@tbeu : If the data reading is similarly efficient than the current approach which puts the table into the C code, then this would be interesting, i.e., if the overhead for parsing the epw file during run time is not significant. Note however that since a few days or weeks, many epw files are now also available as mos files, see for example https://energyplus.net/weather-location/north_and_central_america_wmo_region_4/USA/AK/USA_AK_Adak.NAS.704540_TMY
It appears that this conversion has been done for the US only at this time. I could drill down to see who posted the files and what the plans are.

[tbeu]

If the data reading is similarly efficient than the current approach which puts the table into the C code

Yes, this is the case.

Note however that since a few days or weeks, many epw files are now also available as mos files, see for example https://energyplus.net/weather-location/north_and_central_america_wmo_region_4/USA/AK/USA_AK_Adak.NAS.704540_TMY
It appears that this conversion has been done for the US only at this time. I could drill down to see who posted the files and what the plans are.

Would be interesting why the mos files are displayed such prominently there.

If you are interested in a native epw support available in Modelica interpolation blocks (by library Modelica Tables Additions) we could add it there.

[mwetter]

@tbeu : We would need to be able to conduct interpolation on some of the time series, please see datRea and datRea30 in
IBPSA.BoundaryConditions.WeatherData.ReaderTMY3.

Note however some items to consider in .epw time stamps. From the EnergyPlus support, we obtained:

According to the document description (and what EnergyPlus actually deal with the data), the field "1,30" should be for the minute 30 in the first hour--or the "Hour 1" as explained in 2.9.1.4 (from 00:01 to 1:00).

The best to think about any hour field "n" (Hour n) as the nth hour in a day, which starts from (n-1):01 to n:00.

For the TMY3 to epw conversion, the minutes 60 means it treats the time at exactly at the end of the hour. It should be a correct match. For example, if there are 24 hourly records in the TMY3 files in a day, there are deemed as values for Hour 1 at the minute 60 (so field 1,60), Hour 2 at the minute 60 (2,60), until Hour 24 minute 60 (24,60) when they are converted to epw records.

The minute field matters only when EnergyPlus processes sub hourly weather data (more than one records per hour, or Number of Records per hour is greater than 1).

So, if a file starts at 1999,1,1,1,0, it is for time 1:00. If it is 1999,1,1,1,60,, and the entries are hourly, then the 60 is ignored, e.g.,

1999,1,1,1,0, ...
1999,1,1,2,0, ...

is the same as

1999,1,1,1,60, ...
1999,1,1,2,60, ...

because the minute field 60 is ignored for hourly data. Note that https://www.energyplus.net/sites/all/modules/custom/nrel_custom/pdfs/pdfs_v9.4.0/AuxiliaryPrograms.pdf says "This is the minute field. (1..60)" but USA_CA_San.Francisco.Intl.AP.724940_TMY3.epw uses 0 in the minute field.

However, if the entries are sub hourly, then they seem to be

1999,1,1,1,30, ...
1999,1,1,1,60, ...  // or 1999,1,1,2,00, ... , I am not sure?
1999,1,1,2,30, ...

To be sure, one would need to double check if subhourly data, say at 30 min time stamps, start at 1999,1,1,1,30 or at 1999,1,1,0,30,. I don't recall where this is documented.

Also, because the first entry is for 1:00, and no entry is provided for 0:00, when we convert .epw to .mos, we duplicate the first entry and assign it for t=0. This then makes the file periodic, otherwise it misses one time stamp.

[tbeu]

EPW read support is now available in ModelicaTableAdditions v2.2.0 for MSL v4.0.0. I took ConvertWeatherData.java as inspiration and reimplemented in C. I also noticed that the last data line of the EPW file is not conidered at all. I will tag new version of ModelicaTableAdditions after some feedback.

[tbeu]

I will tag new version of ModelicaTableAdditions after some feedback.

ModelicaTableAdditions v2.2.0 is available now. It also considers leap year settings and data.

[thorade]

Thank you Thomas for this work, and for posting it. Modelon recently ran into a related issue and will give your table implementation a try!
We were using the weather data that is available here (csv, json and epw available):
https://re.jrc.ec.europa.eu/pvg_tools/en/tools.html#TMY

[tbeu]

Hi Matthis, thanks for letting me know. I believe the native EPW support is the real benefit here. Let me also know how your evaluation turned out. In case you like it, I would appreciate if you supported the development through (temporary) sponsorship by https://github.com/sponsors/tbeu. Thanks a lot.