A piece of Question about ADIOS converter bp2ncd #172
Comments
|
Hi,
The error indicates that you are using some characters in your variable
names that the netcdf format does not accept. An example is '/' since paths
are not allowed at all.
BTW, the bp2ncd tool is ancient and is not supported. No one has used it
for anything meaningful for so long, we just dropped to care about it. Why
do you need that conversion for your research?
Best regards
…On Tue, Mar 20, 2018 at 9:42 AM, Lucymugua ***@***.***> wrote:
I am using ADIOS in my research work. However, I got "Error:NetCDF: Name
contains illegal characters" when I using ADIOS converter bp2ncd to convert
bp file format to NetCDF format.
The version of NetCDF is netcdf/4.4-icc16-mvapich2.
Could anyone give me some advance on this error?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#172>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADGMLXfvAzGmOix3EEKJSswHd-FR5dxFks5tgQdJgaJpZM4Sx6le>
.
|
|
hi pnorbert, Thanks for you kindly reply. I have checked all of my variable names in the xml file and found that they contain only letters, numbers and underscores("_") which are allowed in using this tool. Thanks, |
|
Lucy,
Can I see your xml and the "bpls -lav" output for your file? Make sure the
"path" attribute in each <var> element is "" or does not exist.
You are right with your criticism, we either should completely remove this
tool from the release or stop staying it's not supported. I meant it may or
may not work correctly as it has not been used for years and is not tested
regularly.
Best regards
Norbert
…On Tue, Mar 20, 2018 at 9:31 PM, Lucymugua ***@***.***> wrote:
hi pnorbert,
Thanks for you kindly reply.
I have checked all of my variable names in the xml file and found that
they contain only letters, numbers and underscores("_") which are allowed
in using this tool.
[image: image]
<https://user-images.githubusercontent.com/37583049/37690973-7fdfd2a0-2ce9-11e8-9e8c-b7c792ef7651.png>
Futhermore, I am using ADIOS 1.12.0 which is the latest version. And there
is also a part about bp2ncd tool in ADIOS 1.12.0 user guide which is shown
in the above picture. If it have been dropped, why there is also an
introduction about it in the latest version user manual? And what do we
commonly use to convert bp file format to NetCDF or binary file format
currently?
Thanks,
Lucy
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#172 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADGMLREDJluPfzfsZVoAvQM5evE8cd7Wks5tga1ZgaJpZM4Sx6le>
.
|
|
Dear Norbert, Thanks for your kind reply. 1.xml file:
2.output file strcture:
3.bpls -lav results:
Thanks for your help. Best regards, |
|
Lucy,
Your troubles are probably caused by the adios timers in the file. Please
rebuild adios for your runs turning off the timers.
Use the options "--disable-timers --disable-timer-events" when configuring
the build with autoconf tools, or "export adios_timers=OFF" when using
cmake.
If you need this existing dataset, there may be a way to read-in/write-out
again using examples/staging/stage_write, after building that with a
timer-less adios installation.
Best regards
Norbert
…On Thu, Mar 22, 2018 at 11:29 PM, Lucymugua ***@***.***> wrote:
Dear Norbert,
Thanks for your kind reply.
My xml file, output file structure and bpls -lav outfile results are
listed below:
1.xml file:
[image: image]
<https://user-images.githubusercontent.com/37583049/37810048-37aa0906-2e8d-11e8-8777-38dc781c1248.png>
[image: image]
<https://user-images.githubusercontent.com/37583049/37810072-4790f5f0-2e8d-11e8-80e4-bc6aa99706cd.png>
2.output file strcture:
[image: image]
<https://user-images.githubusercontent.com/37583049/37810001-e6a74e06-2e8c-11e8-8cfa-be5d9281b61d.png>
[image: image]
<https://user-images.githubusercontent.com/37583049/37810014-00fdb358-2e8d-11e8-9e48-c89c645d28ca.png>
3.bpls -lav results:
[image: image]
<https://user-images.githubusercontent.com/37583049/37809955-a38d50b6-2e8c-11e8-8bd2-15af5799d4d4.png>
[image: image]
<https://user-images.githubusercontent.com/37583049/37809968-b1bfeec8-2e8c-11e8-99c1-d33ee800e47c.png>
Thanks for your help.
Best regards,
Lucy
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#172 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADGMLS1KPxOpdcGTTUo2pzfRzuEVYzGQks5thGwhgaJpZM4Sx6le>
.
|
|
Lucy,
My suggestion will still not help. bp2ncd is so old and unmaintained that
it does not work with the subfiles generated by the aggregation method.
I am looking for some solution but at the moment I have nothing to offer.
Sorry
Norbert
On Fri, Mar 23, 2018 at 9:22 AM, Norbert Podhorszki <
[email protected]> wrote:
… Lucy,
Your troubles are probably caused by the adios timers in the file. Please
rebuild adios for your runs turning off the timers.
Use the options "--disable-timers --disable-timer-events" when configuring
the build with autoconf tools, or "export adios_timers=OFF" when using
cmake.
If you need this existing dataset, there may be a way to read-in/write-out
again using examples/staging/stage_write, after building that with a
timer-less adios installation.
Best regards
Norbert
On Thu, Mar 22, 2018 at 11:29 PM, Lucymugua ***@***.***>
wrote:
> Dear Norbert,
>
> Thanks for your kind reply.
> My xml file, output file structure and bpls -lav outfile results are
> listed below:
>
> 1.xml file:
> [image: image]
> <https://user-images.githubusercontent.com/37583049/37810048-37aa0906-2e8d-11e8-8777-38dc781c1248.png>
>
> [image: image]
> <https://user-images.githubusercontent.com/37583049/37810072-4790f5f0-2e8d-11e8-80e4-bc6aa99706cd.png>
>
> 2.output file strcture:
> [image: image]
> <https://user-images.githubusercontent.com/37583049/37810001-e6a74e06-2e8c-11e8-8cfa-be5d9281b61d.png>
>
> [image: image]
> <https://user-images.githubusercontent.com/37583049/37810014-00fdb358-2e8d-11e8-9e48-c89c645d28ca.png>
>
> 3.bpls -lav results:
> [image: image]
> <https://user-images.githubusercontent.com/37583049/37809955-a38d50b6-2e8c-11e8-8bd2-15af5799d4d4.png>
>
> [image: image]
> <https://user-images.githubusercontent.com/37583049/37809968-b1bfeec8-2e8c-11e8-99c1-d33ee800e47c.png>
>
> Thanks for your help.
>
> Best regards,
> Lucy
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#172 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADGMLS1KPxOpdcGTTUo2pzfRzuEVYzGQks5thGwhgaJpZM4Sx6le>
> .
>
|
|
Dear Norbert, Thanks for your solution. BTW, is there any other kind of tool have the same function as bp2ncd to do the conversion? Best regards, |
|
Hi Lucy,
If you are comfortable with python/numpy and building libraries for python,
then probably that's the easiest way to do the conversion. ADIOS has a
numpy wrapper (wrapper/numpy/ in the source that has to built and installed
for python separately). There is also a netcdf4 package for python/numpy
http://unidata.github.io/netcdf4-python/
https://pypi.python.org/pypi/netCDF4
If you want to work in C, then there is the
examples/staging/stage_write/stage_write.c code that can read any BP file
of global arrays in parallel (you define the decomposition for reading) and
then writes out again using any ADIOS methods. Replacing the last part for
NetCDF should work for you. Also, your variables are fairly small, so this
should run fine serial.
Best regards
Norbert
…On Sat, Mar 24, 2018 at 9:13 AM, Lucymugua ***@***.***> wrote:
Dear Norbert,
Thanks for your solution.
I am planning to write a program about convert bp format to NetCDF format
on my own without using the tool.
BTW, is there any other kind of tool have the same function as bp2ncd to
do the conversion?
Best regards,
Lucy
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#172 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADGMLXEV8CK2ccEptxE4cLSY2IhCWZqgks5thkZmgaJpZM4Sx6le>
.
|
|
I have been working on developing the Adios python wrapper and made an example to convert ncdf to bp file (wrappers/numpy/example/utils/ncdf2bp.py). If you need, I can help to write "bp2ncdf.py". If this is the case, it will be great to have an example ncdf file.
Best regards,
Jong
…--
Jong Youl Choi
Scientific Data Group
Computer Science and Math Division
Oak Ridge National Laboratory
Homepage: http://www.ornl.gov/~jyc/
On Mar 24, 2018, at 9:47 AM, pnorbert ***@***.***> wrote:
Hi Lucy,
If you are comfortable with python/numpy and building libraries for python,
then probably that's the easiest way to do the conversion. ADIOS has a
numpy wrapper (wrapper/numpy/ in the source that has to built and installed
for python separately). There is also a netcdf4 package for python/numpy
http://unidata.github.io/netcdf4-python/
https://pypi.python.org/pypi/netCDF4
If you want to work in C, then there is the
examples/staging/stage_write/stage_write.c code that can read any BP file
of global arrays in parallel (you define the decomposition for reading) and
then writes out again using any ADIOS methods. Replacing the last part for
NetCDF should work for you. Also, your variables are fairly small, so this
should run fine serial.
Best regards
Norbert
On Sat, Mar 24, 2018 at 9:13 AM, Lucymugua ***@***.***> wrote:
> Dear Norbert,
>
> Thanks for your solution.
> I am planning to write a program about convert bp format to NetCDF format
> on my own without using the tool.
>
> BTW, is there any other kind of tool have the same function as bp2ncd to
> do the conversion?
>
> Best regards,
> Lucy
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#172 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADGMLXEV8CK2ccEptxE4cLSY2IhCWZqgks5thkZmgaJpZM4Sx6le>
> .
>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Hi Norbert, Thank you so much for your advice. Best regards, |
|
Hi Jong, I would greatly appreciate it if you can help to write"bp2ncdf.py". I really need that kind of tool to help do the format conversion. Best regards, |
|
Hi, Lucy,
I have just added "bp2ncdf.py" in wrappers/numpy/example/utils. The usage is as follows:
usage: bp2ncdf.py [-h] [-v] BPFILE OUTFILE
This is the first version and may have some bugs. Please feel free to report any problem.
Best regards,
Jong
…--
Jong Youl Choi
Scientific Data Group
Computer Science and Math Division
Oak Ridge National Laboratory
Homepage: http://www.ornl.gov/~jyc/
On Mar 27, 2018, at 4:32 AM, Lucymugua ***@***.***> wrote:
Hi Jong,
I would greatly appreciate it if you can help to write"bp2ncdf.py". I really need that kind of tool to help do the format conversion.
Best regards,
Lucy
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Hi Jong,
Best regards, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am using ADIOS in my research work. However, I got "Error:NetCDF: Name contains illegal characters" when I using ADIOS converter bp2ncd to convert bp file format to NetCDF format.
The version of NetCDF is netcdf/4.4-icc16-mvapich2.
The text was updated successfully, but these errors were encountered: