Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ADIOS1 String Attributes #269

Merged
merged 2 commits into from
Jun 18, 2018
Merged

Fix ADIOS1 String Attributes #269

merged 2 commits into from
Jun 18, 2018

Conversation

ax3l
Copy link
Member

@ax3l ax3l commented Jun 16, 2018

Fix broken string attribute writes in ADIOS1: Used invalid pointer to string array before and leaked memory as well.

Also enable serial ADIOS1 on OSX for CI testing.

Background

Valgrind says there are a few memory leaks in std::string attribute writing.

Also, there seems to be a SerialIOTest crash in ADIOS1 attribute reading, probably some un-init bytes? Before crash it reads:

ERROR: Attribute element /iterationFormat has invalid value attribute
ERROR: Attribute element /iterationFormat has invalid value attribute

which looks like a string Datatype issue in our backend.

Message origin in ADIOS: https://github.com/ornladios/ADIOS/blob/v1.13.1/src/core/adios_internals.c#L1146-L1161

During valgrind runs, don't get confused by some noise added by ornladios/ADIOS#184

Enable serial ADIOS1 on OSX for CI testing.
@@ -1335,7 +1335,8 @@ ADIOS1IOHandlerImpl::readAttribute(Writable* writable,
}
}

free(data);
if(data)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does not change the OSX issue

@ax3l
Copy link
Member Author

ax3l commented Jun 18, 2018

@C0nsultant from the valgrind issues (annotated inline) and the error message, I think we might actually have a little bug on our ADIOS1 IO backend side, at least for the representation of std::strings.

@C0nsultant
Copy link
Member

C0nsultant commented Jun 18, 2018

Can you narrow down the conditions under which this triggers?
Does this only happen for files that we write? Does it also happen for files written by other codes (e.g. PIC)?

If the former is the case, this might actually be a bug in ADIOS attribute writing, rather than reading

case DT::STRING:
return adios_string;

default:
nelems = 1;

case DT::STRING:
{
using uptr = std::unique_ptr< void, std::function< void(void*) > >;
values = uptr(const_cast< char* >(att.get< std::string >().c_str()),
[](void*){ });
break;
}

Relevant ADIOS manual part:

4.1.8
...
Simple string attributes should be defined 
with the type adios_string,
nelems=1 and
values should be the char* type pointer.

@C0nsultant
Copy link
Member

C0nsultant commented Jun 18, 2018

I may have spotted the culprit (see #271).
The Variant::get() returns a copy.

/** Retrieve a stored specific object of known datatype with ensured type-safety.
*
* @throw std::bad_variant_access if stored object is not of type U.
* @tparam U Type of the object to be retrieved.
* @return Copy of the retrieved object of type U.
*/
template< typename U >
U get() const
{
return variantSrc::get< U >(m_data);
}

When used to assign the uptr, all is fine.
case DT::STRING:
{
using uptr = std::unique_ptr< void, std::function< void(void*) > >;
values = uptr(const_cast< char* >(att.get< std::string >().c_str()),
[](void*){ });
break;
}

But as soon as the assignement completes, the temporary std::string gets destroyed.
The .c_str() pointer immediately becomes invalid.

@@ -977,7 +977,7 @@ ADIOS1IOHandlerImpl::writeAttribute(Writable* writable,
"",
getBP1DataType(parameters.dtype),
nelems,
values.get());
values.get()); // Invalid read of size 1 (could be in ADIOS itself?)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in strlen() in adios_get_type_size (adios_bp_v1.c:2415) in adios_common_define_attribute_byvalue (adios_internals.c:1109) in adios_define_attribute_byvalue (adios.c:442) in openPMD::ADIOS1IOHandlerImpl::writeAttribute on string value: serial_fileBased_write%T

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already pass the invalid c-string.

strlen((const char*)values.get()) is an invalid read of size 1

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very likely because of #269 (comment).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah dang, just found it as well :)
yes, we can not defer/forward the pointer we get there :)

@@ -842,7 +842,7 @@ ADIOS1IOHandlerImpl::writeAttribute(Writable* writable,
{
using uptr = std::unique_ptr< void, std::function< void(void*) > >;
values = uptr(const_cast< char* >(att.get< std::string >().c_str()),
[](void*){ });
[](void*){ }); // leaks memory
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

found it, will replace this part

@ax3l ax3l changed the title Travis: ADIOS1 on OSX Fix ADIOS1 String Attributes Jun 18, 2018
@ax3l ax3l removed their assignment Jun 18, 2018
[](void*){ });
auto const & v = att.get< std::string >();
values = uptr(new char[v.length() + 1u],
[](void* ca){ delete [] (char*)ca; });
Copy link
Member

@C0nsultant C0nsultant Jun 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used invalid pointer to string array before.
Copy link
Member

@C0nsultant C0nsultant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the inline comments removed, this is good to go.

@ax3l ax3l merged commit e50c6f2 into openPMD:dev Jun 18, 2018
@ax3l ax3l deleted the topic-osxADIOS branch June 18, 2018 17:49
@ax3l
Copy link
Member Author

ax3l commented Aug 9, 2018

Fun fact: the long double valgrind reports we get in ADIOS1 are actually a valgrind bug:
https://bugs.kde.org/show_bug.cgi?id=397313

@C0nsultant
Copy link
Member

You know that long double is uncommon when even a tool dealing with such wizardry as memory debugging has not cared for it in the last 16+ years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants