LCM support wanted?

[tbeu]

LCM is for Lightweight Communications and Marshalling. The communication is based on UDP multicast and alternatively supports logfile or memory queue for testing. The LCM protocol is used in robotics by several research institutes, see http://lcm-proj.github.io.

I tested the LCM C library on Windows. The integration of LCM within Modelica, especially M_DD, would be straightforward and similar as UDP or CAN (with one additional library dependency on lib lcm that would need to be part of the M_DD distribution).

The marshalling part of LCM is not possible to use within a Modelica environment (since it is not a nominal type system). Thus the SerialPackager has to be reused for encoding/decoding of the data buffer. Endianess needs to be considered again (as is the case for UDP already).

@bernhard-thiele Please comment if it's alright for you to have support of the LCM protocol in M_DD. I will provide a PR once I am ready then.

[bernhard-thiele]

Sorry for the late answer, I was away from email etc.

I just scanned over the website and it sounds interesting. If LCM can be supported in a user convenient manner, it would be nice to have it in M_DD. From a first superficial look, LCM seems a bit similar to the Google protocol buffers (https://github.com/google/protobuf), but probably LCM is more optimized for real-time applications.

How are you going to map the LCM data type generation to Modelica (lcm-gen ...)? From above I assume that you would create the packages manually by using the SerialPackager. That seems a bit error-prone for the user. Or do you think of writing code that understands the *.lcm message definitions and generates Modelica code?

Another question is how to map the event based subscriber model to Modelica. It is not obvious to me what the best way of doing this would be. The example at the website shows that lcm_handle is called periodically and calls out to a function handler to deal with the message.

In summary: It would be nice to have LCM support, though it is not so clear to me how the "look" and "feel" of such a support in M_DD would be. Fine for me to give it a try and help testing/supporting the code under Linux.

[tbeu]

The lcmgen step is some kind of optional preprocessing once the protocol is defined. It needs to be done once for each kind of application. I do not think that it is possible to use the data strcutures defined by *.lcm or the lcmgen-generated headers in Modelica. Which means the LCM analysis tools are not really useful for this kind of Modelica applications interfacing LCM.

I was thinking of supporting the generic functions lcm_publish and lcm_subscribe and let the SerialPackager of M_DD do the buffer handling. This indeed might be error-prone for the user. It also might be possible that the SerialPackger writes a *.lcm file for the used protocol once during simulation runtime such that the user get's some feedback what the protocol is. The generated *.lcm file still can be used with LCM tools then.

Instead of sampled communication (like UDP or other blocks of M_DD) the LCM blocks could work with continuous time, i.e. lcm_handle will be called in each solver step. Once a new message is available the SerialPacker will get this data, however it is eventless in sense of Modelica. This would be different to the existing communication blocks of M_DD.

I will try to implement it but it may take same days or weeks. I got drone.io running for the x64 build of liblcm.a but I am still failing with cross-compilation for x64_86, see https://drone.io/github.com/tbeu/lcm/admin. Maybe @dietmarw has an idea why glib:i386 fails on Ubuntu 12.04. You are now admin of this project at drone.io, too.

[tbeu]

@ashuang FYI.

[dietmarw]

@tbeu You probably want to take a look at https://circleci.com which provides also support for artifacts AND a more modern building environment.

[tbeu]

LCM support is now added to new branch lcm. @bernhard-thiele Please check.

What's left to do:

• Adapt some CMake files for e.g. test_MDDLCM and ITI_MDDLCMWrapper. It takes me 3 min to set up the VS projects but 3 days for the CMake files. Thus, I do not feel in the mood for it.
• Add new test_MDDLCM to drone.io and Travis CI once CMake is adapted.
• Add LCM to the GitHub project description.

[tbeu]

@bernhard-thiele Did you have some minutes to check the new branch?

[bernhard-thiele]

Yes, I will check it later today.

[bernhard-thiele]

Had problems to build the example model (Examples.TestSerialPackager_LCM). Building the example model with Dymola on 64 bit Ubuntu 14.04 I needed to use the library dependencies {"pthread", "glib-2.0", "lcm"} and create the symlink ln -s libgthread-2.0.so.0 libgthread-2.0.so within the directory /usr/lib/i386-linux-gnu. By default Dymola builds 32 bit binaries, so we need to ensure that the 32 bit libraries of glib-2.0 and others are installed.

Now the model builds, but does not yet run. I get the error

The following error was detected at time: 0
MDDLCM.h: Could not allocate LCM object for network provider "udpm://"

Too late and to tired for looking into that now.

[tbeu]

1. As I wrote in the commit message liblcm on Linux requires libgthread-2.0 and libglib-2.0 as system dependency. I wonder why it worked with libpthread for you. For Windows I compiled lcm.dll w/o explicit dependency on glib/gthread, i.e., the required dependencies are statically included in lcm.dll. It would be nice if we could do the same for Linux. @bernhard-thiele, @dietmarw , @sjoelund Do you have an idea how to not use a shared object but rather include the dependent libs?

2. The run-time error is due to lcm_create returning NULL which is strange. I debugged it (on Windows) many times and could not see any problem with it. The lcm project comes with some tests. You could try these tests of the C interface and check them on Linux.

[sjoelund]

@tbeu choosing to link XXX statically is just -Wl,-Bstatic -lXXX -Wl,-Bdynamic. This won't work in a Library annotation I think, and as far as I know if cmake is used it is a bit tricky to get it right, but it can be done.

[bernhard-thiele]

@tbeu Maybe my firewall is responsible for the run-time error. Just had a quick test using the project examples which suggests that (it worked after deactivating the firewall). Need to look into this and also should play with the library dependencies if I find some time in the evening.

[bernhard-thiele]

It is a bit difficult to find time in the moment, it is not forgotten and I hope there is some more time next week.

[tbeu]

Thanks for the reply. Fine from my side.

[tbeu]

Any time spent on the topic?

[tbeu]

I can now confirm that it works with Dymola on Windows.

Did you find out what is the case with Dymola on Linux?

[bernhard-thiele]

I could get it run on my Ubuntu 14.04 64 bit, but maybe it's not very satisfactory.

1. Needed to disable my firewall
2. I had to change line 86 of MDDLCM.h from lcm->lcm = lcm_create(url); to lcm->lcm = lcm_create(NULL);

After that I could simulate the example and saw that values were received in the getXXX blocks.

@tbeu the example in the lcm distribution just uses NULL as argument, but I guess that is not acceptable here?

Preceding to this, I had to do one of the following (though it doesn't matter if one does both).

Variant 1: Follow the approach described in a comment above (repeated for convenience)

Use library dependencies {"pthread", "glib-2.0", "lcm"} and create a symlink ln -s libgthread-2.0.so.0 libgthread-2.0.so within the directory /usr/lib/i386-linux-gnu. By default Dymola builds 32 bit binaries, so we need to ensure that the 32 bit libraries of glib-2.0 and others are installed.

Variant 2: Installation of a 32 bit lcm installation

Since Dymola defaults to 32 bit code generation on Linux one needs to install the 32 bit version of lcm, which can be done by building the source distribution of lcm (v1.3.1 in my case) with

./configure --without-python --enable-static --build=i686-pc-linux-gnu "CFLAGS=-m32" "CXXFLAGS=-m32" "LDFLAGS=-m32"
make
sudo make install

In this case it is not required to have any lcm linux library shipped with MDD. Furthermore, no additional library dependencies need to be declared in Modelica (Library="lcm" is sufficient).

[tbeu]

Thank you for your time evaluating the lcm branch on Linux.

1. lcm_create(NULL) calls a default URL udpm://239.255.76.67:7667 which - of course - is not generic enough.
2. For Win (both Dymola and SimulationX) I created a lcm.lib linking with lcm.dll that already contains glib. For Linux I'd see the most benefit providing an liblcm.a along with M_DD that also already contains libglib. Shouldn't this be possible. I'd avoid to require any sources of lcm or glib as part of the library.

[bernhard-thiele]

Well, if you can compile glib for Windows, shouldn't it be possible to simply provide both libraries, "glib." and "lcm.", seperately instead of merging them together?

The advantage would be that

1. We wouldn't introduce a library which is called lcm, but is at the same time glib (hence, we wouldn't introduce something unexpected)
2. We easily could use the services provided by glib in other (future) code, too

[tbeu]

You are right. I will check it out on Win and update the lcm branch if it worked out.

[tbeu]

Right, bb4ad8a provides separate MSVC libs for glib-2.0 and gthread-2.0. The later one actually is not needed, but the LCM project calls it a requirement.

[bernhard-thiele]

Okay, I'm currently trying to update the cmake build system so that the glib and lcm dependencies can be managed with some convenience.

[tbeu]

I asked in lcm-proj/lcm#76 about the obsolete dependency on gthread-2.0. I recommend to remove it again from M_DD. Do you agree?

[bernhard-thiele]

Yes, it is fine to remove it.

[tbeu]

Done by 9713dbd.

[bernhard-thiele]

@sjoelund please correct or improve my answer if necessary.

-fPIC, as I understand it, is useful/needed for shared libraries, because a shared library can be loaded by several applications at the same time and -fPIC is one way to ensure that there will be no address collisions when different applications share the same library. However, -fPIC creates a level of indirection that usually has a slight performance penalty on Intel 32-bit systems. As I understood it, there is almost no performance penalty on Intel 64-bit systems, because some addressing details are handled there differently. Hence, having -fPIC seems not necessary in our case (static libraries), but it probably does not do much harm either.

[sjoelund]

It is necessary if the static library is at any time linked into a shared library (such as an FMU). For this reason, I would always use -fPIC for Modelica libraries.

[bernhard-thiele]

I do hope that the CFLAGS -O2 is actually used

Seems it was not used. Just reproduced it on my machine. If you want to set that variables when calling make you need to prepend them, e.g.,

CFLAGS="-O2 -fPIC" CPPFLAGS="-DNDEBUG" make install

The other way is to pass the CFLAGS etc. already to the configure script (that's how it is done in my cmake files).

[tbeu]

Oops, I cannot believe it. Look how I call make and how cc is invoked later in https://drone.io/github.com/tbeu/ModelicaStandardTables/172.

[tbeu]

But you are likely right. For the 32-bit build of glib I passed the CFlags already to configure which might explain the larger size for an optimized binary.

[bernhard-thiele]

It is necessary if the static library is at any time linked into a shared library (such as an FMU). For this reason, I would always use -fPIC for Modelica libraries.

Thanks, this is a valid point if one has a scenario in which one instantiates such an FMU several times.

What was surprising for me, is that ./configure doesn't include -fPIC into CFLAGS, even if one compiles without static library support (hence, only shared libraries). However, if one enables more verbose output (option ./configure ...... --disable-silent-rules), one sees that -fPIC is added anyway (at least for the 64-bit build that I tested, for both, static and shared libraries). Hence, not really necessary to add it to CFLAGS. But I think I will do so anyway and update the cmake files.

[tbeu]

But I think I will do so anyway and update the cmake files.

For the master or lcm branch?

[bernhard-thiele]

I committed to the lcm branch. I did small changes to the options for the 32-bit build and deactivated debug informations for glib completely (the project default is to add minimal debug information). In general, I really prefer to stick close to the compiler option defaults used by the respective external projects. I trust that the maintainers of the external projects mostly know what they are doing and choose reasonable and robust defaults. Hence, I also want to stick close to the defaults for the libraries that we compile for distribution with MDD, e.g., using -O2 instead of -O3 if the project maintainers decided that -O2 should be the default option for their release configuration.

Regarding the windows part, it would be good to make building on Windows platforms (Visual C) comparable convenient and automatized as in Linux.

Do you have any experience with Appveyor? Do you think that you could extend the cmake external project part to also cover Windows?

[tbeu]

I managed some projects with Appveyor, yes. The glib I used (https://github.com/winlibs/glib) and also lcm come with VS project files ready to use. Ic compiled both on VS 2010 (like all other M_DD binaries) on my own machine. Thus there is no need to adapt any CMake files for them. Do you think we should build both libs there? I doubt that it will work out as I simply was not successful compiling both 32-bit and x64 build. Sorry for that.

It is OK for me if you will stick with the default project settings for the Linux binaries. Just rebuild what you want to and commit them to the lcm branch.

[tbeu]

Ready to squash cherry-pick ab57cc5..4d6edaa from my side.

[bernhard-thiele]

Okay, I was experimenting with integrating the glib project into cmake.

We don't need a super tight integration into cmake, but it would be good if we managed to provide some support, e.g., downloading the external project in the correct version and printing some messages that tell the user to build the libs from the VS project. This is what I tried to achieve in 7cbf434.

An alternative would be to simply document what needs to be done, so that it can be reproduced.

A more tight integration might be possible with the cmake command "include_external_msproject", but I haven't experimented with that.

What are your thoughts about that?

[bernhard-thiele]

The glib repo that you use for VS (https://github.com/winlibs/glib) is not the original glib repo. What I find strange is that they only provide a single release in this repo. Any idea why they provide only a single release and whether this is supposed to be a stable repo or whether it is more some sort of an ad hoc approach?

[tbeu]

What are your thoughts about that?

CMake generated VS project files are really a pain. To be honest I generate them once and (hope to) never touch the CMake files in favour of the VS project files. For me it is a couple of minutes to setup a working VS env compared to hours/days with CMake approach.

Any idea why they provide only a single release and whether this is supposed to be a stable repo or whether it is more some sort of an ad hoc approach?

I have no idea how official the repo is, who it maintains and often it is synchronized with the GNOME repo. I once opened a PR (winlibs/glib#1) which never got any feedback before I closed it again. At lest the repo is Win-specific and the VS project files work out of the box.

[bernhard-thiele]

CMake generated VS project files are really a pain.

Supporting different platforms is anyway and I don't know a silver bullet for it. Cmake at least helps to maintain a common (revision control friendly) build dependency view at one place instead of potentially diverging build systems for different platforms. That's why I don't want to give up on it and why I'm eager that new developments integrate into it even if this means additional work. Otherwise, I fear the pain is just moved to a later point in time.

At lest the repo is Win-specific and the VS project files work out of the box.

Hm, I think we need to be a bit careful with it. They have not even forked it from the official repo, just committed the files in one big push (hence, not easily possible to track their changes to the official repo code). Who is behind the whole https://github.com/winlibs effort is not so clear to me. I just forked the repo, at least this prevents that the repo just disappears and we cannot rebuild the Windows glib. However, the dependency to this repo feels a bit brittle and is not entirely satisfying from a longer term maintenance perspective. I just currently don't know what to do best about it.

[tbeu]

I just currently don't know what to do best about it.

Building glib from official source with VS is not that easy (https://wiki.gnome.org/Projects/GTK+/Win32/MSVCCompilationOfGTKStack#GLib). Thus I was glad to have found these winlibs hosted repos. Even the official Win binaries have not been updated for years. See http://ftp.gnome.org/pub/gnome/binaries/win32/glib and http://ftp.gnome.org/pub/gnome/binaries/win64/glib. Let's kindly ask @weltling directly and hope for an answer: Who maintains the winlibs repositories? How offcial is the winlibs/glib repo? How often is winlibs/glib synchronized with the official GNOME/glib repo? Why is there only one release of winlibs/glib?

[weltling]

@tbeu thanks for asking. The winlibs repos serve to satisfy dependencies of the PHP project on windows. You can fetch the correspending bins http://windows.php.net/downloads/php-sdk/deps/ . The updates are being done case by case basis. Patches are being applied to fix possible Windows issues, but non necessarily.

So this is official in the sense for PHP. We've started to post the sources a few years ago. It's dedicated especially for PHP, but OFC it makes sense if other projects can share same sources.

Thanks.

[bernhard-thiele]

@weltling thanks for the explanation. It's great that you have put the things public and I'm grateful that this makes it easier for us to rely on glib within our Modelica_DeviceDrivers project.