Build and run custom version of EVerest on the uMWC #51
Comments
|
Connection refused while compiling the environment. See below. 
|
|
I think it was getting blocked by the NREL firewall. Was able to access using the hotspot network. |
|
I am currently on the third step on the Pionix guide where you have to run the It seems like something is wrong with To get here, I did |
|
this is the same error that Faizan got. When you untar the toolchain, what directory does it create? Maybe you have to untar it in a different way or move it around or sth. you need to show that you have a |
|
When I untar the toolchain it creates a directory called
It is weird because I do have a Any ideas on where I would move it or where I should untar it instead? |
|
I think that this may be our issue, I'm figuring out how to install this dependency right now to see if that fixes it. |
|
I don't know a lot about Linux, so please correct me if I'm wrong, but it seems like the issue we are experiencing is because I am trying to use Alpine Linux (via the docker container) instead of Ubuntu. Apparently Alpine uses In the Alpine Linux wiki, they have a way to run |
|
Perfect! Unfortunately, we bricked the uMWC while experimenting today so I can't test this until I get some new firmware. But maybe you can zip up the directory that we are supposed to copy over using We will need to repeat this once the docker image with the |
|
Unfortunately I've hit another roadblock, instead this time with the final I figured it may be because the docker container doesn't have enough resources, but I'm decently below my max utilization for everything. I'll keep working on this and see if I can get this to work. |
|
It might just be slow. There is a reason that "compile the Linux kernel" used to be the standard workload for performance testing in the old system papers.
Bump up the resources anyway just to be on the safe side and then just wait for 30mins or so
|
|
I tried bumping all the resources to 75% of what my computer can handle (12 gb RAM, 6 cores, 3 gb swap) and it still is stuck in the low 60s. I left it for about a hour and it doesn't seem to be making any progress past that. I did get an error now, however, although it seems a bit unhelpful. Here are the logs: |
|
I read somewhere online that the |
Building EVerest Using DockerThe cross compiling steps are outlined here, in the EVerest docs. You are required to have EVerest built natively, which means you need a Linux machine, and the rest of the commands are Linux based as well. We can circumvent this hardware requirement by doing this build process on a docker container which 1) has EVerest built on it, and 2) runs Linux Alpine. Thankfully, we have one line demos which builds EVerest to these containers for us. Below is a complete walkthrough of the steps I took to utilize Docker to build for hardware. Steps
|
|
Affer switching to the custom build that includes smart charging, ran into this error This was caused by EVerest/libevse-security@8777cc2 Double-checked the uMWC, and found So we will have to revert the libevse change. Let's try only removing that dependency and see if it works. EDIT: Removed dependency in EDIT: cmake completed successfully, starting build... |
|
Build failed with Replaced all the text in /ext/source/build-cross/_deps/libevse-security-src/lib/evse_security/crypto/openssl/openssl_supplier.cpp with the version before the latest changes: Ran into issues with Copied over deleted code. There are still failures. Trying to just revert the dependency and rebuild. This was only changed 2 weeks ago, and computers are faster than me. |
|
Reverted back to version 0.6.0, OCPP compile is failing with Trying to revert to the commit just before this change. EDIT: Setting the |
|
Now it is required in Not easy to roll this back, so let's just try to edit and assume that the dependencies are not too bad. |
|
Broke in the same way I guess the interim changes are not known to work. Off to check the openssl code... |
|
ok so verified that these are from SSL 3 https://man.archlinux.org/man/X509_add_cert.3ssl.en And grepping seems to find the old value as well. Did this ever work? When was it added? There are no symbols for Having said that, I do see a |
|
Final issue: it looks like openssl cleaned up their code by adding consts and plumbing them through. So the calling code has consts but the function does not. I am making the assumption that the underlying implementation has not changed significantly, so we just need to un constify it. I did consider removing the const, but it is passed in as an argument, so we would need to go through and remove all the call sites and potentially their call sites and so on, which will make this very complex. I can create a new struct fairly easily, but copying over the fields is complicated. And some of them are functions. So I just use This code, copied three times, lets the compile succeed! |
|
Verified that there is also a firmware file and only one firmware file which is built here Double-checked and the instructions from Pionix say that it should be in the "/libexec/everest/modules/YetiDriver", which is not the same, but at least it is in the YetiDriver directory. Just to be on the safe side, I will first backport the changes and check in the SIL before pushing it to EVerest, and will update EVerest without flashing the firmware first (learned my lesson last time!) |
|
edited Recompiled. But I can't kill the existing process! Need to pull all the patches and recompile to restart the new version. |
|
SIL fails Trying SP2 and then SP1 as backup |
SP2 got the same errorBut SP1 workedSo it is almost certainly a cert issue. We are going with SP1 tomorrow and will have to figure out how to debug cert issues in the future. |
|
With all these changes, SP1 works with EIM. PnC still fails because of SP1 and because the CSMS can't validate the cert. Checking in these changes and re-cross-compiling... EDIT: The demo is cursed. I finally got to the point where I had a stable build that included all the changes required for crosscompiling and was running (at least in a restricted version). And then the pionix release server went down, so no toolchain for me... EDIT: This is because I was on the VPN which blocks port 8888. getting off allows the download; phew! |
- Dependencies.yaml: set the security module version EVerest#51 (comment) - libevse-security: remove the validation of sub-certs EVerest#51 (comment) - Make the context be unconst EVerest#51 (comment)
We now need to: - reconfigure (dependencies.yaml) - rerun cmake - apply code patches (these can't be applied earlier because the right dependency isn't there) - rebuild - apply non-code patches that would be overridden Testing done: - Ran with SP1, started everest, charged with EIM EVerest#51 (comment)
|
Cross-compile was successful.
That won't actually work any more because the modules in the config have changed. Let's copy over the config that we use with our SIL It actually has 2 EVSEs and two connectors It now starts up, but tries to connect to the local OCPP server, which fails. Resetting the host... The OCPP connection is now set up correctly and we can receive a setChargingProfile! But it is invalid because the rate is incorrect. Fixing that... SetChargingProfiles now worksNext step: I see a message saying: Let's enable the uMWC BSP on this config after copying it to a safe place.
Now the error is I guess it is time to update the firmware but let's check the code and verify first. |
|
Bingo! Let's update the firmware. Firmware updates are failing Reverting back to the simulator temporarily. This should allow us to get the communication to work again and have a "new" everest starting up by default. Then, we can try to downgrade the BSP and see if it works. |
|
This might actually be easier than I thought, and might be a bug in the code. The simulator, phytec board, and BSP all set the connector type, but the uMWC does not Let's try hardcoding it and rebuilding... |
/ext/source/build-cross/modules/API/API: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, not stripped
# ...
/ext/source/build/modules/API/API: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, with debug_info, not stripped ... the native build includes debug info, but cross compile doesn't. We could probably set this flag somewhere in the source tree / make files, but I haven't dug too deeply.
|
I was able to replicate these steps and also see a function that I added in the logs! |
|
When I go to check the manager... /bullseye-toolchain/crosstool/bin/armv8-rpi4-linux-gnueabihf-objdump -S /ext/source/build-cross/dist/mnt/user_data/opt/everest/bin/manager > /tmp/temp.log...I'm not seeing the signatures of the functions I added. They still appear within the intermediate phases' ARM code (API & YetiDriver) -- am I checking the wrong file? We should expect to see these signatures within the final manager, no? I deleted and re-compiled to ensure that I'm not looking at an old version... I suppose there's a non-zero chance that my dummy files got optimized out via gcc (I don't know what flags we use, but IRRC, higher levels of optimization may remove "useless" code during optimization). From what I know about GCC, this shouldn't be an issue since we're defining a whole function and writing to the log stream. Anyway, I'm glad to hear that you can replicate my build process @Abby-Wheelis ! LMK if those functions show up on the hardware -- I'll spend a bit more time digging into the manager's ASM, then I'll try running on the VM. |
Yes, after copying over I can see my function when I FWIW I can't find it in It seems like I am updating the code, and the updates are copied over, so I'm going to add LOTS of logs and see if I can get anything to show up on hardware since it seems like the changes are persisting so maybe it's just crashing too early for the logs I tried to add earlier. Edit: added |
|
Good to know that we're seeing changes to the individual ELFs after copying! That rules that out, at least.
Since we're not seeing the changes in |
|
Let's trace the workflow to see how we compile
The YetiDriver process defined in Makefile2 above seems to be the last mention of the module; likewise, nowhere in this section do I see a linking phase similar to |
|
Investigate possibility of path being hardcoded:
|
|
Yay! I knew it had worked for me! Such a weird issue, and so annoying |
I think so! There were a few parts of the config that I commented out, so I want to make sure we're testing with the "right" config, and debug anything else that comes up, but it does run, and we now know that code must be in /mnt/user_data/opt to be called as expected. |
|
I think we are tracking that as a separate issue, so once you have checked over the configs, and confirmed that it consistently runs, we can go ahead and close this one (finally!!) |
|
I will also point out that this only works if we disable the energy manager, which results in a max current of 0. In the SIL, if I enable the energy manager, it still crashes. @Abby-Wheelis can you confirm that this is true even on hardware? Will spend a time-bounded effort tonight to see if I can get the energy manager to stop crashing so that we can try to actually send a real current value. |
|
ok so the energy manager thing is weird. It looks like everything crashes ~ 10 seconds after the manager starts I enabled debug logging in If none of the sleep statements are in place, the energy manager crashes after 999 runs If all the sleep statements are in place, the energy manager crashes after a single run In both cases, the crash occurred after around 10 seconds So I suspect this is related to some kind of threading/timing issue and not a particular issue with the code. Note also that the optimizer doesn't actually return any values; in the |
|
Looking at logs from a prior successful SIL run (non-cross-compiled), I can see that the optimizer returns a vector with a single value let's try to return that value and see how it goes. |
|
After the changes, I see And then a crash Will fix the UUID to |
|
Removing the infinite loop causes the process to not crash, but goes back to 0 power, similar to disabling the module completely |
|
What is interesting is that with the loop and the forced limits, the That's weird, because I set the hardcoded value to be valid for 3600 seconds, which should be an hour. Next tries:
|
Yes, I can confirm that the thisAnd then ran, seeing lots of logs that I added, and a crash from logs: |
Bump up the validity to 3600,000 (just in case it is milliseconds) -- did not get fixedSet the limits in the loop without calling the optimizer - does not crash, but we are still getting the `Power budget expired, falling back to 0.` message. Note that this is more of an indication of how the problem is with the mutexgenerates So we do need to muck around with the |
Found those logs in `modules/EvseManager/Charger.cpp`
So that is where the original (expiring) limit comes from. Why aren't we getting any values from the EnergyManager? Aha! It looks like And both of them check to see if the response is "for them" Let's see if that is why they are ignoring the limit... |
Bingo!Changed the UUID to match and everything seems to work now@Abby-Wheelis can you also apply this additional patch and re-enable the energy manager? I have also attached the expected logs. Once you confirm that this works, we can close this issue and move to #75 EDIT: Actually attached the logs |
|
I see you are running into a lot of issues, but I do not understand what problem you are solving here? The umwc yocto image comes with an sdk that can be used to cross compile Everest and install it under /var/everest and it just works. |
|
Why do you want to hardcode the uuid? This is definitely wrong. |
|
@corneliusclaussen our current plan for the CharIN milestone is to experiment with plugging in and yocto builds in parallel.
|
|
Iirc all you want to do is cross compile Everest and run it on the umwc, that's why I don't understand why you are running into so many issues. I think it is a combination of a very old Debian image and a very old Everest version with lots of changes. Once you are on yocto it can be updated with rauc again. And a custom build can be installed under /var/everest In this build there is a symlink /etc/selected-everest which you can point to the /var/Everest install, and then you can also use this custom version from the ui. So it's all prepared for that. |
Yup. Note that the first issue that I ran into was the SSL version - everest had rolled forward to SSL3, but debian (or at least the toolchain) had only SSL1 installed - 90% of my hacks were related to rolling back to the older version of SSL. I am still not sure what is causing this most recent error with the I think that part of it was also a little bit of "too many cooks" - AFS made the smart charging changes; I did the initial cross-compile, and then tried to hand it off to @Abby-Wheelis and @faizanmir21 who are on-site and can access the EV lab, but don't have a lot of experience with EVerest. But going forward, as we update to a more recent version, we will use yocto, and hopefully it will be simpler. We will be tracking the work required to get the yocto build to work in #76 Fingers crossed that it will be smooth sailing! |
I made this patch (was there another patch I needed to make too? It looks like most of the other work was logging/troubleshooting) and restored the manager, and it no longer seems to be crashing.
Are these the logs you were talking about?
If so, then I believe that is what I am seeing, after initialization I am seeing a loop of messages: |
|
No, only this patch. Further updates on creating a custom version of EVerest using the new yocto method will be tracked in #76 Hopefully rolling forward will be smoother. |
The EVerest project has open hardware as well https://everest.github.io/nightly/hardware/pionix_belay_box.html which is available as a kit from Pionix. Pionix also sells the uMWC (https://shop.pionix.com/products/umwc-micro-mega-watt-charger). This is is a non-open device in a housing that shares some hardware with the Belay Box although it has a different power module that is limited to 1W output.
In this issue, we will track the steps to run a custom build of EVerest on the uMWC so that we can perform HIL testing.
@faizanmir21 The instructions are here:
https://everest.github.io/nightly/hardware/pionix_belay_box.html#developing-with-everest-and-belaybox
They are for the Belay Box, but I'm hoping that they will apply to the uMWC as well. If not, we can ask the community for help.
My suggested steps are:
everest-dev.serviceand verify that it starts the dev build from/mnt/user_data/opt/everestdocker exec -it ....manager /bin/bashORdocker run -it ghcr.io/everest/everest-demo/manager:0.0.16 /bin/bash@drmrd @couryrr-afs @wjmp for visibility
The text was updated successfully, but these errors were encountered: