Add OTA reflashing support on TX2 #113
Conversation
atharvanan1
force-pushed
the
dunfell-l4t-r32.4.3+ota-components
branch
from
9eadaec to
8720d27
Compare
There was a problem hiding this comment.
Hey @atharvanan1
Thanks for sharing this PR, I guess that you tested this only with Jetson TX2
Do you need support/help to validate this with other Jetson's platform ?
I'm seeing some errors in the last traces ? Do you need help to fix these errors ?
|
@atharvanan1 Looks like this is due to a difference in the NVIDIA flashing tools between L4T R32.4.3 and R32.5.x. If I directly flash your R32.4.3 image onto my TX2, this is what I see for the eMMC partitioning: Note that the starting sector for I'll update |
|
@atharvanan1 I've updated |
|
@madisongh Great! Thanks for your input! I'll try it out on my end, and report back with my findings. @ichergui Thanks for the offer. I created this PR so that it's easier for Matt to review my testing with OTA flashing update. |
|
I was wrong about the root cause being a difference in the L4T tools.... The actual root cause is due to the But since APP-first is more typically found, I'll leave things as they are for now, then make sure that when I generate the |
@ichergui Let's keep this PR for TX2 device. With Matt's setup, I'm testing on TX2 c-boot setup, and will be porting to TX2 uboot setup from L4T-28.2. I'll update the status of my testing and add some additional information on what I've tested so far on https://github.com/OE4T/meta-tegra/wiki/Over-the-air-reflashing-process |
Perfect. Thanks @atharvanan1 Let me know if you need help |
|
@madisongh Thanks for that fix! The update process works with I was wondering if we could remove the dependency on PPTSIZE being 16896. If we could use partition_table file to get PPTSIZE that was used by the build and try to flash according to that. Are there any other dependencies we would want to watch out for? |
atharvanan1
force-pushed
the
dunfell-l4t-r32.4.3+ota-components
branch
from
8720d27 to
85ac2da
Compare
That should be doable, as long as the actual size is 16896 or greater (if it's less, normal Linux GPT tools won't work). There's already a field in the partition_table file for specifying the starting sector number, and that could be calculated from the info in the XML file. I'll see if I can work that in along with handling the alignment tags. |
Sounds good! I was thinking about PPTSIZE set up by image_types_tegra.bbclass when I said that. Let me know once the changes are done, I'll help test that on my end. |
atharvanan1
changed the title
OK, the update to |
atharvanan1
force-pushed
the
dunfell-l4t-r32.4.3+ota-components
branch
from
85ac2da to
faec342
Compare
|
Tested OTA update:
EDIT: Nevermind, this was the Update process worked perfectly fine! Additional note: You can also use USB stick to deliver the files if the USB stick has ext4 filesystem. |
layers/meta-tegra-support/recipes-ota/packagegroups/packagegroup-demo-base.bbappend
Outdated
Show resolved
Hide resolved
layers/meta-tegra-support/recipes-ota/tegra-bup-payload/tegra-bup-payload_%.bbappend
Outdated
Show resolved
Hide resolved
layers/meta-tegra-support/recipes-ota/tegra-tools/tegra-sysinstall-tools_1.6.1.bb
Outdated
Show resolved
Hide resolved
layers/meta-tegra-support/recipes-ota/tegra-tools/tegra-sysinstall-tools_git.bb
Outdated
Show resolved
Hide resolved
atharvanan1
force-pushed
the
dunfell-l4t-r32.4.3+ota-components
branch
3 times, most recently
from
91477de to
c8d33cb
Compare
|
I'll be running mender-torture-tests on |
|
Errors while running on While running the script, we get Where does it come from? Here This is caused by mounting new-rootfs as read-only. I think I've seen a comment that we need to do this for supporting delta-updates. While, we are trying to update the Errors while running on Errors while running on Errors while running on Will try to see what's causing this, and report back Concluded that the errors were due to me building on a branch that didn't have the latest commit as mentioned here - #113 (comment) |
|
From my testing,
Apparently, jetson-nano-emmc device fails to update the bootloader. Probably, this applies. |
Could you please share the logs ? |
|
Here's the log for Xavier-nx mender-torture where I tested with 20 mender updates and then a reboot torture - I don't know if this isn't sufficient, however I feel that this is good enough to call successful. Let me know if you want me to test more. Errors with nano: When I ran the jetson-nano-emmc's EDIT: Here's the bash -x on the artifact install script. UPDATE: Jetson TX2 u-boot logs - |
|
@ichergui @madisongh I have couple of questions and might need help on this. It seems that how we handle VER partittions for |
|
Based on further investigations, I found that the errors with Here's a hexdump for VER and VER_b partitions from Here's a hexdump from VER and VER_b partitions from Based on Mender Tegra Partition Layout, we see that the UBENV is located around 0x3bb000 on /dev/mmcblk0boot1 device. fw_env.config 0x3bb000 + 0x20000 = 0x3db000 Since, VER_b is located at 0x3e0000 and VER is located at 0x3f0000. They will get overridden. It looks like we have already handled this here - OE4T/meta-mender-community#5, which exists in Looks like the nv_update_engine doesn't care about VER partitions being wiped - because mender updates work fine! I think we need to apply above-mentioned patch on p3450_0002_defconfig as well. I added that just now. I tested and the mender update works fine! Here's the layout after the fix: |
nv_update_engine isn't used on the t210 machines (TX1 and Nano). There's a Python script that NVIDIA provides for bootloader updates on those platforms. The logic I have in tegra-boot-tools is more or less the same as what's implemented in that script. The reason Mender updates work fine with TX1/Nano is because Mender updates don't do bootloader updates on the t120 platforms. So you'd never notice this. For an OTA reflash, we should be erasing the boot device if it needs to be reformatted - which it would if you've got the Uboot-env-overwriting-VER problem. I added some improvements to tegra-boot-tools and the tegra-sysinstall scripts to better handle the needs-erasing case, but they'll also need an updated |
|
NOTE: this change doesn't work when you move from It breaks because VER partitions are overriden and My solution was to pack the VER partition contents into the new image and write them. However, @dwalkes mentioned UBENV location being changed - which might break things. @dwalkes do you want to jump in and give your input. |
...ta-tegra-support/dynamic-layers/meta-mender-tegra/recipes-bsp/u-boot/u-boot-tegra_%.bbappend
Outdated
Show resolved
Hide resolved
|
I have a change in place to remove the partition layout change, and updates to the script to add verbose messaging in a local branch. I'll run some tests and update you on that! |
|
Log showing the warning for Will run some mender torture on this device. Pushing cleaned commits. |
atharvanan1
force-pushed
the
dunfell-l4t-r32.4.3+ota-components
branch
2 times, most recently
from
cac6ed4 to
c9e46b3
Compare
I have something in works to add these changes into Tested mender updates on
|
I don't think we support mender-torture on u-boot environments yet, see https://github.com/mendersoftware/meta-mender-community/tree/dunfell/meta-mender-tegra/scripts/test#tegra-mender-torture-tests |
- These changes add OTA reflashing support as per https://github.com/OE4T/meta-tegra/wiki/Over-the-air-reflashing-process Signed-off-by: Atharva Nandanwar <[email protected]>
to include the full_init_payload for OTA reflashing purposes. Signed-off-by: Matt Madison <[email protected]>
This is done to match Matt's setup here madisongh/tegra-test-distro@ebec504 Signed-off-by: Atharva Nandanwar <[email protected]>
…tools replacing the NVIDIA bootloader update tools. Note that for cboot platforms, the InstallEnter script may still need to use nvbootctrl for getting the current boot slot, since the script would run in the context of an older rootfs that only has the NVIDIA tools installed. Signed-off-by: Matt Madison <[email protected]>
Signed-off-by: Matt Madison <[email protected]>
to provide a modified platform-preboot script for cboot platforms that performs a bootcount check. Signed-off-by: Matt Madison <[email protected]>
* tegra-boot-tools-nvbootctrl added to provide the nvbootctrl compatibility script for handling downgrades to images that assume that the NVIDIA bootloader tools are in use. * tegra-boot-tools-lateboot added for marking boots successful. Signed-off-by: Matt Madison <[email protected]>
which uses tegra-boot-tools for getting the current boot slot. Signed-off-by: Matt Madison <[email protected]>
to install an extra copy of the fw_env.config file so we can detect an environment location change during an udpate. Signed-off-by: Matt Madison <[email protected]>
…boot to handle U-Boot environment location changes and allow for automatic bootloader updates for Nano-eMMC targets from older versions where the U-Boot environment clashed with the VER partitions. Signed-off-by: Matt Madison <[email protected]>
to mount the new rootfs read-only, as is now done with the scripts in meta-mender-tegra. Signed-off-by: Matt Madison <[email protected]>
for cboot platforms, to make /run available in the chroot. Signed-off-by: Matt Madison <[email protected]>
- Add verbose messaging about VER partitions being corrupted - Continues the update without exiting, added a message about that Signed-off-by: Atharva Nandanwar <[email protected]>
Signed-off-by: Atharva Nandanwar <[email protected]>
atharvanan1
force-pushed
the
dunfell-l4t-r32.4.3+ota-components
branch
from
c9e46b3 to
e46fe08
Compare
Signed-off-by: Atharva Nandanwar <[email protected]>
|
Tested this today with latest tegra-boot-tools changes. Updated the README with a notice for change in boot-tools. |
|
@atharvanan1 LGTM now.... is there anything else that needs to be checked before I merge? |
|
Let's sit on it for a day, I'll think if I'm missing anything. |
|
Here's an overview of what's changed and tested:
Let me know if I should test any other devices (SD Card based devices), however as I understand they should work as their emmc counterparts work. |
|
Thanks for all the work on this, @atharvanan1 ! |
|
Thanks for helping me on this! @madisongh |
Hey,
I tried to integrate the OTA setup highlighted here - https://github.com/OE4T/meta-tegra/wiki/Over-the-air-reflashing-process
Setup Information:
Pre-update Software built from - https://github.com/madisongh/test-distro/tree/warrior
sysinstall-upgrader-initramfswith modifiedtegra-sysinstall-toolsas per https://github.com/BoulderAI/tegra-sysinstall/tree/add-UDA-supportUpdate Software built from - this PR
meta-tegra-support/classes/image_types_tegra.bbclassto replace themeta-tegra/classes/image_types_tegra.bbclass-> this is done to change the PPTSIZE to 16896 as required by the OTA process.recipes-devtoolsfolder from https://github.com/madisongh/tegra-test-distro/tree/dunfell-l4t-r32.5.0/layers/meta-testdistro/recipes-devtoolspackgegroup-demo-base.bbappendfrom https://github.com/madisongh/tegra-test-distro/blob/dunfell-l4t-r32.5.0/layers/meta-testdistro/recipes-testdistro/packagegroups/packagegroup-demo-base.bbappendsysinstall-partition-layoutpipes outpartition_tablefile asBelow is the log from after partitioning gpt disk
Upon reboot the bootloader fails with this: