Continuous Integration / Continuous Delivery woes #268
Comments
|
Not too important, but for the sake of interest, the history of the CI was as follows. We initially built packages on a buildbot in my shop. The ARM builds were painfully slow because they were running in an ARM chroot on x86 hardware in emulated mode. Later, we moved CI to Travis. ARM builds took longer than the 50 minute limit on the 2-CPU Travis builders, so somebody hacked the build system to split the POSIX, Xenomai and RT-PREEMPT builds into separate builds. That worked to get everything built, except it ended up causing another problem. I think you're right that @ArcEye set up the Jenkins server and recombined the ARM builds in order to finally solve this problem: no time limits on the combined ARM builds. Somewhere in there, I went in and got proper cross-building to work, and later set up the Docker containers we now use; this was an alternate solution, since (combined) ARM build times were reduced to 10 minutes or so on Travis. The GitHub Actions and Cloudsmith you're looking at weren't around when we were working on this, so maybe there's an easy solution for package distribution in there. Packagecloud's problem is the draconian data storage and transfer limits made it impractical to use for public package distribution. If I were to try again today, and couldn't find any other simple, inexpensive solution like Packagecloud's but without limits, I would use Travis or any other CI system to upload built packages to a VPS running Docker images for receiving new packages and publishing them in a repo. I probably have scripts around somewhere that do this, or I'd likely try to integrate aptly. Let me know if you need help, but it sounds like you have a more current understanding of the latest tools available today; things have changed quite a lot in the last several years. |
|
Thank you for the historical details about build system, @zultron. I guess this video is related to this subject, right? Do you have any idea how much traffic the Machinekit project needs? Packagecloud gives 25 GB of storage and 250 GB traffic (currently the watermark is at about 10 GB of stored packages and the traffic is negligible, as it wasn't really used). Cloudsmith gives 250 GB of storage and 1 TB traffic (they say these limits can be upped based on needs, but hard to say from the get go). I asked on the Debian IRC channel on Freenode if anybody doesn't know of any FOSS project which uses the Cloudsmith service for Then there is the Oracle Cloud Always Free Tier with 10-100 GB storage and 10 TB transfer limits. And I was also considering using the GitLab Pages for repository hosting. It is possible to maintain this by hand (or this) or probably use the Aptly. They have 10 GB limit on repository and unlimited traffic (yeah, riiiight...). We will probably have to solve this for the helper repository But in general, I would like for every piece of infrastructure to NOT BE dependent on any specific Machinekit developer or member. I have created pull request #269 which eats the Dovetail-Automata/mk-cross-builder repository and closes the #195 issue. It would be nice if you could eyeball it. Later, I will take a knife and will go gut your other repository for goodies. To make it work, I had to add |
|
Bullseye is a problem. There are missing dependencies: Given that the same dependencies are in the Zultron/mk-cross-builder repository and there are built Docker images in DockerHUB, they must have deleted them in the last 5 months. I am going to create a pull request with Bullseye cross builder, as this has nothing to do with it. But, bloody Python. |
|
There is some related discussion to this issue on Machinekit Google groups Forum. |
|
I have just downloaded (I don't understand Python and never liked it.) |
|
Update about the Github Actions logic flow for those who are interested: Now that the Machinekit-HAL integrates the 'mk-cross-builder' functionality, the building of new images has to be integrated into general Machinekit-HAL CI/CD flow. That means the system has to decide when just download the image from Docker repository and when to build new images. I think that the most sensible solution is to build new packages in moment when the pull request event or push event has commits changing files from which the Docker images are build, so There lays the first problem - Github Packages is free solution for Open-Source projects which uses the same login credentials as other Github services, so access from Github Actions is straightforward. But you cannot delete public packages, once it is there, it is there. And to download the package - even public one - you have to log in into the repository, and not just by your username and password, but you have to create token with So after deciding if to build or pull, in case of building, there will be jobs which will build the builder images, one per type. The build is no problem. It takes on average up to ten minutes, given that these workers run in parallel, it is 10 minutes. Problem is how to share these Docker images with another workers which will run the testing and package building. One way is to upload them to Docker repository, that works very well. However, these images are not yet proven and you cannot delete them afterwards. So that is no good. Another way is to upload them as an artifact to GitHub storage and from other jobs then download them. And there lies a problem. GitHub says that for Open-Source, there is unlimited storage, in reality 'unlimited' means 5 GB per file and 50 GB in total (from other users testing). I was not able to upload the Docker saved images bigger than around 0.5 GB, much less the 2 GB tars. (It will upload the first five, then error out on all else.) The only way how I was able to get them to upload is by compressing them with Fortunately the downloading and loading the artifact image into Docker in another job is fairly quick (2 minutes tops). Minor issue is, that it sometimes fails. But that may be because I am currently using the This takes about an hour of parallel running (maximum is 20 workers). Not great but doable. Another way how to do it is to not build the Docker images beforehand in job on which are other jobs waiting, but build them in each job separately. It will mean that all 14 jobs will build the Docker image and run the tests/build packages. Then these images will be thrown away and another job will rebuild the 11 builder images and push them to Github Packages repository. (Cannot be done in the previous job given that all jobs in given matrix must finish with success.) Problem with this solution is obvious - testing and building will be done with different images and images stored will be different too. Given the time differences there could be upstream changes which would have chance to introduce subtle bugs. I personally don't know which solution I should use. Maybe I am leaning more towards the one with crazy wait times on |
|
Sounds like @cloudsmith-io would be a lot easier, plus it supports debian and redhat natively. ;) It's worth remembering that Docker isn't package management, so building the packages first, then containerising that, is usually a much slicker, flexible and more efficient solution; plus it lets native users install natively. If you need any help with that, just let us know (I work for Cloudsmith). |
That sounds like more than it used to be. Also be careful: If stored files ever exceed 25 GB, which will happen after many CI builds but old ones are not pruned, they will suspend further uploads until the next billing cycle, even if you go in after the fact and clean up space. I emailed them about that, and they said it wasn't a bug, but a feature.
That became my goal after realizing my mistake hosting CI in my shop, but I never quite realized it. The $5/month VPS with Docker images to manage the service was the best I ever figured out, but again, there are new services out there today and you seem pretty on top of it. One other that could've worked was Suse's OBS. IIRC, since the repos migrated to deb.mk.io, they started supporting ARM-arch Debian builds, something that wasn't there before. It might be possible to build on Travis or other CI and upload results to a repo there.
That looks like a good way to pull the images in. I originally anticipated it becoming a new repo under the MK GH org, but integrating with the mk-hal repo sounds just as good, without having thought about it too hard.
Have you checked that the "hanky panky" is still even necessary? I think there was a problem in older gcc versions where Ultimately, someday Debian packaging multi-arch support might work properly, and then the need for ALL of that ugliness will go away. I bet that might never happen with TCL/TK, but maybe MK will be TCL/TK-free someday. I also remember cython being one of the blockers. When that's all resolved, cross-building Debian packages will be possible with automatic dependency resolution using standard tools and simple scripts out of Docker. Maybe I'll have to keep smoking my pipe several more years first, though. |
Because APT installing machinekit .debs can't pull in those dependencies unless they're also referenced in an APT repository. This is a packaging problem, not a Python problem. It used to be the same problem with e.g.
If this issue is turning into an arena for a battle about favorite programming languages, then my gambit is, "I like Python and I'm happy that many other projects I'm involved with, like ROS and RedHat, make extensive use of it." |
I was thinking about getting the ZeroMQ packages from there. But it seems that not all architectures currently supported in Machinekit project are available there. I will add it into to investigate roster. Frankly, the upload can be pretty much anywhere and everywhere.
Actually, it was based on your comment. It just made more sense to me to do it this way given that you need to somehow upgrade the images when new Debian package dependencies are added or updated as it is a build input. This way it will be part of normal Machinekit-HAL testing/package build flow.
No, I discovered that commit 05bad16 needs reverting otherwise build But when it is time to purge Jessie parts, that's the time to look into it more, I think.
There is (was) developer - @Shaeto -in legacy Machinekit/Machinekit repository interested in making Fedora Of course, maybe with |
This issue is about what is specified in the title and first introductory post. I am a programmer, I do solve problems. That being said, solving problems in areas I care about means that I understand the particularities of given territory, and so I am able to produce much better solution, i.e. elegant code. So far, I have been able to come up with following ideas:
There is python-zmq available from the official ZeroMQ Open Build Service account. But no arm builds. The yapps2-runtime is build from yapps2 source, from which yapps2 package is build. Then there is python3-pyftpslib version of python 2 But as you do like Python and have much bigger practical experience with it, I will gladly defer it to you. I just want Machinekit-HAL on Bullseye. In this light, I would postpone the discussion about favourite languages and the associated altercation to later date. Looking through the calendar, my preference is never. How is that working for you? (We will have to consult with the |
|
@lskillen,
Actually, the Docker images are only used for the The end result is distributed as a If you need any help with that, just let us know (I work for Cloudsmith). I will take you on the offer:
|
The MK project has been known to do that in the past. The Jessie repo has (had?) kernel, ZMQ-related and other packages unavailable in the upstream distro. The effort hand-roll a package for a 3rd-party APT repo turns out to an annoying, but often trivial (using existing packaging sources) and one-time pain.
Using
Looks like a pretty cool project for someone (else) to dig into.
These are a potential source of packaging to help with option (1).
Let's go with option (1) and publish
Perfect! 100% on the same page, calendar page or otherwise. On the other hand, I can find time on my calendar soon to help build packages for missing deps, a trivial task. If you had a list of exactly which packages are missing from Bullseye, that would save me half the work. If the Docker CI images are still useful, @ArcEye submitted a PR at Dovetail-Automata/mk-cross-builder#8 to support Bullseye, which I'm feeling very embarrassed about having dropped right now. I think the non-trivial part of the APT packaging equation is still going to be building the EMC application. If help is needed with that, I'll volunteer once MK-HAL is packaged up and online. Part of my LCNC-EMC port was to work out many of the fundamental build system issues left from the HAL/CNC repo split. |
|
@cerna Just noticed your reply now. :-)
Fantastic; we have similar techniques for our own internal environments at Cloudsmith.
Yup! Use it if you can. Any suggestions or code enhancements are also welcome.
We're more than happy for you to upload dependencies, assuming that you utilise the repository for distribution of your primary artefacts as well; i.e. it can't be for dependencies only. Other than that, the only real requirement is a (polite) link back to Cloudsmith. However, you're free to organise your pipeline into multiple repositories; e.g. you can create a repository just for the dependencies, as long as you have another repository for your primary artefacts. That would keep your outputs separate from the artefacts. In the future we'll help organise this for automatically by labelling dependencies explicitly. |
I think all of these issues are connected to Python 2 |
|
@lskillen, Turns out I am a moron, I read what I have written, and 🤦♂️. What I actually meant to ask:
In other words, I left out the most important part of the question. 🤦♂️
Sure, that's given.
So if I created new repository |
Yes, Docker CI images are useful. I am not going to fundamentally change something which is currently working and I don't have that deep knowledge about it to feel comfortable to do deep cuts. I might do some hacking. I actually picked it up in #270 and it is now part of Machinekit-HAL proper. The comment I described how I build the package use this work.
I haven't looked into it yet. I somehow hoped that there will be CMAKE based build flow first, but that's probably not going to be the case. But I will get the Machinekit-HAL packaging up and running with publishing to @cloudsmith-io first and foremost. Taking the |
We have one that's been forked from a user of ours (who's OK with us taking ownership): It's incredibly spartan at the moment, but we'll almost certainly be tidying this up and publishing it to the GitHub marketplace as well. PRs welcome!
I did wonder!
Yes, yes and yes. :-) |
|
I have looked at the Debian packages produced from the original Jenkins builder and discovered that these were signed by the And it got me thinking that this is probably a very good idea - to help differentiate that the package was built from official Machinekit/Machinekit-HAL code-base and not from some fork. (Or the opposite, that the package is from specific fork.) But does anybody (@luminize, @zultron, @cdsteinkuehler) have the original Machinekit Signer primary key with which a new sub key could be created for this job? |
|
I have been playing with I don't think that should be a problem, but it is. As I cannot get This fails on If anybody knows how to solve this, I am all ears. |
|
I started taking a look at the Bullseye deps. I suspect it'll take a full python3 upgrade to support Bullseye, and that will probably take some real rework. Some issues I already find:
I started a branch, but only got as far as fixing the Docker image build, protobuf python bindings and comp generator before stopping. (This certainly deserves a new issue to further discuss python3 porting.) |
Well, Python2 was EOL for few months and the situation will be worsening with packages going missing left and right, so it is probably high time to make the switch. But I just hope (given that the discussion about Python 3 is pretty old: #114 and machinekit/machinekit#563) it will not turn into rabbit hole and take both Bullseye and Python 3 projects with it to never land. (As I will not be very useful for this endeavour.)
Looking at
Jessie will be slashing LTS in two months. I am not sure it is still worth the effort (given that this will take few weeks) to have it supported for 14 days before I turn the builder off. I am just saying. Jessie had a good run, but it's time to let it go.
LinuxCNC's @rene-dev recently started Python3 work (at least he said so on IRC #linuxcnc-devel). Maybe it's worth to have a looksie into his work and port parts which are still the same in both LinuxCNC and Machinekit-HAL? |
Bullseye is going to be a real endeavor to bring up because of the python3 issue. As I said offline, I'd like to coordinate with the LCNC folks over this, since any changes to support python3 on the HAL side need to be mirrored on the EMC side, and going forward, building LCNC EMC against MK HAL is still the most sustainable plan for making packages available for MK HAL and its top application.
I'm definitely in favor of factoring out anything TCL/TK, but I do think tools like
No disagreement from here. I'd love to jettison some of the ugly stuff we have in the packaging and CI configuration needed to support Jessie. Last I heard, though, @dkhughes has a lot of BBBs out in the field still running Jessie, which is why instead of ditching it back then, we replaced the unmaintained Emdebian tool chain with Linaro.
LinuxCNC/linuxcnc#403 pretty much lays out what's already been done. @gibsonc appears to have done the heavy lifting porting the C-language bindings I started bumping up against a few hours into my naive attempt referenced above (like Still, I'm most interested in getting packages for released distros online first. If you'd like to point me to where the project is with that and how I can help, I'll spend a few days hammering on it. |
And I am up there with you. However, I think it should be in separate repository/separate packages and solved as the original idea dictates. (You will find, many of my ideas are pretty in line with the original Haberler's ones.) Ring buffer from real-time side to the transshipment point, where the ring buffer frame will be sent on ZeroMQ socket to display application. That way it can run on the same machine as Machinekit-HAL or on notebook next to the device (like service guys are wont to do). I have been thinking about doing something like it based on WPF/Noesis GUI (first for HAL meter) but it is on low burner so far for me. (More important things need to be done.)
That was a year ago, no? I am hoping since then @dkhughes twisted his customer's hands and is now mostly running on distribution which will be supported little longer. Otherwise, they will not be able to upgrade, of course if he won't support it on the side (or just don't upgrade Machinekit installation, it will be the same thing [pretty much] without security upgrades).
I think it is mostly done. I just need to decide which packages to upload to Cloudsmith. After @luminize merges the #274 which implements the Docker image caching and auto-build, I will just redo some less important parts. Like transform the What I would like to see is the #246, #250 and #200 done - even if the Machinekit-CNC will lag behind. I know @kinsamanka wanted to first wait on it, but it dragged on too long. It shouldn't be such problem with Machinekit-CNC package building and it has to be done. Better for it to hurt little bit from start than wait another year for it. So if you could take look on it, I would be glad. (Not only it would mean better IDE support for Machinekit-HAL, but it would be great for other purposes too [I am in mostly for the IDE support 😃]). |
|
OK, I have reimplemented the build functionality in basthon 3: commits and surprisingly it seems all to work (package building, image building and Machinekit-HAL testing). Hopefully it will be cleaner for potential developers and I haven't just spend time reinventing a wheel. I just need to patch the Github Actions and then open pull request. (Even it is definitely not perfect, it is good enough and while trying to have nice linear history in git, I completely trashed it while rebasing multiple times. So I am little tired of the endless history rewriting. I can change what needs to change later.) And it looks like Github changed its GrapgQL Packages API, so the current Github Actions workflow will fail. (It was preview and now it should be integrated into the proper specification.) Not sure how long it hasn't been working, but nobody complained, so hopefully not that long. |
|
Pull request #288 removed the Now onto implementing the Travis CI And mainly actually building and packaging EMCApplication. BTW, good to know that |
|
@cerna Sure, albeit can you a bit more low level verbose state what you need to have tested and how ? |
|
@the-snowwhite, Well, the code of the application itself was not changed, all that was changed was few minor build recipes and the way how packages are build. When running RIP What I need to know - in a nutshell - is if you can see some degradation when running the newly built packages in your standard usage cases in comparison to the old ones. I have looked at the binaries, and they all have the correct ELF header, but I could miss some. Also, there was a change in how the Python binaries are named. So, what I would appreciate is:
|
|
@cerna sorry for my late reply, for some reason I got no notification of your response. It would be nice to be able to run these setups without machinikit-cnc or l-cnc |
|
@cerna Culprit is this run file requires machinekit (executable): |
|
@the-snowwhite, Yes, the cut between Machinekit-HAL and Machinekit-CNC and now what will become the EMCApplication is not that clean (pretty bloody actually) and there is definitely work to be done to clean it up. I personally consider the split into smaller repositories as one of the in the set of best decisions (too bad it wasn't done sooner, given that the talk about it goes to the beginning) and wouldn't want for Machinekit-HAL to start growing into Machinekit [original] like repository. That being said, I think that the CNC specific Python modules should go into own git repository with separate package distribution route (while creating clean tree dependency structure between all parts). Basically push out all CNC stuff dependent on other specific CNC parts from Machinekit-HAL into own logical homes. I quite like what @kinsamanka started in #250 in https://github.com/machinekit/machinekit-hal/blob/df5e884d31f8e2668ad80c1c2be66028a64cc3b4/debian/control.in - putting things into own logical packages for distribution. (I have been getting my CMAKE knowledge into hot, useful state and so far I am liking the modern approach a much better than I last remembered, so transforming the Machinekit-HAL into CMAKE is next logical step on the line But until that happens, it will be kinda unusable for these kinds of tasks. |
|
@zultron in #293 was bitten by problem stemming from Docker image caching in Github Actins CI - it can happen that Docker image built for one branch can be used for test build workflow of another branch. And because both branches can have fundamental differences, the run fails even though it should not. I realized this problem at the time I was implementing it but given that for my work style it would not cause a problem, I decided to postpone solving it to later date. So, now I had to think about it, had an aha moment and came up with following design flow (development of pull request in progress):
Hopefully, this workflow will be a lot more immune to cross-branch issues and also solve the issue with force pushing. BTW, the renaming in #295 left out the Docker image upload jobs and Debian packages upload job, so that should also be shortened to be in-line with the change. |
|
I'm looking at all the amazing CI infrastructure @cerna has put together here to help me get off the ground more quickly with #297, building the EtherLab driver for the same OS and architecture variants that the CI system builds for Machinekit. After a whole lot of copy'n'paste, I'm starting to wish that some of that work was pulled out of the Machinekit-HAL repo and made independent, since so much of it is reusable, such as the Docker base images, Python container build scripts, JSON distro settings and entrypoint script; that is, almost all of it. I don't want to expand the scope of this issue, so maybe this goes in a new one, but this would be very welcome if the project ends up with many separate repos all building packages in Docker for the same OS+arch matrix. |
I'm sure you have this handled already, but just in case, for another project, I needed to do something similar. We wanted to know if a checked out revision matched a Docker image, and pull a new one if not. The scheme I devised was to find a way to generate a reproducible hash of the files used to build the Docker image. The hash would only change when the image input files would change, and never otherwise. This hash was then built into the image tag, although it could have been put into an image label as well, say So for this application, you'd do something similar: compute the hash from the repo files, query the Docker registry for an image with a matching I can produce the hash generation commands if needed. They're not rocket science, but there are a few gotchas we had to address before they were 100% reliable. |
Here's an example of what they do in a bunch of ROS-Industrial repos. Projects that use it simply check out the repo into their source tree in CI and run the scripts. This is pretty versatile, and works with a bunch of CI systems, a nice touch. |
I've spent quite some hours now with the GH Actions CI config while working on #297, and while not done yet, at least the basic Docker image and package build flows work. One of the things I've done is try to remove bits specific to the MK-HAL repository and make them generic and configurable, in hopes of doing something like the above, separating out a shared CI config. As I know very well from personal experience, CI configurations for the project are necessarily very complex. @cerna has done a fantastic job building up this new system, as well as vastly simplifying the Docker builders we used to use (which were nasty and hairy, and which I wrote!). Starting from the MK-HAL configuration for #297 has saved me unknown dozens of hours, since I could copy and paste 90% and make it work with only minor changes. I'm really pleased with it, so please keep that in mind even as I propose improvements below! As it is now, there is a LOT of logic built into the workflow file in the form of embedded shell scripts. It's likely my own deficiency that these turn out to be quite delicate for me, and going through repeated iterations of pushing changes to GH, waiting several for the Actions run, and going back to fix problems has been frustrating and time-consuming. If the CI scripts were able to run stand-alone in a local dev environment, these iterations could be drastically shortened by being able to run individual steps independently, without having to queue up the entire workflow in GH Actions. The basic workflow could stay the same, with the GH Actions workflow keeping the same general structure and maintaining the same use of output parameters for carrying configuration data between jobs, encrypted secrets for access credentials, Docker registries for caching images, etc. There are already Python scripts used to build the container and packages, so it makes sense to convert workflow file shell script logic into Python; then, the differences between running the workflow in a local dev environment vs. the GH Actions environment could be encapsulated using object-oriented programming. In the same way, the workflow could be adapted to other CI systems, should the need arise, and of course the workflow can be shared between MK-HAL, the EtherLab Master and Linuxcnc-EtherCAT HAL driver repositories (and potentially the MK-EMC, though it already has a CI configuration), and improvements will benefit all repos. Does the problem description make sense, and does the proposal sound reasonable? |
|
The truth to be told is that I wasn't thinking about using this outside Machinekit-HAL. Of course, I am not saying that it is not possible. It is possible. But some analysis of common requirements and processes across all repositories or projects where it could be used will be needed.
This looks like a warehouse for commonly used scripts. Well, one can do something similar to this in Github Actions with own actions. These can be written in Typescript, or can be a Docker container (and then use practically any language possible) or newly can be a script. The best part - and really the only part which make it specific - is that the Input/Output is already solved. Drone has something similar with its modules, but not so versatile, I would say. I don't know about Travis, I don't think so as the whole ecosystem is not pluggable and modular (I think that they will try to introduce similar concepts to stay in the game and relevant, but it surely won't be overnight). And - at least in documentation portions I have read so far - talk about simple bash scripts. (But at least they introduced the workspaces - something like immutable shared drive that following jobs can use.) The Python scripts are nothing to write about, but they sure could be abstracted to some classes and then the repository would have only some basic settings Python object (or configuration JSON) which would specify the labels or arguments send to the Docker builder. (I am also afraid if this would not start to reimplement some already existing project - isn't there something readily available and usable already?) Problem also is that the Drone and Travis CI are both somewhat working but actually in pretty terrible conditions. Turns out the GitHub Actions are pretty advanced (even through I originally thought otherwise). And to enable the same functionality on both Drone and Travis, one has to use local git hooks managers with (probably Dockerized) running environment - because only Github Actions (and Azure, as far as I know) allow to dynamically create and alter the recipes at runtime (up to a point, obviously). So I put is at back-burner (because for now it is good enough and there is the #200 and so). Nice thing about that repository - That thing has more pull requests than the whole Machinekit-HAL. If Machinekit implemented something similar, should that presume that the Machinekit-HAL would be integral for all, in other words that it would be building and testing targets dependent on the Machinekit-HAL or do you want more universal system/approach which would not be dependent on Machinekit in any way? Because this looks like it is targeted at modules. |
|
Travis CI integration in #299 is behaving oddly - for example job 108.5 failed, the log says that it failed, but the whole job is green, i.e. it passed. Basically first four jobs, which are the build RIP and run runtests ones, failed (as they should), but the Debian packages building ones all passed - even though they should have failed too. This means that some part of the bash script is eating up the error code and the Travis runner gets 0 in cases where it should have got something else. |
That's because the script needs to either https://github.com/rene-dev/machinekit-hal/blob/python3/.travis.yml#L144-L163 |
Yup, I did it at the same (similar) time. I am not surprised I fucked both the same way. I will create a PR later today. |
Due to problem described in machinekit#268 and machinekit#292, bash commands run in CI scripts can fail with non-zero exit code, but the following command will hide this from the CI runner by itself exiting with zero. Two basic ways of solving this issue exists: add `set -e`at the top of the script block or pairing individual commands with && operator. Given that the pevious work (machinekit#292) uses the operator solution, this one uses it too.
Due to problem described in machinekit#268 and machinekit#292, bash commands run in CI scripts can fail with non-zero exit code, but the following command will hide this from the CI runner by itself exiting with zero. Two basic ways of solving this issue exists: add `set -e`at the top of the script block or pairing individual commands with && operator. Given that the pevious work (machinekit#292) uses the operator solution, this one uses it too.
|
@lskillen, The simplest way how to reproduce this (most important information is at the end): |
For some OS and distribution versions, the automatic Cloudsmith's script has a problem and in the end no repositories are installed. Fortunately, the script allows user to explicitly specify which repositories (for which OS) he wants to install.
|
Looking again at the Travis CI configuration (which is now in very precocious state and in need of rework), I started looking in the remote API, specifically starting a new build by sending some data to remote endpoint as described at Triggering builds documentation. In practical terms, it would mean two "job" or "builds" per every Travis CI supports (I don't know for how long, but they are calling it still beta version, so probably not that long) Importing Shared Build Configuration, which is something to investigate, if Machinekit organization will go with The structure of Also, what must be decided is if Machinekit wants to test 32bit versions on 64bit platform (where possible without the use of QEMU). That is testing i386 on amd64 machines and armhf on most amd64 servers (not all arm64v8 processors suppots the arm32 instruction set). Because that will also need some form of representation in |
|
Well, Machinekit as of now has about half of the trial minutes left. Too bad that the Graviton2 based VM were quite good and there is no alternative for it as of now. I am going to limit the Travis builds to only testing of the arm64 platform to preserve the minutes. |
Tracking progress:
dpkg-sigCMOCKAtestsamd64,armhfandarm64runtests in Drone Cloud CIyamlscript into saner form - JSONScript with autoregeneration by git hookyamlscript into saner formAs you probably know, the Machinekit buildserver (
jenkins.machinekit.io) was turned off. It was done quietly, so I have no idea when exactly it happened. It ran test on pull requests and uploaded debian packages into the Machinekit'sdeb.machinekit.iorepository. This is important for the C4 as without testing nothing can be merged, respective everything has to be merged.So as an emergency measure the Travis build system was turned back on for the Machinekit-HAL repository. In response to discussion with @luminize - where we agreed that some artifact output from CI run would be nice, so users can download the
.debpackages and usedpkg -icommand to install - I implemented simple Github Actions workflow in thequick&dirtystyle as Travis doesn't allow keeping artifacts for later download and it's nice for users to have automated build of packages in theirs forks. Github keeps the build artifacts for 90 days after the fact.Given the volatile situation with
*.machinekit.ioserver, I think that its state should be conserved and package upload should be resumed into the Packagecloud and Cloudsmith repositories (for redundancy). I can start the upload now or when the package build for Machinekit-CNC is ready (currently Machinekit-CNC repository has no CI for testing and package build), but I think that the time for it is after both Machinekit-HAL and Machinekit-CNC can be installed. I also think that it is a time to drop Jessie support (Jessie will be obsolete in 3 months) and make sure that Machinekit-HAL runs and produces packages on Bullseye.If I understand it correctly, the reason behind own Jenkins server was the 50 minuter running limit of Travis CI. Well, situation is vastly different today with everybody's mother giving Open-Source Projects free minutes on their cloud services. So how relevant is for example this issue for the current status of the project? Given that CI/CD rework will have to include solving the #195 issue, there is quite big window for any changes. Including the build Dockerfiles will also hopefully solve the current issues with image availability. (Machinekit-HAL currently uses my own DockerHUB account, as there are some missing images in the DovetailAutomata account, which caused #263.)
CI tools of today are also using the
container-firstapproach, for which the current scriptbuild_with_dockeris poorly suited. This script should be left for home use and the build commands exported to functions that then can be called directly from the container.There is also provider called Drone Cloud which gives Open-Source projects free build minutes directly on
armhfandarm64hardware. This could be used to test Machinekit-HAL not only onamd64as it is now but also on ARMs.Last but not least, there are now
cmockatests in Machinekit-HAL. I am all in for unit testing. However, no tests are actually run now and the build server are not even installing thecmockadebian package. So that will also need to be addressed.The text was updated successfully, but these errors were encountered: