Proposal for simplifying repetitive tasks of the development, test, lint, format, build and flash processes

[qlrd]

When reading the official documentation over and over again (initially, with the intention of translation), I noticed that some tasks (development, build, lint, update, test, etc) require, on an extended time scale, a considerable amount of repetition of the commands.

I think this could be simplified, through a Makefile.

I didn't open a PR to not affect some improvements being made by @jreesun and @odudex , but I share a gist that, apparently, simplified the tasks.

Some of the simplified tasks:

• make update: replacing git pull origin main && git submodule update --init --recursive
• make install: replacing poetry install and some optional dependencies (in this sense, a proposal to modify pyproject.toml to)
• make test: replacing poetry run pytest --cache-clear --cov src/krux --cov-report html ./tests
• make test/verbose: replacing poetry run pytest --cache-clear --cov src/krux --cov-report html --show-capture all --capture tee-sys -r A ./tests
• make build/sign: replace some tasks to installing from the source:
  • vagrant up
  • vagrant ssh -c 'cd /vagrant; ./krux generate-keypair'
  • vagrant ssh -c 'cd /vagrant; ./krux pem-to-pubkey pubkey.pem'
  • copy the hex pubkey
  • replace in src/krux/metada.py file
• make build/maixpy_m5stickV: replace vagrant ssh -c 'cd /vagrant; ./krux build maixpy_m5stickv'
• make build/maixpy_amigo_ips: replace vagrant ssh -c 'cd /vagrant; ./krux build maixpy_amigo_ips'
• make simulator: run and logs simulation process

To see all options, type make help

I also prupose a change on simulator/run-all-sequences.sh, with the characteristic of reducing the console output and allowing the writing of log files during simulation.

[ghost]

Great idea! I think that would be extremely useful and help to organize all the commands.

One important issue, however: I want people on Windows, Mac, and Linux to all be able to pull down the code, build from source, and flash to their devices. That is a large reason for why the project uses Vagrant in the first place. As it is now, if you're on a vanilla Windows platform, all you need to do is download vagrant and git and you'll be good to go. With the introduction of a Makefile, this could complicate the process for Windows users.

Is there a way to make this easy to run on Windows without requiring a lot of additional effort? Ideally, supporting it would require at most one more .exe to install.

[qlrd]

One important issue, however: I want people on Windows, Mac, and Linux to all be able to pull down the code, build from source, and flash to their devices.

A very important issue. In theory, Mac users can run the make commands smoothly since OSX is a Unix Like. At most they will need to install something through brew. But i'm not a mac user...

Is there a way to make this easy to run on Windows without requiring a lot of additional effort? Ideally, supporting it would require at most one more .exe to install

I believe there are close possibilities:

• Replace a Makefile to:
  • a Rakefile (taking advantage that vagrant runs in ruby environment, and it is possible to detect OS );
  • a similar python tool, like pypyr
• Usage of windows chocolatey (see here);
• Usage of nmake with a custom Makefile.win (see here)
• Usage of cygwin(see here)
• Usage of NSIS (see here)

EDIT:
An installer build here

[odudex]

Isn't it possible to just login Vagrant, than run all the commands as you were on Linux?

[qlrd]

Taking the first short steps. I think I found a node.js framework -- electron-builder -- that makes it possible to create programs for Mac, Linux and Windows using a separate instance of chrome.

I think now it's integrating the build process (vagrant, etc.) and installation steps in .js scripts

Isn't it possible to just login Vagrant, than run all the commands as you were on Linux?

Yes, its possible. But I'm kind of lazy person (:face_with_head_bandage: ) and I like to type few commands when they are repetitive.

[ghost]

The krux shell script was written so that it could be run either on a linux machine directly or inside Vagrant.

Because I was trying to avoid platform-specific instructions (and troubleshooting), the website documentation for installing from source just says to run the krux script inside vagrant with the vagrant ssh -c 'cd /vagrant; ./krux prefix before all commands.

As for the development of Krux, it's less important that it be cross-platform. So, the Makefile could be written assuming it will be run on linux or inside Vagrant just like the krux script is, and the documentation for installing from source on the website could remain as it is. But the development instructions on github could be updated to refer to the new Makefile commands.

Would that be easiest? That may have been what @odudex was suggesting.

[ghost]

A GUI installer based on Electron would be pretty neat if it could automate the installation commands.

Something like this:

• a Makefile to make development easier and faster
• a cross-platform electron GUI installer to make installation easier and faster (for less technical users)

Is that what you're proposing?

[qlrd]

As for the development of Krux, it's less important that it be cross-platform. So, the Makefile could be written assuming it will be run on linux or inside Vagrant just like the krux script is, and the documentation for installing from source on the website could remain as it is. But the development instructions on github could be updated to refer to the new Makefile commands.

Would that be easiest? That may have been what @odudex was suggesting.

I think it will be.

Something like this:

• a Makefile to make development easier and faster
• a cross-platform electron GUI installer to make installation easier and faster (for less technical users)

Is that what you're proposing?

Sorry for not expressing myself properly before. But yes, that's exactly what I wanted to propose. In this sense, a GUI could (in a ideal and a similar way to what Raspiberry Pi Imager does):

• Give the option to write pre-built firmwares to a microSD;
• Give the option to flash the firmware on a supported device;
• Give the option to install from source code. In that case, the Electron app would also do:
  • Download and extraction of .tar.gz or .zip;
  • sign, build (and other) processes described in the documentation, via the child_process.spawn method
    • In case of windows, run vagrant through child processes

[ghost]

That sounds great to me! To keep things organized, can you create two issues: one for the Makefile with a small explanation about it (similar to your original post here), and one for the installer? That way we can track their development separately (ideally in two separate PRs)

Going to convert this issue into a discussion

[qlrd]

Created a repo to GUI https://github.com/qlrd/krux-installer. Work in Progress

[qlrd]

Well, it's been a while since the idea came.

At the moment the GUI, in AppImage format, runs the firmware flash without any problems, at least with me.

However I don't know if this is the case for executables on Windows or MacOS, although the builds are running smoothly

A v0.0.1-alpha-1 release is likely on the way, but I think it would be interesting if some fellow developers on these OS could run the builds on their respective machines and report specific issues.

Below, some screenshots:

Main Menu

Select firmware

Select Action

Select Version

Official version verification after download...*

After all options selected

Some Warning before flash

Flashing

Flashing results

[ghost]

Awesome work! I've been tracking your progress the past few months and it's been cool to see everything coming together.

Here's some feedback if you're interested.

For the Main Menu:

1. I think the first option should be "Select Device" instead of "Select Firmware"
2. Instead of having to "Select Action", why not add two buttons at the bottom: one for writing to SD and another for flashing. Then you could get rid of the "Execute" button and make those two buttons the last step instead.
   So the new layout could be simplified to:
   | [Select Device] |
   | [Select Firmware] |
   | [Write firmware to microSD] [Flash firmware to device] |

The orange background doesn't look good to me. I think I know why you picked that particular color though ;) But my preference would be to make the background black and the "Krux Installer" and logo white instead so the installer matches the site theme (https://selfcustody.github.io/krux/) and existing logo.

[qlrd]

Nice Jeff.

I will work on these suggestions. At the time, Write firmware to microSD proved more challenging (to adapt on multiplatforms) than flash, due to the assembly methods of each system. I will try, first:

| [Select Device] |
| [Select Firmware] |
| [Flash firmware to device] |

The orange background doesn't look good to me. I think I know why you picked that particular color though ;) But my preference would be to make the background black and the "Krux Installer" and logo white instead so the installer matches the site theme

No problem. I will change that too.

Doubt: You think its better to (I) give all versions options; (II) give only the latest official release and test version; (III) only the official release?

[ghost]

I think option one (I) is fine, but perhaps breaking them up into two separate dropdowns: one for official releases and another for odude'x experimental builds. You could sort each list in descending order so the most recent is on top/pre-selected

[qlrd]

Hi,

I made some modifications that @jreesun asked for (buttons and style) and some modifications that @odudex made (check if a resource exists before download it).

Maybe a pre-v0.0.1-alpha-1 ( 😵‍💫 ) can be checked before releasing it for linux developers?

An .AppImage for linux can be found here as a build artifact (at the end of page).

[qlrd]

Now with both Linux (.AppImage) and Windows (NSIS setup Installer with .exe) implementations for the installer:

https://github.com/qlrd/krux-installer/releases/tag/v0.0.1-alpha-2

Opinions about an additional step on windows:

qlrd/krux-installer#2

[qlrd]

Since I was invited to join the selfcustody organization, I thought whether it would be better to put the krux-installer source code in the organization (selfcustody/krux-installer) instead of my personal account (qlrd/krux-installer).

What do you think?

[ghost]

Sounds good to me! Do you have permissions to do that or do I need to give you them?

[qlrd]

dunno.. I will check that

[qlrd]

I think that was made with success. I will try to make a commit.

https://github.com/selfcustody/krux-installer

[qlrd]

@jreesun,

I made some updates on code (sorry for taking so long) on a branch called linux-appimage-wdio.

(more changes i made on this pseudo-mirrored repo)

Since the source is, now, part of selfcustody organization, to merge the linux-appimage-wdio's branch changes on main branch, its needed some reviews on this pull-request (i think its more transparent to show them as a public process than doing them with a local merge, commit and push).

When possible, could you do a review? So it will be possible to publish a new release.