Standardize a strategy for packages that use artifacts downloaded from the internet

[esteve]

There are some packages (mainly in perception), that download artifacts during the build process via CMake. This not only breaks the Debian generation tasks, but forces users who build Autoware to have access to the internet during the process, which is at best incovenient and at worst a security risk.

I thought of several options to avoid this that I'd like to discuss in the next ASWG meeting:

• Option 1: Provide separate ROS packages with the artifacts, so that actual packages only need to the depend on those
• Option 2: Use git-lfs (or similar) to store the artifacts in the Git repositories (we already use git-lfs to store the Debian packages)
• Option 3: Users download the artifacts manually following instructions in the documentation

I had already started working on option 3 (autowarefoundation/autoware.universe#3137), but moreover we need a policy so that new packages do not download artifacts from the internet.

[esteve]

@xmfcx I'd like to discuss this in the next AWSG meeting (2023-07-11)

[xmfcx]

Discussed in: https://github.com/orgs/autowarefoundation/discussions/3653

[esteve]

For reference, here's the relevant section in the Debian policy manual regarding packages that access the internet during the build step:

https://www.debian.org/doc/debian-policy/ch-source.html#main-building-script-debian-rules

This ensures that package builds are reproducible and avoid potential security risks.

[mitsudome-r]

At least there seems to be no objection from TIER IV about removing the download script from CMake.
I also think Option 3 would be a best way to go as discussed in Software WG.

I do have one concern about how we want to set the default path in our example launch files.
They currently points to the relative path within the package. If we download the model files manually or by ansible, should we make an assumption about where to download? (e.g., somewhere like /opt/autoware/sample_ml_models)

[mitsudome-r]

Another option is to pass autoware_model to launch file as an argument to autoware.launch.xml just like in autoware_map.
Pros: User can choose which folder to download the models
Cons: We have to pass down the argument all the way down to the included launch file. (e.g., autoware.launch.xml -> perception.launch.xml -> detection.launch.xml -> camera_lidar_fusion_based_detection.launch.xml)

[xmfcx]

I suggest we pass autoware_data directory.

And we pass down this path to the downstream packages in the launch files.

We would predefine the structure of this directory:

.../autoware_data/map/map1/map.pcd
.../autoware_data/map/map1/map.josm ?
.../autoware_data/models/model1.onnx

Another option from @esteve (I hope I got it right)

.../autoware_data/package_name/map.pcd
.../autoware_data/package_name/map.josm ?
.../autoware_data/package_name/model1.onnx

[evshary]

I prefer @xmfcx proposal. It makes new commer clearer to all the artifacts in Autoware.
Also, maybe using environmental variables to pass the path is another option.