Can I create a deeper directory for packages in autoware.univese?

[KYabuuchi]

I am planning to merge YabLoc into autoware.universe. Please see this discussion for what YabLoc is.

Currently autoware.universe has directories for each component level (e.g. localization/, perception/, etc.) but no deeper hierarchy than that (e.g. localization/pose_estimator, localization/twist_estimator/).
YabLoc is composed of many original packages and placing all of them in universe/localization/ would be complicated.

So, can I create a directory localization/yabloc for clarity? For example, I am considering the following structure.

autoware.universe/
　├ perception/
　├ control/
　├ localization/
　│　├ ekf_localizer/
　│　├ gyro_odometer/
　│　├ ndt_scan_matcher/
　│　└ yabloc/
　│　　　├ particle_filter/
　│　　　├ segment_filter/
　│　　　├ graph_segment/ 
　│　　　├ line_segment_detector/
　│　　　├ graound_estimator/

I heard that autoware.universe does not have deep directories for packages because that was discussed a long time ago and the current structure was adopted. But I cloud not find that discussion. 😢

If additional directories are not created, it would require significant and time-consuming changes to YabLoc in order to simply merge YabLoc into autoware.universe. Of course, if YabLoc needs to be modified, I will do it.

Please tell me how you feel about this! 🙏

[KYabuuchi]

c.c. @yukkysaito

[maxime-clem]

In planning I made a sub-folder for my algorithm which contains multiple packages.
But I did not ask permission...

[mitsudome-r]

If those packages are only meant to be used together, and if it makes it easier for the user to find the package, I personally don't have an objection.

[kenji-miyake]

If we really need sub-directories, I personally think it's okay.
However, before adding sub-directories, we should consider several things, for example:

• Are packages under a sub-directory specific to the domain?
  • I mean, is there no reusable package under a sub-directory?
• Are the package-split criteria well organized?
  • Can some packages be integrated into one package? Then, sub-directories might be unnecessary.

Seeing the package names in https://github.com/tier4/YabLoc, several packages seem to be generally-reusable packages.
If they will be under yabloc/ and be depended on from non-yabloc packages, that would look strange.

By the way, do we really need to merge YabLoc into autoware.universe?
I think we can keep it as a TIER IV's extension and refer to it in autoware.repos.
That doesn't violate the concept of Autoware Core/Universe.
I'm concerned about making autoware.universe a huge mono repo.
It's easy to continue adding packages to one repository, but we may be stacking technical debts.

[KYabuuchi]

Thank you for your comment!
First I will reorganize the YabLoc packages and reconsider whether the sub-directories are really necessary.
Based on the reorganization, I will consult whether YabLoc should be placed in autoware.universe.

[yukkysaito]

By the way, do we really need to merge YabLoc into autoware.universe?
I think we can keep it as a TIER IV's extension and refer to it in autoware.repos.
That doesn't violate the concept of Autoware Core/Universe.
I'm concerned about making autoware.universe a huge mono repo.
It's easy to continue adding packages to one repository, but we may be stacking technical debts.

@kenji-miyake
This is a problem within TIER IV and I am requesting that it be merged into the universe from me.
This is because I want the autoware universe to support camera based localization. We can discuss the details separately.

[mitsudome-r]

I'm concerned about making autoware.universe a huge mono repo.

This is a good point. However, I think adding in autoware.repos is essentially going to cause the same issue. (larger dependency, longer build time) We probably have to think of a way to enable registering packages as Autoware packages without merging the code in to Autoware. We might have to come up with a framework like what ROS has, e.g., registering to the build-farm and adding to ROS Index without merging code to ROS repository. Obviously we would need much more maintainer to support the framework though.

[yukkysaito]

The number of packages for each component is increasing and becoming difficult to understand; I agree with adding a sub-directory.

However, @kenji-miyake is right, I believe that the yabloc package can be integrated a bit more. Because the packages seem to be separated to the point where there is no possibility of reuse.

[doganulus]

As far as I understand from the descriptions, YabLoc is an independent set of localization packages for Autoware and is distinct from the current localization packages. If this is the case, making it a subfolder doesn't look good in my opinion.

I do understand the earlier decision against deep hierarchies, which may be problematic in many areas of software engineering. I think another concern is that it is impossible to merge everything coming from universities and other companies into a single monorepo universe. Definitely, some must stay outside. It might be good to define some structure for such (external) collections of autoware packages. For example,

• Standardize the location /opt/autoware/<collection>/<component>.
• Standardize certain component names across the Autoware ecosystem.
  • Example components are localization, perception, planning, control, evaluation, ...
• Allow collections to be independent and have their own repo:
  • Example collections: core, universe, yabloc, <company_name>, or <university_name>
• Not every collection has to contain every component.

Then the yabloc repo would look like this under such an organization:

autoware.yabloc/
　├ localization/
　│　├ particle_filter/
　│　├ segment_filter/
　│　├ graph_segment/ 
　│　├ line_segment_detector/
　│　├ ground_estimator/
　├ evaluation/ 
　│　├ ape_monitor/
　│　├ ... /
　│　├ path_monitor/