Monorepo

[gchoqueux]

Adopt a monorepo structure for itowns. The goal is to split itowns down into reusable and standalone feature packages.
As an example, I've started with the geodesy features. (@itowns/geodesy)

Context

For the moment, itowns provides an api in a single package.
this architecture poses several problems :

• Users must install the full api even if they want to use a specific feature.
• The majority of features aren't independent, as users are obliged to use a Viewer instance.
• It's more difficult for developers to add feature, re-use, test and dependency management.

Description of the proposal

Split code in packages for clearly structured code.
Simplifies development and facilitates contributions.
Increases the scope of users who only want to use a few features.
This structure makes it necessary to make features independent and improves the architecture

For the moment only one new package is proposed: Geodesy

Identified use-cases

Users can use only the packages they are interested in.
Three.js developers can use itowns features more easily.
Developers more easily merge their codes (I received feedsbacks for this architecture)

Implementation

see #2300

Comments

This monorepo structure supports Yarn
There is not breaking change for itowns users

Potential Problems

Rebasing but generally Git does the job, Except when there are bumps and new files or move files
The new publishing, only tested on my npm account

iTowns Monorepo Structure Proposal

1. @itowns/core : ⚠️

• Description: Manages processing, scheduling, rendering and 3D scene management

2. @itowns/geodesy ⚠️

• Description: Contains the Coordinates, Extent, Crs, Ellipsoid and OrientationUtils

4. @itowns/sources

• Description: Handles data sources
• Sub-packages:
  • @itowns/sources/wms
  • @itowns/sources/wmts
  • @itowns/sources/3dtiles
  • @itowns/sources/gpx
  • @itowns/sources/kml

4. @itowns/layers

• Description: Manages different types of layers and Style
• Sub-packages:
  • @itowns/layers/raster
    • @itowns/layers/colorLayer
    • @itowns/layers/colorElevation
  • @itowns/layers/vector
    • @itowns/sources/3dtiles
    • @itowns/sources/featureGeometryLayer
  • @itowns/layers/terrain

5. @itowns/mouseControls

• Description: Manages user mouse controls for navigating the 3D scene (panning, zooming, rotating, etc.).
• Sub-packages:
  • @itowns/controls/Globe
  • @itowns/controls/Planar
  • @itowns/controls/Immersion
  • @itowns/controls/firstperson

6. @itowns/widget ⚠️

• Description: Contains additional modules for graphic user interface
• Sub-packages:
  • @itowns/widget/navigation
  • @itowns/widget/minmap
  • @itowns/widget/scale
  • ...

6. @itowns/plugins

• Description: Contains additional modules or extensions for non-essential features.
• Sub-packages:
  • @itowns/plugins/debug: dev debug tools ⚠️
  • @itowns/plugins/measure: Measurement tools (distance, area)
  • @itowns/plugins/analysis: Analysis tools (elevation profiles, cut sections)

To validate

The first validation step is the specified modules with the logo ⚠️ that are included in the PR #2300.

The Following modules must be validated to begin the PR review :

• @itowns/core
• @itowns/geodesy
• @itowns/widget
• @itowns/plugins/debug

[Desplandis]

@gchoqueux First, thanks for your time writing a proposal. I have a few remarks that I would like to address.

Geodesy package

I personally agree that we could first split the geodesy package (or geographic, let's have a vote folks) out of iTowns. While I agree Coordinates, Crs, Ellipsoid and Extent (and not the tiled part of extent, see #2412) could form the basis of this package, I have some concerns about the GeoidGrid and CoordStars modules.

As I understand (from @mgermerie), the GeoidGrid was implemented due to limitation in the composition of ElevationLayer (i.e. one and only one elevation layer active at a given level). Since we'll remove it when we'll have a more generic solution for composing elevation layers, is this a good idea to expose it in the proposed package? Or are there any need outside of this elevation feature?

The CoordStars module only expose a getSunPositionInScene which as its name indicates return the sun position in a 3D scene, I'm not sure this is relevant in a geodesy package... What are your needs for this module (outside of iTowns)?

I also agree that we should expose OrientationUtils module since it exposes nice utilities to compute quaternions for different projections.

Widgets package

Is there any need to expose them in their own package if there are private? Is there a reason why we should not expose them in a public @itowns/widgets package?

Utils package

I personally don't see any reason to expose them in their own package until we rewrote them from the ground up.

Future roadmap

The more important point of this proposal is to discuss the future steps after splitting the geodesy/geographic package. I wouldn't like us to have a PR that only starts the job and then stops there. So before merging or reviewing anything, I would like to have at least a medium/long-term roadmap on how we should continue splitting iTowns and the architectural implications of such changes.

[gchoqueux]

GeoidGrid is used to read Geoid data it looks like it belongs in a geodesic package.

CoordStars doesn't really belong in this package, plus an astronomical one.

I can easily make the Widget package public

Utils package : I took advantage of PR to move this part of the code, as we both decided at previous meetings. I think this code will end up in a package, but I can see if it's not too long to move.

Future roadmap
This PR has no change for the users and proposes a first decoupage.
If we validate the monorepo and this first decoupage, we can review and merge this PR as it requires regular work to maintain it on the master.
we can then discuss the future packages, as the integration of this PR will not hinder or constrain our choices in any way.

[jailln]

My 2 cents:

It is a needed and valuable evolution (a monorepo architecture) but as we haven't plan (IGN / Ciril Group) to work on this subject in the short term, we need to sketch a plan for this migration before merging the creation of a first module so 1- we make sure we will be able to pick up the subject and finish it at some point 2- we don't create a first module that will change in the future 3- we don't start a migration that will stop after the first step and never finish it.

Do you have any idea on how you would foresee the next steps of this separation?
Also, do you plan to work more on this subject after the first module is done?

[gchoqueux]

I add a road map with my next contributions

[gchoqueux]

hello everyone
did you see that I posted a proposal?

[jailln]

Thanks for adding a roadmap :)

I have a few comments and questions:

• Geodesy:
  • I'm not sure we should have CoordStars in it (not directly related to the earth)
  • GeoidGrid should be in the Terrain package in my opinion since it can be seen as an extension of the ElevationLayer from my understanding
  • What do you mean by:

Place/orienting/scaling a object3D in geographic system in real time, replace part of code by this new feature, in these classes : FeatureProcessing, Feature, TileMesh/Tiled, examples

That you will use the geodesy package in these classes?

• Widgets:

widget : already in separate structure in src/Utils/gui and a specific bundle itowns_widgets, in this PR this module is private and it expose in itowns package

• why not making it public directly?
• we should take this opportunity to move them to src/widgets that would be clearer
• I think they should not be exposed in the itowns package because they are not part of the core and some users will only use itowns without the widgets

iTowns Core package with private sub-packages

Sorry I didn't really understand this part, can you give more details please?

In my opinion we can start reviewing once we agree on the geodesy, widget and debug packages (since they are concerned by your PR).

Regarding the rest, in general I would vote for a separation that is easy to understand (and use) for external users; roughly: a core - that can be separated in several sub-packages (geodesy being one of them) and it's that separation that we will mainly discuss I think -, and a separation by formats (@itowns/3dtiles, @itowns/potree, @itowns/wmts, etc.).

I think that one way to go for the rest would be to:

• list all itowns files and try to attach them to a module
• create a uml diagram with target modules, their relations and the files and classes they contain. A uml diagram would be a good and clear way to express it I guess, clearer than text.

I'll try to find time topropose something but don't hesitate to initiate it too.

[gchoqueux]

• Geodesy:

  * I'm not sure we should have `CoordStars` in it (not directly related to the earth)

I could move this class, but I'm not sure where it will eventually end up. The Geodesic library could also include spatial elements (such as GNSS, satellite navigation).

  * `GeoidGrid` should be in the  `Terrain` package in my opinion since it can be seen as an extension of the `ElevationLayer` from my understanding

ok

  * What do you mean by:

Place/orienting/scaling a object3D in geographic system in real time, replace part of code by this new feature, in these classes : FeatureProcessing, Feature, TileMesh/Tiled, examples

The idea is to create a class derived from THREE.Group to position an THREE.Object3D on the earth's geoid, from any geographic projection.
This example illustrates this feature : misc_clamp_ground

This class could be used in FeatureGeometryLayer, TiledGeometryLayer to replace cloned code.

That you will use the geodesy package in these classes?
* Widgets:

widget : already in separate structure in src/Utils/gui and a specific bundle itowns_widgets, in this PR this module is private and it expose in itowns package

* why not making it public directly?

To simplify, this first PR will be limited to publishing two packages, but we can make it public without issue.

* we should take this opportunity to move them to `src/widgets` that would be clearer

could give the entire path?

* I think they should not be exposed in the `itowns` package because they are not part of the core and some users will only use itowns without the widgets

I expose in itowns to avoid breacking change, it's possible to add deprecated message

iTowns Core package with private sub-packages

Sorry I didn't really understand this part, can you give more details please?

It's itowns Core package, almost all the iTowns libraries will depend on iTowns Core.
Quadree spatialization is the class to extract from TiledGeometryLayer to make the use of FeatureGeometryLayer independent, by example

In my opinion we can start reviewing once we agree on the geodesy, widget and debug packages (since they are concerned by your PR).

Regarding the rest, in general I would vote for a separation that is easy to understand (and use) for external users; roughly: a core - that can be separated in several sub-packages (geodesy being one of them) and it's that separation that we will mainly discuss I think -, and a separation by formats (@itowns/3dtiles, @itowns/potree, @itowns/wmts, etc.).

Maybe separate in thematic :

• @itowns/Geodesic
• @iTowns/Core
  * @itowns/Processing
• @itowns/Loader
  • @iTowns/WMS
  • @iTowns/WMTS
  • ....
• @itowns/3DLayer
  • @itowns/TerrainLayer
  • @itowns/PointCloudLayer
  • @itowns/3DtilesLayer
  • ....

I think that one way to go for the rest would be to:

* list all itowns files and try to attach them to a module

Yes, we could attach files when modules are valided

* create a uml diagram with target modules, their relations and the files and classes they contain. A uml diagram would be a good and clear way to express it I guess, clearer than text.

stateDiagram-v2
@itowns --> @itowns/core
@itowns -->  @itowns/Loader
@itowns --> @itowns/geodesy
@itowns --> @itowns/widget
@itowns/Loader --> @itowns/3Dlayer
@itowns/geodesy --> @itowns/3Dlayer
@itowns/core --> @itowns/3Dlayer
@itowns/Loader --> @itowns/Rasterlayer
@itowns/geodesy --> @itowns/Rasterlayer
@itowns/core --> @itowns/Rasterlayer
@itowns/Rasterlayer --> @itowns/Terrain
@itowns/3Dlayer --> @itowns/Terrain
@itowns/3Dlayer --> @itowns/3DtilesLayer
@itowns/3Dlayer --> @itowns/PointCloudLayer
@itowns/core --> @itowns/viewer

Loading

I'll try to find time topropose something but don't hesitate to initiate it too.

[jailln]

I could move this class, but I'm not sure where it will eventually end up. The Geodesic library could also include spatial elements (such as GNSS, satellite navigation).

Yes, or we could add all that in another module? No strong opinion on that, it could as well be in it. I'll let @Desplandis give his opinion too.

The idea is to create a class derived from THREE.Group to position an THREE.Object3D on the earth's geoid, from any geographic projection. This example illustrates this feature : misc_clamp_ground

I agree that we should provide an easier API to position 3D objects in a geographic context as many users need to do so and it is the cause of many opened issues. Do you already have an example of how the API would look like in your opinion?

To simplify, this first PR will be limited to publishing two packages, but we can make it public without issue.

I'd directly make it public since it is mainly intended to external usage.

• we should take this opportunity to move them to src/widgets that would be clearer
  could give the entire path?

I don't understand your question? I mean moving them from src/Utils/gui to src/widgets, which would be clearer

* I think they should not be exposed in the `itowns` package because they are not part of the core and some users will only use itowns without the widgets

I expose in itowns to avoid breacking change, it's possible to add deprecated message

Are they already exposed in itowns package ? I thought they weren't. In that case, I would take them out and add a deprecation message yes.

iTowns Core package with private sub-packages
It's itowns Core package, almost all the iTowns libraries will depend on iTowns Core. Quadree spatialization is the class to extract from TiledGeometryLayer to make the use of FeatureGeometryLayer independent, by example

Oh ok, I understand what you mean. We will need to discuss boundary cases but on a general view I agree on what you listed to be in it.

In my opinion we can start reviewing once we agree on the geodesy, widget and debug packages (since they are concerned by your PR).
Regarding the rest, in general I would vote for a separation that is easy to understand (and use) for external users; roughly: a core - that can be separated in several sub-packages (geodesy being one of them) and it's that separation that we will mainly discuss I think -, and a separation by formats (@itowns/3dtiles, @itowns/potree, @itowns/wmts, etc.).

I agree @itowns/3dtiles, @itowns/pointcloud because the source and the layer are linked

For WMTS is source it could only use on Terrain, is it necessary to separate a library if we cannot use it without another library ?

Yes, 3D tiles and potree are easy cases because the source and the layer need to be used together 😁 Some sources can indeed be used with different layers - e.g. WMTS with ElevationLayer and ColorLayer or VectorTileSource with ColorLayer and FeatureGeometryLayer.

maybe separate in thematic : @itowns/Raster @itowns/Vector @itowns/pointCloud @itowns/Terrain

That being said, I think separating in thematics won't resolve these cases either I think: where would you put WMTSSource for instance ? In @itowns/Raster or in @itowns/Terrain? In my opinion we have three choices:

• 1- put all these sources that can either be displayed as raster or vector in a module: @itowns/common-geospatial-sources (we might find a shorter name 😁 ) : wms, wfs, wmts, tms, vectortile, etc.
• 2- put them in @itowns/core
• 3- create sub-modules for each one of them: @itowns/wmts , @itowns/wms, @itowns/mvt, etc. and modules for layers and conversions: @itowns/raster (containing ColorLayer, Feature2Texture, and all the stuff related to raster conversion) ; @itowns/vector (FeatureGeometryLayer, Feature2Mesh)

I like the third option better. What do you think ? Do you see other alternatives?

I'll draw a diagram

Thanks! Can you use an open tool and maybe something that can be edited online so we can collaborate on it? I know draw.io but you can choose another one that suits you.

[gchoqueux]

CoordStar

Ok, I Move CoordStar to initial folder

Place/orienting/scaling a object3D in geographic system in real time,
Do you already have an example of how the API would look like in your opinion?

THREE.Group with the Coordinates property for placement

I remove this part from the proposal because it's more a refactoring/feature and because it's too precise
I remove New Class : Quadree spatialization from the proposal for the same reason

Widget package

I move src/Utils/gui to packages/widgets/src/, would you like me to place it in the packages/main/src/widget folder?
My bad Widget isn't exposed in itowns

where would you put WMTSSource

I had proposed to put it in the @itowns/loaders, which amounts to separating the sources as in your third proposal but with a different granularity (Both are even possible).

I will continue working on the segmentation in a shared diagram

In my opinion we can start reviewing once we agree on the geodesy, widget and debug packages (since they are concerned by your PR).

To start the code review and avoid continuously rebasing the branch, we need to decide on the PR structure

Geodesy
Widget
Debug

Which modules do you approve ?

[jailln]

Hi @gchoqueux , thanks for making a diagram and re-working the proposal! 👍 Sorry I add a busy week, I will take a look at it at the beginning of next week.

[gchoqueux]

ok
with the last proposal I think there's no need for a diagram

[gchoqueux]

@iTowns/core-devs I've rewritten the initial proposal, removing the PR-related specifics (#2300) and focusing on the architecture to be validated.

[jailln]

@gchoqueux thanks for the work, it's way clearer now!

I'm waiting for incoming @Desplandis review before reviewing the PR. Regarding the proposal, I agree with most of what is described in it. The only remarks that I have are:

• Related to the ongoing PR:
  • I think we should avoid having widgets and plugins since it is two similar notion and I'm not sure we need a separation at this point. I would rather put @itowns/plugins/measure and @itowns/plugins/analysis into widgets (making them @itowns/widget/measure and @itowns/widget/analysis). And so I think we should transform @itowns/plugins into @itowns/debug.
• For the rest:
  • I would rename @itowns/mouseControls to @itowns/Controls because it is closer to threejs naming and because we may add other type of controls such as mobile controls

All the rest looks great 👍