Splitting the package? #65
Comments
|
I think this could be a really good idea! I feel, that a lot of packages in Julia are already splitting plotting features and data analyzing tools. The package has a lot of python dependencies that are totally unique to mapping. |
|
I could certainly be more useful in helping to maintain this library if you wanted to split it as well. Your call. |
|
Once I finish updating the code to 0.7/1.0 I'll split the package. For one thing, the package will just load faster when you only want to do some calculations. Perhaps adding small function for simple contour plots might be useful in ClimateTools (no projection, etc... ). |
|
Steps about what should be done to properly migrate mapping functionalities over ClimateMaps.jl.
|
I give several thumbs up to that, even though I feel like not many support the idea. :D I'll try to think a bit about the "bonus point". For providing own lat-lon bounds this seems more straightforward than custom projection. Maybe a good starting point is to first add a couple of "specific" projections and see how well they interplay and how easy it is to generalize. |
|
Yes, that's a good plan for the projections. I guess we could mimic the cartopy API, which might be easier to then implement a generic function. Here's some (throw-away) code I did for a specific project using cartopy. This might be useful. function draw_quebec(gevparams::DataFrame, param::Symbol, filename::String)
central_lon, central_lat = -66, 48
projection = ccrs.LambertAzimuthalEqualArea(central_lon, central_lat)
projection = ccrs.NorthPolarStereo(central_lon, central_lat)
extent = [-81.2, -55.5, 40.5, 63]
fig, ax = plt.subplots(1, 1, figsize=(8,6), subplot_kw=Dict(["projection" => projection]))
# ax = plt.axes(projection=projection)
ax.set_extent(extent)
ax.gridlines()
ax.coastlines(resolution="50m")
cs = ax.scatter(gevparams[!,:lon][:], gevparams[!,:lat][:], s=10, c=gevparams[!,param][:], transform=ccrs.PlateCarree())
out=fig.colorbar(cs,extend="both")
out.set_label(convertsymbol(param), size=10)
plt.savefig(filename, dpi=600)
# plt.show()
plt.close()
return ax
endFor the org, I agree with you that it helps with the discoverability of packages. I'll look into it. |
|
JuliaClimate already exist sadly. https://github.com/JuliaClimate Will try to think about other names but here's some thoughts: JuliaAtmosphere (seems restrictive) |
|
Darn, JuliaClimate is a good name, the best one. I wonder who "stole" This name, and has no logo, no repos, no public people... uuuuurgghhh so terrible!!! |
|
We can consider ClimateJulia, JuliaClimateOrg as well. |
|
Yeah, indeed I was hoping for at least some public repos on JuliaClimate. ClimateJulia is indeed one to consider! Well, I'll take it and see what we do! |
|
What is the reason to not just go with JuliaGeo? |
|
I'd also be interested in helping out with a JuliaClimate organization! Maybe @natgeo-wong too. Long shot but we might be able to ask around Slack for the owner of @JuliaClimate? I'd vote for JuliaAtmosOceanClimate but maybe that's too verbose.
Maybe a merger would work but from their website "JuliaGeo is an organization that contains a number of related Julia projects for manipulating, querying, and processing geospatial geometry data" which seems different from climate but they do have some pretty general-purpose packages like NetCDF.jl. |
|
I see your point about the different scope of JuliaGeo. By the way, @Alexander-Barth if we make the organization discussed here, are you willing to move |
|
I'd be interested in helping out with JuliaClimate. Tagging @aramirezreyes and @simonbyrne here. We can also name it JuliaAOC (Atmo,Oceans,Climate). Bonus points if you guess the other meaning. Last note about Cartopy - I find it pretty slow actually. Not sure if this is a computer problem or a PyCall thing, but I find in general Python backends tend to run very slowly compared to pure Julia codes so if in the future we could do a pure Julia mapping/projection package it would be cool (maybe attached to Makie?). This is especially important when exporting images for animations. |
Something like this started in https://github.com/NHDaly/Maps.jl |
|
Saw this, but the link is gone from the Julia Slack and I couldn't retrieve it xD. Thanks! |
|
I like the JuliaAOC name, as Atmosphere, Ocean and Climate ecompass things that are not properly climate (Although Julia Alexandria Ocasio Cortez may sound weird for some!) |
|
Happy to see the climate community going forward for an organization! :) From what I have read JuliaGeo is for "backend" manipulations (can't find the Github thread where I read that). There was some discussions JuliaAtmosOcean already exist. JuliaAtmosOceanClimate is too verbose imho and I'm not sold on acronyms like JuliaAOC. For mapping purpose, cartopy (and basemap) is in my opinion a temporary solution that allowed me in the past to produce maps for my job without creating everything from scratch in Julia. I was mostly focused on extracting and subsetting netCDF files and creating climate scenarios with quantile-quantile bias correction and some climate indices. Mapping and plotting was mainly for diagnostic purpose. If people are interested to develop something in Julia, I'd be more than happy to remove those features from ClimateTools! Functions for mapping in ClimateTools are not generic. Using Makie, there is also GeoMakie that should be looked into. Maps.jl does seem interesting also and have a common objective. |
|
I remember that Julia Atmos Ocean was announced in the discourse some time ago. It doesn't have much activity, perhaps we could get help from the owners and rename it to add climate? pinging @rickedanielson |
|
Yeah, I'm not happy with Python dependencies either :/ I think that just migrating the code in ClimateMaps as a first step is still useful and have some benefits: using ClimateTools would be much easier and quicker to load. Then, when GeoMakie or Maps is ready, we can simply modify the code under the hood or simply redirect users to the new packages. |
|
As far as mapping is concerned, I think we should all consider Makie: it is extremely strong (performance-wise I mean) and allows for crazy interactivity. Have a look here for a nice taste: https://juliadynamics.github.io/DynamicalSystems.jl/dev/interact/od/#function-video The downside is that its relatively new, and thus has bugs and much less quality of life things than what you'd get from a mature library like PyPlot. But if we work together I am very positive we can make it happen and have something very strong in the end. I remember when I was making these interactive applications for DynamicalSystems.jl in Makie, the main dev @SimonDanisch was very helpful and always there. Looking at the contributors page, seems like Makie got a second lead dev @anthony-wang . What are the thoughts of you Makie guys on making a proper climate plotting package that has support for both 2D and 3D (like, on a globe)? We could start with @NHDaly 's code as well. |
|
It would definitely help in all geoscience/planetary aspects as well, not just climate. The conversion of coordinates into coordinate form should not be too difficult when we have other examples to follow (currently looking at using the R-CRAN projection package as a template), it's when we start doing things like contour maps that things start getting really tricky. But this is a discussion best done in another thread. GeoMakie seems to rely more on shapefiles at the moment, which is why I'm a bit more hesitant to build off that. |
|
While we are at it, I think a more representing name, and easier to identify its functionality, is ClimatePlots.jl instead of ClimateMaps.jl . Depending on your background, Map could mean transformation between different grids, but Plot is more unambiguous. |
|
Good idea, this would certainly be more general. |
|
Glad to see so much interest in collaborating on JuliaClimate. I'm not a climate scientist, but keen to help out if I can. In any case it would be good to work together with JuliaGeo as much as possible. In my view field specific tools would fit great in JuliaClimate, and more widely applicable geospatial tools could easily go in JuliaGeo as well. Though in general I don't care too much where something lives, as long as we build some great tooling together :) In my experience climate scientist are especially good in working with large geospatial datasets, if I look for instance at xarray and pangeo initiaves. I think these are good examples of projects that are driven by mostly climate & ocean folks, but generally very useful for all geospatial workflows.
That's true, so if you go for that it will not so much be "building on", rather "adding to". I would bring it up with the author though, it would be nice to have a single go-to package for geospatial plotting, based on Makie. |
|
Thanks @aramirezreyes, @Balinus, and @Datseris for your thread-prompts. @Balinus happens to be the only owner of JuliaAtmosOcean with 2FA, so perhaps those who are as careful may wish to take over/rename that group and are welcome to do so (happy to be a regular member of the new ClimateJulia otherwise). |
|
I'd be very happy to see makie used for this ;) I don't have much time to spare atm, but I'll try to give guidance and fix bugs wherever I can! |
|
Another reason I'd like to have a repo for plots in climate/geo applications is so that we have a common place to discuss design. Using makie will require a lot of communication and its nice to have a dedicated place to talk. |
|
@Datseris The repo is available here -> https://github.com/JuliaClimate/ClimatePlots.jl Right now, I'm mostly doing a functional split of ClimateTools into ClimateBase, ClimateTools and ClimatePlots. All three will be migrated over JuliaClimate when I finish this split. The actual implementation is using Basemap under the hood. The idea would be to indeed switch to Makie, Maps, or something more Julia as opposed to switching to cartopy. Hence, I would not further develop the Basemap/Cartopy approach, the idea with the 1st tagged version of ClimatePlots would be to have at least one version that is working with ClimateTools Hope it makes sense! @gaelforget Really nice map! Basemap is a chore to install these days. Since ClimatePlots.jl is quite simple now, maybe it's still worth it to convert the code to cartopy and use that until we have a functional package that uses Makie. |
|
Split is done! Was easier than I expected. :) |
|
@gaelforget I think I'm gonna need an owner access. Right now I migrated ClimateBase and I only have a "read" access. Cheers! |
|
@rafaqz I think you might be interested to look into @JuliaClimate |
|
Thanks for the heads up. I probably don't have much to contribute to a climate org as an ecologist, except making sure GeoData.jl could be used for your ClimGrid as you mentioned some time. |
|
I actually thought that GeoData might replace the whole extracting process that I did in ClimateTools so that the package can concentrate on analysis and processing of climate data. Perhaps I misunderstood the objective of GeoData though! Cheers! |
At the moment it seems like the plan is to establish a direct path from NCDatasets.jl to DimensionalData.jl (or GeoData.jl), see Alexander-Barth/NCDatasets.jl#60, which may mean that none of that would need to be taken care in ClimateTools (which anyway has NCDatasets.jl as a dependency). And I also think this is the most reasonable way, as what you do with the data after you load them seems to be more the goal of ClimateTools and company. |
|
@Balinus GeoData could pretty much do that if that's what you intended. We would have to make sure it covered all the use cases, you seem to be handling complex projections so we would need to test those as you know I don't really use them enough. I just wont have time to test or integrate it with ClimateTools so you would have to take the lead on that after I get it started. It might help to make a branch where we swap out ClimGrid for GeoArray and get it working. The NCDarray/GeoArray nearly maps onto your ClimGrid already. Also, the NCDstack type handles multi layer arrays if you use them. Using GeoData would mean you could use tiff and hdf5 and other sources too if you ever need to without changing much, although I'm not sure you do? Edit: but yes that is pretty much the role I imagined GeoData.jl could have. It might be a while until it's seemless though. @Datseris that seems to be your suggestion more than the plan. It's not my plan anyway. |
|
In #60 I suggested that a conversion should exist inside NCDatasets.jl that converts the type |
|
@rafaqz Ok, we share the same understanding. Of course, I'm not expecting that you integrate GeoArray into ClimateTools. Right now I keep an eye on the development of GeoData. I think this is going in the right direction and I'm gonna swap from my ClimGrid to a GeoArray once this is more mature!
Right now it's only netCDF, but having support for hdf5 and tiff baked-in through GeoData.jl would be awesome. Cheers! |
Just added admin privileges to the ClimatePlots-owners team |
|
On the netcdf front I feel I should mention https://github.com/gaelforget/NCTiles.jl As for other packages mentioned here, it's a work in progress. For now the focus is on integration with / support of the gcmarray & gcmgrid types (from https://github.com/gaelforget/MeshArrays.jl) and C-grid variables (MITgcm output first; other models hopefully soon). Essentially we use either Netcdf.jl or NCDatasets.jl as back-end and build on that. Not entirely sure how this relates to other efforts yet... |
I'm not able to add Julia registrator on ClimateBase and ClimatePlots as I do not have owner privilege. You'll need to either manage those admin aspects or add someone as a co-owner! |
Not sure that I understand. I thought that giving admin privileges to the I am hoping we can get the team approach to work with the apps & plugins we need. Otherwise, we will have to give organization-wide owner privileges every repo owner...
|
No, this is guaranteed not necessary. In JuliaMusic I am the only owner (because we don't have anyone that is regularly contributing to more than 1 repo besides myself), yet the repo MusicXML.jl is fully controlled by another user, including installing apps. @gaelforget maybe you forgot to add the repos themselves to the team? (this is a mistake I very frequently do I am embarrassed to say). Because besides the members of the team, one needs to configure individually which repo belongs to each team. Then, after adding a new repo to a team you also have to change the priviliges with respect to this repo to "Admin". |
Thanks for jumping in @Datseris I believe that I did what you describe earlier this morning (before the message from @Balinus). Please see screen shot below. Could it just be a latency thing (which waiting 24h would fix or something like that)?
|
|
Ok! Let's wait another 24h and see how it goes for other repos (I'll create some dummy test repo). Cheers! |
|
But I must admit, doesn't it make sense to have at least one more owner, which should be Balinus, given that at least at the moment he develops most of the repos of the org? It will just make everything easier and I think Balinus has been a member of the community long enough to be trusted? |
Did that work? |
|
I don't think so 😞 Created a repo in my account, transfered the repo to JuliaClimate and specified Climateplots-owner as the destination and ended up with "read" access only, see screenshot below.
I don't mind being an owner but it's not something that I'll want to manage long-term. I have a new job and development time on Julia is in the evening. Hence, I do not plan to spend my evening on admin! 😉 edit - With that being said, with the admin access on a team level, I can activate apps on those repos without problems once the app is approved by the org like codecov, travis, etc... |
Sorry if my question was unclear... I meant to ask: are you now able to add Julia registrator on ClimateBase and ClimatePlots ? It would be great to know that this part will work in the future without having to add organization wide ownership systematically. But you did not list
Isn't that consistent with what is discussed above regarding teams though? (i.e. after transferring a repo, someone with organization wide privileges needs to add admin privileges to a team for that repo; I just did that for
I was going to add organization owner privileges to you as suggested by @Datseris after we confirmed that the app-related issue had been resolved but I am kind of confused by the above statement now. Do you indeed want to take on the role of org wide owner even if that turns into a long term thing? ps. congrats on the new job Is this you btw? https://www.researchgate.net/profile/Philippe_Roy5 |
Specifically for Registrator, only the owner of the org can add/remove the app. Just tried with ClimateTesting (with the admin privilege) and got an error message : "You do not have permission to modify this app on JuliaClimate. Please contact an Organization Owner."
Yes, true. Just re-read the message.
Well, I guess I had some fear about more work (wrote the message after a 3-hr long meeting!). After some thinking the work involved does not seems to overwhelming. So yes, no problem about a long-term commitment! I'll gladly help you manage the day-to-day operations.
Thanks! Yes indeed, this is me :) Cheers! |
Great to hear! You now have full owner privileges and responsibilities 🎉 Thanks for your patience @Balinus ps. I also created a team for meta with the both of us in it and one for MeshArrays (me) as I just moved to the org |
Cool. Thanks for confirming |
Thanks! Let's build a vibrant climate community! :) |
|
Hi @Balinus , sorry for the question but I am a bit confused about the organization of packages now. I've started writing some very handy dandy functions like ("interannual_variation"), which seem to be extremely similar with the climate tools discussed here: https://balinus.github.io/ClimateTools.jl/stable/indices/ But where are these now? Where should I contribute these functions? |
|
@Datseris I think that the process of extracting, handling and exportation of data should be taken care by ClimateBase, the computations should go in ClimateTools: indices, timeseries processing (bias corrections), regridding. Then, plotting the results should be taken care by ClimatePlots (histograms, timeseries, maps, etc). Right now, there is a functional split between the 3 packages as of version 0.16 of ClimateTools. However, extraction and exportations functions are still in ClimateTools (but will be moved over to ClimateBase when I have time). So, to answer directly your questions, indices should go in ClimateTools! I'll move it to JuliaClimate as soon as I have some free time. :) Hope this helps! edit - note that the split is fairly recent and documentation is still 100% in ClimateTools repo and now outdated because of the split. |
|
Hi! Related to this last comment, I would piggy back and ask for some advice: |
Wondering if splitting the package would be a good idea: ClimateTools and ClimateMaps? There's a lot of dependencies problem caused by the mapping features.
The text was updated successfully, but these errors were encountered: