Inquiry into the desire for Dendrochronology implementations

[souma4]

Currently, Julia does not have registered packages for dendrochronological methods such as Superposed Epoch Analysis, Cross-dating, chronologies from such datasets, etc. EcoJulia is probably the most appropriate organization to build such a package under. Additionally, I know researchers and analysts that constantly complain about the speed of their analyses but also don't have the time to dive into lower level languages. Julia seems like a prime candidate for these people. I want to know whether you all are interested in adding such a package?

Most of the people I know use R, MatLab, and Python, mostly using the dpl ecosystem. The source code for dpl uses FORTRAN. If we can implement the same functionality in native Julia, this can also provide paleoecologists a pipeline to being a contributor.

I've paired this with an email to opendendro to screen their interest.

Let me know what y'all think.

[rafaqz]

Sounds like a good idea.

If you want to do a direct translation it can be nice to convert Fortran to Julia then "juliarize" it with better Julia idioms. I used this script for MAESPA and a few other photosynthesis libraries, to at least get me started:

https://gist.github.com/rafaqz/fede683a3e853f36c9b367471fde2f56

[richardreeve]

Yes, that would be great. Happy to help any way I can (though I am extremely short of time like everyone!).

[mkborregaard]

Wow, opendendro has a pretty big codebase, in all the languages (C++, Fortran, R, Python etc). This would be a big task (but then I guess most of these projects are). Ironically the stated goal of the opendendro organisation would seem to be more compatible with porting to julia than with what they are doing, building R wrappers around legacy fortran code:

For decades, much of the software used in dendrochronology has been in legacy programming languages which have been maintained by a small number of volunteers. These codebases are at risk of becoming inoperable or incompatible as advances in computing architecture accelerate.

Did the Opendendro people respond to your mail?

[souma4]

I did get an email from OpenDendro

Hi Jeffery[sic], We’d love to see a Julia port of the openDendro core tools

So, there is the go ahead for it.

The codebase is impressively large. I think a coarse first stage is to 1:1 convert fundamental library code. The Fortran to Julia script given by @rafaqz will be very helpful in that endeavor. Then we can juliarize it (likely in a fork because it will be a large refactoring game). From there, it would be incorporating other dendrotools that have been developed on top of dpl, either as extension packages or as base code. Still uncertain about how exactly that will play out, but baby steps.

Honestly, as I interrogate my drive to help Julia grow in scientific computing, I am hatching a plan for some level of collaborative work that can fall under the NSF (Geosciences Open Science Ecosystem (GEO OSE))[https://new.nsf.gov/funding/opportunities/geo-ose-geosciences-open-science-ecosystem] grant. I have some colleagues that have received this grant and know other strong PI's for similar projects, so I'll get some feedback from them, then figure things out, so I'll likely follow up with something on that front.

[mkborregaard]

That sounds very interesting. I have a few experiences with porting code from other languages to julia, and it can be surprisingly difficult to get the ideoms right. But @rafaqz has much more experience with these things than I do. Will be cool to follow what you do here. I'm guessing the geo channel on the slack should be a good place to find people who might have an interesting in the GEO approach you mention.

[souma4]

I finally got around to creating this and having the time to share it out. It's extremely WIP, so I won't register it or shift organizations or anything like that until it's in a stable functional spot. But, I think it's setup so that people can open pull requests, and they'll go through review. Let me know if there are any issues.

DPL.jl

[richardreeve]

Great. Apologies that I only just saw this - I've been on holiday for several weeks. I should have some time to look at it in the next couple of weeks, but one thing that's worth bearing in mind is the rules on package naming.

https://pkgdocs.julialang.org/v1/creating-packages/#Package-naming-rules

I don't think you'll be able to call it DPL, because they won't approve of acronyms that aren't immediately obvious. Just something obvious like Dendrochronology.jl would be simpler I'd have thought?

[souma4]

it's all good, this is currently all volunteer and I myself haven't gotten the ball rolling a lot. You're right about the naming rules. I forgot about that. Dendrochronology.jl is the solution.

The perfectionist scientist in me wants to use "Chronology.jl", because technically any layer based dating uses the same methods, but that would most likely confuse all the people that do it. It's not common parlance AFAIK. I think I can add description text to specify that these can be generalized.

[richardreeve]

So I've had a look at the package just to get a feel for what you've done, and I think there's still quite a way to go obviously. Apart from renaming, I think obviously adding in testing, which presumably can be adopted from the OpenDendro packages - maybe dplR? - would get you a feel for how far you are away from being able to reimplement the APIs that they provide. For what it's worth, I wrote a Diversity package in Julia based on one in R, and as well as its own testing, I added in a cross-validation suite for validating against the R package that helped me a lot later on with identifying inconsistencies by randomly generating datasets and testing against both packages. Obviously you've got a bit of work to do before then, but some preliminary testing will pick up early problems with the first things you implement I imagine. Anyway, good luck with it - I hope you find the time to do it justice!

[mkborregaard]

The OpenDendro project is both about internal Fortran implementation and R wrappers. Are you planning just to port one of these, or both? Or are you starting with one of these? Sorry if this should be obvious from the above.

[souma4]

Yeah it's a long way. Between the Holidays and Current end of fiscal year workload I haven't had the time or energy. But, it'll make progress.

As far as internal fortran vs R wrappers, I'm think of leaning into dplR since it's the most mature between R and python and has functional capability. Plus most of it is Rcpp rather than a wrapper of pure fortran, which is nice. I'll fill out existing functions until (if) the package reaches current dplR / dpl.py implementations, then I'll start getting into the Fortran and update things as needed.

It'll be a bit slow because of commitments, but I'll slowly make progress when able.