Big docs reorganise and expand. #109

pp-mo · 2025-01-16T12:30:33Z

Closes #80
Refactored version of #105

Adds new sections on core data and general operations.
Integrates the how-to sections also

Todos:

prune for repetition.
review + clarify 4-way Diataxis-like top-level division =
- Getting Started (intro + tutorial)
- User Guide
- API
- Detail Notes
move existing details sections to "detail notes" sections (possibly with short hinting links)
add a how-to on chunking control
review for changes to equality provision (no comprehensive == on variable/ncdata)
re-proof-read all

…string data fix.

pp-mo · 2025-01-29T10:30:16Z

@trexfeathers welcome and thanks for looking !
if reviewing, scope here is really : do the docs structure and coverage generally look appropriate to you now (as a first proper attempt). i.e. notably

are there things badly misplaced
or plain missing
is it clear enough from the entry point level what is available + where to find things

Great also to have anyone else comment on this,
I think it's the only thing really needed now prior to v0.2 release (except release whatsnew drafting)

pp-mo · 2025-02-06T12:52:20Z

OK I have now reviewed all the API docs builds, fixed (hopefully) for correctness and updated, generally smoothed over and introduced some more cross-links into newer docs sections where it seemed obvious.

So I reckon this is up to scratch now, pending any suggestions about structural improvements.

trexfeathers

OK I've read most of it. It is impressively comprehensive 💐

To-do for @trexfeathers:

Review the details section
Consider the overall structure/approachability

docs/userdocs/user_guide/data_objects.rst

docs/userdocs/user_guide/common_operations.rst

trexfeathers · 2025-02-06T16:15:10Z

docs/userdocs/user_guide/howtos.rst

+    >>> if "_fillvalue" in var.attributes:
+    >>>     var.attributes.rename("_fillvalue", "_FillValue")
+    ... 


Missing indenting

docs/userdocs/user_guide/howtos.rst

trexfeathers · 2025-02-06T16:25:37Z

lib/ncdata/utils/_compare_nc_datasets.py


-    Accepts paths, pathstrings, open :class:`netCDF4.Dataset`\\s or :class:`NcData` objects.
+    Accepts paths, pathstrings, open :class:`netCDF4.Dataset`\s or :class:`NcData`


Sphinx domains not rendering

Coincidentally, just fixed this docstring in a parallel PR : #112

(should now come good when this is rebased or merged from main)

lib/ncdata/threadlock_sharing.py

docs/details/known_issues.rst

Co-authored-by: Martin Yeo <[email protected]>

trexfeathers

Here is the rest of my review.

General thoughts

I think this is some of the most complete documentation we have, because it caters to all the different types of readers.

It is laid out in an approachable way that is also low maintenance, so I'm happy to merge with the current structure 👍👍👍

However

Given the excellent breadth of 'all-angles' content, it feels like a missed opportunity to not go all-in on Diataxis. From offline conversations this sounds deliberate, but I'm hoping we can come back in future to 'finish the job'?

docs/details/interface_support.rst

docs/details/threadlock_sharing.rst

trexfeathers · 2025-02-10T13:34:59Z

docs/details/threadlock_sharing.rst

+In practice, Iris, Xarray and Ncdata are all capable of scanning netCDF files and interpreting their metadata, while
+not reading all the core variable data contained in them.
+
+This generates objects containing `Dask arrays <https://docs.dask.org/en/stable/array.html>`_ with deferred access


You have Intersphinx for Dask, so I recommend using it.

To achieve a link to this specific page, you can use this syntax (not sure about correct way to pluralise):

Suggested change

This generates objects containing `Dask arrays <https://docs.dask.org/en/stable/array.html>`_ with deferred access

This generates objects containing Dask :external+dask:doc:`array` s with deferred access

More:

Official documentation

Working example in Iris:

Docstring

Rendered

docs/details/threadlock_sharing.rst

docs/change_log.rst

docs/userdocs/user_guide/common_operations.rst

docs/userdocs/user_guide/howtos.rst

trexfeathers · 2025-02-10T14:19:39Z

docs/userdocs/user_guide/howtos.rst

+    >>> variable.set_attr("x", 3.)
+    >>> variable.get_attr("x")
+    3.0
+    >>> variable.set_attr("x", "string-value")


No, it's not the same as __setattr__ , because that would be like a variable.x = 'string-value'

But this is a variable.attributes["x"].value = 'string-value'

Confusing, eh ?!

OK then __setitem__. Ignoring my technical inaccuracy, I'm still interested about whether this design philosophy is a potential direction to explore in future, or whether it's just impossible. The current need for documentation shows that we're not adhering to the least-surprise principle.

pp-mo · 2025-02-11T01:29:16Z

Progress : New commit 3433c29 covers, I think, all the "original" review comments
- and just a couple of the newer set
I have to stop now, and it makes a good point to pause.

Meanwhile I start to think about the "newer set" of suggestions

pp-mo · 2025-02-12T18:17:59Z

@trexfeathers thanks for your careful attention !
I think I've now addressed all-to-date

Probable exceptions, issues you've raised which I think I want to "pin" for now :

recognise diataxis explicitly
consider rename, replace or remove the "set_attrname" and "get_attrname" methods
update changenotes including release version Complete changenotes for v0.2 #115

Please check it out + see what you think.

On inspection, I think I probably don't need to merge from main before completing this (read on...)
So easier for you to check it all out as-is without further re-hash, and I'll merge from main later.

Differences currently waiting on main are only those from #112, to do with unpinning numpy to v2. Associated changes affect array printout and its tests. That does not create major problems here, as changes here are all docs with no functional effect.
It may affect some code examples, but as they don't run as doctests, we will only find that later by looking.
Not exactly a good thing, but not a merge blocker !

pp-mo · 2025-02-12T18:39:54Z

Hang on ....

I just realised the initial "Introduction" tutorial has chunks of code that ought to be in rst "code-block"s.
Left-overs from my initial ignorance.
I will attend to that ...

pp-mo · 2025-02-12T18:48:08Z

Hang on ....

I just realised the initial "Introduction" tutorial has chunks of code that ought to be in rst "code-block"s. Left-overs from my initial ignorance. I will attend to that ...

OK done that. Not sure what else I'm still missing at the last minute though 😥

trexfeathers

This is great! We're down to 4 outstanding options, plus a final 5th conversation that needs no action.

trexfeathers · 2025-02-13T16:33:55Z

docs/userdocs/user_guide/howtos.rst

+    >>> for name in wanted:
+    ...     data.variables.add(data.variables[name])
+    ...
+    >>> to_nc4('output.nc')


Suggested change

>>> for name in wanted:

... data.variables.add(data.variables[name])

...

>>> to_nc4('output.nc')

>>> for name in wanted:

... data.variables.add(data2.variables[name])

...

>>> to_nc4(data, 'output.nc')

trexfeathers · 2025-02-13T16:59:21Z

docs/userdocs/user_guide/general_topics.rst

+Very briefly :
+
+* types (1) and (2) are equivalent to Python strings and may include unicode
+* type (2) are equivalent to character (byte) arrays, and normally represent only


Suggested change

* type (2) are equivalent to character (byte) arrays, and normally represent only

* type (3) are equivalent to character (byte) arrays, and normally represent only

trexfeathers · 2025-02-13T18:08:50Z

docs/userdocs/user_guide/howtos.rst

+    >>> variable.set_attr("x", 3.)
+    >>> variable.get_attr("x")
+    3.0
+    >>> variable.set_attr("x", "string-value")


What do you think?

I was gonna make a branch to illustrate, but I think that would need too much time.

The current handling of self.attributes is moved into self._attributes. You no longer have to warn people against working with it directly - it's implicit/self-explaining, especially since it's undocumented. But it's still easily accessible for ncdata developers, and is the object that is in charge.

A new self.attributes is created as the 'public face'. This has a __setitem__ and __getitem__, which perform the work currently done by set_attrval get_attrval, writing/reading from self._attributes.

IMO this would be the best of both worlds.

I consider my original comment to be actioned. Please feel free to Resolve conversation once you have read this.

Big rework and expand docs.

a51f251

pp-mo mentioned this pull request Jan 16, 2025

Lots more documentation #105

Closed

4 tasks

pp-mo changed the title ~~Big rework and expand docs.~~ Big docs reorganise and expand. Jan 16, 2025

pp-mo added 2 commits January 16, 2025 17:04

Lots more improvements + move sections.

5e81543

More fixes to correctness, consistency, readability. Add example for …

8b3c52a

…string data fix.

pp-mo requested a review from trexfeathers January 29, 2025 10:13

trexfeathers assigned trexfeathers and pp-mo Feb 5, 2025

Overhaul all API docstrings.

0e83165

trexfeathers requested changes Feb 6, 2025

View reviewed changes

pp-mo mentioned this pull request Feb 6, 2025

Numpy v2 support #112

Merged

pp-mo and others added 10 commits February 6, 2025 17:35

Update docs/userdocs/user_guide/data_objects.rst

dce4b72

Co-authored-by: Martin Yeo <[email protected]>

Update docs/userdocs/user_guide/data_objects.rst

de38b89

Co-authored-by: Martin Yeo <[email protected]>

Update docs/userdocs/user_guide/common_operations.rst

10a6bee

Co-authored-by: Martin Yeo <[email protected]>

Update docs/userdocs/user_guide/common_operations.rst

cf79296

Co-authored-by: Martin Yeo <[email protected]>

Update docs/userdocs/user_guide/common_operations.rst

2356d12

Co-authored-by: Martin Yeo <[email protected]>

Update docs/userdocs/user_guide/common_operations.rst

872aa19

Co-authored-by: Martin Yeo <[email protected]>

Update docs/userdocs/user_guide/general_topics.rst

33232da

Co-authored-by: Martin Yeo <[email protected]>

Update docs/userdocs/user_guide/general_topics.rst

28b3ca3

Co-authored-by: Martin Yeo <[email protected]>

Update docs/userdocs/user_guide/general_topics.rst

eea69fb

Co-authored-by: Martin Yeo <[email protected]>

Review changes: links, indents, rewording.

a1fa515

trexfeathers requested changes Feb 10, 2025

View reviewed changes

Completion of original review comments (mostly, a few from new set).

3433c29

pp-mo added 4 commits February 12, 2025 00:57

Fixes to data types documentation.

06cd859

Fix external link.

4e563c1

Fix list of core object container properties.

41701f9

Fix bad formatting on installation page.

e5007f1

pp-mo added 2 commits February 12, 2025 18:21

More review changes + tweaks.

a9afc60

Include basic changelog update in the release process docs.

d526b0c

Fix code blocks in introduction.

12eb3a2

trexfeathers requested changes Feb 13, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Big docs reorganise and expand. #109

Big docs reorganise and expand. #109

pp-mo commented Jan 16, 2025 •

edited

Loading

pp-mo commented Jan 29, 2025

pp-mo commented Feb 6, 2025

trexfeathers left a comment •

edited

Loading

trexfeathers Feb 6, 2025

trexfeathers Feb 6, 2025

pp-mo Feb 7, 2025

trexfeathers left a comment

trexfeathers Feb 10, 2025

trexfeathers Feb 10, 2025

pp-mo commented Feb 11, 2025 •

edited

Loading

pp-mo commented Feb 12, 2025 •

edited

Loading

pp-mo commented Feb 12, 2025

pp-mo commented Feb 12, 2025

trexfeathers left a comment •

edited

Loading

trexfeathers Feb 13, 2025

trexfeathers Feb 13, 2025

trexfeathers Feb 13, 2025


		Accepts paths, pathstrings, open :class:`netCDF4.Dataset`\\s or :class:`NcData` objects.
		Accepts paths, pathstrings, open :class:`netCDF4.Dataset`\s or :class:`NcData`

	This generates objects containing `Dask arrays <https://docs.dask.org/en/stable/array.html>`_ with deferred access
	This generates objects containing Dask :external+dask:doc:`array` s with deferred access

	* type (2) are equivalent to character (byte) arrays, and normally represent only
	* type (3) are equivalent to character (byte) arrays, and normally represent only

Big docs reorganise and expand. #109

Are you sure you want to change the base?

Big docs reorganise and expand. #109

Conversation

pp-mo commented Jan 16, 2025 • edited Loading

pp-mo commented Jan 29, 2025

pp-mo commented Feb 6, 2025

trexfeathers left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trexfeathers left a comment

Choose a reason for hiding this comment

General thoughts

However

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pp-mo commented Feb 11, 2025 • edited Loading

pp-mo commented Feb 12, 2025 • edited Loading

pp-mo commented Feb 12, 2025

pp-mo commented Feb 12, 2025

trexfeathers left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pp-mo commented Jan 16, 2025 •

edited

Loading

trexfeathers left a comment •

edited

Loading

pp-mo commented Feb 11, 2025 •

edited

Loading

pp-mo commented Feb 12, 2025 •

edited

Loading

trexfeathers left a comment •

edited

Loading