Repository vs Packages Distros #17
Replies: 2 comments 4 replies
-
|
So you are pointing out that we have two types of audiences for the content of this repo: library maintainers, and library users. I definitely see how you are thinking and why your suggestion here have great merit. But let's ponder alternatives and pros and cons.
Let's look at some effects:
Let's think a bit about this before taking the decision. But, please add what other things you think needs to be considered, and also what you consider the biggest reasons for or against a particular solution. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, all of your observations are right, and you're probably more aware of the authors' approach to this, since you've been hosting ALAN as both a repository and website — in the past the library and docs were also part of the ALAN repo, so how many users/authors would get them directly from the repository (Bitbucket) vs from the website? An important consideration is, I think, that there's no getting around the fact that we'll need to provide the libraries as Zip packages (on library per package only), with all the library sources, license, documentation, and examples. If an author is a Git user, chances are that he/she'll also be a collaborator of the project anyhow (if he/she sees a bug, he/she will most likely create a PR, rather than mention it on the mailing list). But the average author landing on this repository, would probably click on the DOWNLOAD button visible in the main page, or go to the releases page, both of which lead to downloading the whole repository code (including the So, although the repository will always contain all the required files and docs (in some form or another), these GH-generated Zips will always contain too much clutter for end users — repository settings, test suites (solutions, transcripts, etc.), source documents in AsciiDoc or markdown, whereas most users will probably only want a specific locale library, the HTML docs, and the bare examples (no solution nor transcripts). Definitely, if we want to ensure that authors navigating this repository can see all the documentation, this means we'll have to provide it in Markdown, since HTML docs can't be previewed on GitHub (except via LiveHTML links, which are no better solution than a GHpages website), and since GH doesn't support So it's the documentation aspect that which worries me most. After all, the same is true for the ALAN Docs project, were you can't really read most of those documents in AsciiDoc, not directly on GH, because of the various included assets, or because their are split, etc. Having a GHPages website for those who wish to read online documents related to this repository would be much easier for us, for we'd only have to convert to HTML selections of the ADoc sources (i.e. without dev stuff). On the other hand, the Git wise author can always clone the repository and build all the Git-ignored contents and view the offline — but then, again, the problem is mostly related to previewing some documents on GitHub, via the WebUI. My arguments in favour of this approach are that, since we'll have to produce HTML versions of the docs to include in the distro packages as Zip files (e.g. adding them to the Releases page of the repo, along with the auto-generated archive), we might just as well park the generated HTML docs in the Obviously, if an author visiting the repo wants to study the library code, than the repository if more useful than the GHPages website, but if he/she wishes to read the documentation, than probably the GHPages website is more practical. For other type of info, there's always the Wiki (e.g. for guides on how to hack the library, etc.). I don't necessarily see these three distinctions as being so drastic, it's more about deciding where how priorities go. It's just that I'm starting to realize that keeping all these markdown READMEs up to date, in each subfolder, is becoming quite time-consuming, and that we're doing it mostly for the occasional reader than the final users who's going to download and use the library — and you know well that I do put a lot of efforts to ensure that README files are well organized, contains all the required links, credits, etc. It's just that they often feel like overkill, especially in projects like this one, were the final product will contain full HTML documentation with all the required info (and more). The fact is that problem is exponential in nature: each new library introduces a new maintenance burden in this respect. Automating the documentation via commonly shared files, advanced ADoc features, and scripts, would make life easier for us, but will not ensure all these accurate README files in each folder (which here means each sub-project, really). The goal of this repository is to provide authors with libraries packages they should download and use to create adventures — in this respect, even an advanced Git users would still have to go through the download process, for it's the more practical solution for the ALAN workflow. Hence, granting a smooth navigation of the repository in terms of docs being previewable in the browser, of a GHPages website presenting the project, are both just intermediate steps leading to the download stage. We probably shouldn't even host the libraries docs on the website, for whoever downloads the package should just open the HTML docs locally (and if he/she's not going to download them, why reading the full docs anyway?). It's not as if they're about to download a 2Gb file; these packages will be a few MBs only, so even the curious one would probably just download, unzip, read, and then decide if to keep or trash it. That's fine. The number of projects is growing, but not so the number of contributors, so automation and adoption of well tested standards could make a huge difference in terms of "maintenance costs". But at the end of the day, the final products will be the same, as they have always been — but better in quality and delivery. ALAN has always been the product of a well coordinated toolchain, with automation and all, we're now basically applying the same principles to documentation, libraries, and any product that's related to ALAN. So the real question is, how many authors used the Bitbucket repository (or even visited it) in the course of the years, vs using the website and the Yahoo list (which also offered downloads)? But of course, by using the AciiDoc coalescer we could still automate the creation of As for packages, the Zip file with each library can be made available via releases (auto-generated and added via Travis or GH Actions), but this would be more complicated for it would require all libraries to match a specific relase, whereas the website could offer the latest version of each library without requiring them to be updated (released) in synch. |
Beta Was this translation helpful? Give feedback.
-
I agree with you on this and the rest. So I'm also leaning towards thinking of this repo as the "development site" for those "end products" which are the individual, downloadable, libraries.
Just to explore one more structuring option: what about making each library a separate git submodule? Then this repo is the collected "development repo", but actually working with a library, versioning and downloading would be on the language specific repo. It is a breach of "mono-repos" and might get a bit fragmented, but it's an option. Also, that could be done down the line if we feel it would be to some benefit. |
Beta Was this translation helpful? Give feedback.
-
The main problem I see with that regards share assets (from toolchain scripts to ADoc snippets, etc.), for whenever we updated those and rebuild we'll be propagating changes to multiple repositories, which would then have to be committed and pushed separately. That, and the risk that changes to individual repositories might end up conflicting with the overall automation part. Again, it's the documentation I'm most worried about here. Submoduling just the libraries' code wouldn't be a huge issues in itself, but documentation sources should (IMO) be part of this repository, and only their finalized version (converted or coalesced) would be worth storing into the submodule (i.e. library maintainers would have to apply changes to the documentation here, not in the library repository). That's for the cons. As for the pros, the obvious advantages would be that each library could have its own GHPages website (in the target language), its own Issues, Discussions, Dashboard and Wiki (again, in its own locale, e.g. Spanish, Sweedish, etc.), which could be a benefit in terms of contributors being able to communicate in their native language. As an added bonus, each repository could leverage GH Releases for its own library, whereas this repository would have trouble with that since not all libraries can be synched to a same release. As an added minus, coordination of efforts regarding improvements that should be shared across libraries (e.g. to embrace new language features) might turn out to be more complicate in terms of cross-projects communication. I'm confident that it could be possible to come up with some strategy to make this possible, e.g. by structuring folders in a smart way, but it would still be essentials that maintainers and contributors of the submoduled libraries are aware of the toolchain needs of this repository, to prevent breaking things up — which makes me wonder if it wouldn't then make it more practical to just work here. Unfortunately Git submodules are not as good as they could have been, and can be quite tricky to handle (especially for Git newbies), with all the update strategies, the need to remember to integrate upstream changes, etc. Git trees are probably even worst than submodules though. Let's put it this way, it would much harder to shape submodules around the needs of this repository's toolchain then to shape this repository's toolchain around the needs of submoduled libraries. So, if we're going to go for this solution, we need to decide upon it early, and build the project with that in mind. |
Beta Was this translation helpful? Give feedback.
-
Errrata About ReleasesActually, now that I come to think of it, it is indeed possible to control which files go into the Zip archives of GH Releases and Download button. We actually use this in the StdLib (well, once it's merged on But then, the StdLib repository only contains one library (plus docs and examples), so it's easy to control what goes into a Release package. In this repository, we could still used this approach to exclude all ADoc sources, folders and scripts, and other dev-related stuff. But we'd still be facing the problem of having to handle single-library releases. Possible Releases Strategy for ArchivesIf we wanted to circumvent the need of creating custom Zip archives for library packages, we could still create GH Releases each targeting a specific library only. A release is just a Git Tag, and it's not bound to any branch, so we could have different exclusion patters in the In other words, downloading the Zip via the DOWNLOAD button from the repo main page would result in a Zip file with all libraries (but no dev files), whereas locale-specific branches would produce a Zip with only the files of that locale (also with DOWNALOD button, if the branch is selected). Releases would be created on the locale specific branch, instead of All of this might sound less practical, but it has its own merits since it won't depend on any CI services but only native Git features — which means that it would also work locally, with a fork or clone. The downside is that in order to provide the HTML documents in the packages, we'd be force to track all the HTML docs too (which is probably what we'd be doing anyway, if we're going to host a GHPages website via the Probably this is not the most practical solution, but I thought it was worth mentioning it, for the sake of completion and brainstorming. |
Beta Was this translation helpful? Give feedback.
-
Packages ProposalAfter having thought about it thoroughly, I've come to the conclusion that best solution for separating maintainers files from distribution files and packages, the following solution might the best one, which is very similar to the one adopted in the StdLib:
What I had in mind here is something similar to what we've been doing in the StdLib repository, to simplify maintenance of the documentation. I.e. to use real Alan sources for the examples code and transcripts shown in the documentation, by leveraging Asciidoctor We then added a custom SED script to further "sanitize" those examples which are intended for inclusion in the final package, by removing the region-tags comment lines. Also, we've added region tag comments to the StdLib sources, which allows us to comment excerpts from the StdLib sources in its Manual, which are also sanitized away by the scripts that deploy the library sources to the package folder. All of this might seem a bit complicate, but it's not, especially when automated (it's a set it and forget it think) and allows us to ensure that every snippet of code in the documentation (whether from the StdLib sources or from its example file) actually mirrors the latest sources, and that all the transcripts shown in the docs also mirror any changes to the messages and/or ALAN default messages and behaviour. Not only this allows us to ensure always up-to-date references, but it also makes it very easy for us to catch any significant changes in the StdLib or ALAN behaviour (whether intentional, accidental or due to a bug). So, with this system, we have ample margin to decide how each folder should be named, including adopting version based conventions, since Ruby could handle all that in simple and neat way from within Rake files. For any complex tasks, there is most likely a Ruby gem that can handle it (SemVer, Git interfacing, stream editing a la SED, etc.). But Rake already provides most of the needed functionality to create packages via its If it's OK with, I'd like to start working along these lines, e.g. to add the documentation and prepare the Rakefile, and experimenting with packaging (on dev branches). The whole idea is that to create/update distribution packages, it should be sufficient to run Rake locally (e.g. |
Beta Was this translation helpful? Give feedback.
-
@thoni56, I was thinking that we should consider this repository as the development site of the various libraries, and that we'll probably end up having a GitHub Pages website where end users can download the stable libraries and their documentations.
This implies that we should be free to add test suites to each library, in order to ensure that code tweaks and new features don't break previous one, and that we can freely use complex scripts to handle automation, and even use AsciiDoc sources for creating the final HTML (or PDF, etc.) docs that will end up in the final distribution package — all of which would, of course, make the repository less friendly to authors non involved with libraries maintenance work.
I'm already seeing that the native library documents (
INDEX.md,NOTES.md, etc.) are going to pose a challenge in terms of keeping them up-to-date with library changes, especially if we keep them in Markdown. AsciiDoc would simplify the task, especially if we embed ADoc text snippets in the actual library sources comments, and theninclude::[]them into the documentation. We can always strip those ADoc comments (or their tags) from the distributed sources (just we're doing it already in the StdLib source).That was just an example (it doesn't need to that way), but it demonstrates that separating the repository contents from the actual distributed packages allows us more freedom in terms of how to organize our developers' contents, and automating how the distribution packages are generated.
The original documents in these libraries were simple plaintext files, which are not so exciting, but probably are better for end users than a markdown document which they might not have a tool to render it with. So my guess is that the ideal solution would be to deliver any docs in HTML format, since everyone has a browser, and AsciiDoc definitely delivers nicer HTML docs, and allows us to centralize many shared snippets and attributes thanks to
include::[]directives (not to mention that we already have a good experience with Asciidoctor, and many ready-made custom resources to exploit).Do you agree that this should be the general direction this repository is going to take? (details being defined as we go along, toward our first v1.0.0 release).
Beta Was this translation helpful? Give feedback.
All reactions