Suggestion: Link back to Github for github-fetched generated docs

[jrochkind]

I suggested (https://rubydoc.tenderapp.com/discussions/suggestions/4-suggestion-links-back-to-github) that http://rubydoc.info ought to do this.

Then lsegal suggested it would be best to do this with Javascript. That seemed an odd way to do it, but I said, okay, give it a shot.

Then I wrote this javascript that seems to work pretty darn well if it's applied to any rubydoc.info page in the github area. I just wrote/tested it 'greasemonkey' style, I didn't try running rubydoc.info locally and putting it into source myself.

https://gist.github.com/1305162

Then I realized that in fact these days the majority of my rubygems.info use is for 'gem' area not 'github' area, and this wouldn't do anything for me there, even if the gem source was in github. Then I got depressed and lost interest in spending TOO much more time on this.

Then I noticed that the JS code I wrote conflicts with your JS code for hide/show source. Only because of where it inserts the "link to github" link into the DOM I think. Either it needs to be changed to insert into the DOM somewhere else, or the hide/show source JS needs to be changed to be more precise in it's selectors instead of blindly assuming whatever's next() in the DOM after it is the source.

Now I file a ticket here with my JS code, as lsegal suggested, in case anyone else wants to run with it. I think I'm done. It is pretty cool though! I might make a little JS bookmarklet for myself to use this code, if it doesn't make it into rubydoc.info deployed.

[lsegal]

It's unfortunate that we can't make this work for gems. I can't imagine how it would work, though. Maybe if there was some kind of dotfile that was in the project (maybe we can add metadata to yardopts somehow) or if the gemspec could actually specify this stuff (the gemspec can contain metadata). But users would have to opt in, and it would be hard to track / maintain. You would have to point to the tag / commit of the code in the gem, which isn't easily mappable. As I mentioned, we could not just assume master.

[lsegal]

Also I just made this change in yard to do better filtering on the "next()" element so that your greasemonkey style script should now work properly, and it's deployed on rubydoc.info.

[jrochkind]

Is there even any way to identify if a gem does come from github, and
what repo? If there's a way to identify that, then in many cases
assuming a tag that matches the version, as recommended by semantic
versioning would work. (http://semver.org/ ). eg: v3.1.0

In many but not all cases, but enough that it might be worth it at
generation time, the generator could just HEAD request on https://github.com/owner/project/tree/v3.1.0
and see if it gets a 404 or a 200. If it gets a 200, tag found,
github links can be included. If not, not. (I do LOVE it that not
only are you harvesting gems, but you keep old versions rdoc around!
And old versions could have links to github too using this scheme,
neato.)

But prerequisite is being able to tell from the gemspec what the
github repo is at all. I don't know if gemspecs currently include
that; Theoretically it could be added as an optional field to the
gemspec, but perhaps they are still smarting from making us all
include rubyforge_project as non-optional after most had stopped using
it. Now, it is a metadata field on rubygems.org itself ( "source
code", easy enough to see if it begins https://github.com"), but I
don't know if metadata fields on rubygems.org that aren't included in
the gemspec are available through your gem harvesting?

[jrochkind]

PS: If you release with the "bundle release" rake task, it adds a "semantic versioning" style tag to your repo for you. Plus many people do it intentionally. I think a great many gems (although certainly not all, esp ones whose latest release is old) will have semver-style tags in the repo. That part doesn't actually seem as hard as identifying the repo!

[lsegal]

I just did this kind of stuff last week, and it turns out that it's really hard to get the source code URL from rubygems.org, and even harder from a gemspec. We actually ran this across 140 gems and only got back 80. I guess that's either bad or not bad depending on your viewpoint.

We could do it that way...

Note however that yard doesn't follow the "vX.Y.Z" scheme, we drop the "v" (we were tagging way before this "semver" stuff existed :)), but that's an easy heuristic check.

[lsegal]

The problem here is two fold though:

1. We'd need to reprocess all gems to get this data and/or we'd need to build out some db to support it. That is kind of painful and messy (reprocess in the case of HTML, DB in the case of JS).
2. From an ideological POV I'm not sure it's consistent to link to github "sometimes", especially since our heuristic will not guarantee that the failure case means there is no associated github repo. We open ourselves up to a lot of gem authors complaining that their gem does or doesn't have links to the github repo. This is all to be seen, though.

[jrochkind]

You could HEAD for both (inc with the "v." dropped), or you could just say "use semver if you want auto git links (and fix yard! :) ).

But the harder thing is clearly getting the github url in the first place. Now, from my perspective as a user having github links work on just over half of gems (if 80 out of 140 was representative) would still be exactly halfway better than none having links back to github! But it's whether it's worth the coding or not just to get those, is up to you. I don't know if your methods including looking at gemspec 'homepage' to see if it is set and just happens to be https://github.com/.... , but that would get some gems, which have their home pages set there.

It is not surprising to me that it's hard to get it from the gemspec alone, since, sadly, there is no gemspec param for it. It is surprising to me you can't get it from rubygems.org itself, since it's a clearly specified seperate field in the gem admin interface. They don't have an api for pulling it out? Well, you told me before that one could call it 'using the DOM' instead of 'screen scraping' and just get it from the HTML representation instead then (heh, mostly kidding).

Also, did I mention I really do love rubydoc.info? Also, now that I've noticed you are keeping multiple versions of auto-generated-from-gemspec docs around, of course another feature occurs to me -- apidock.com-style compare-and-contrast different versions of the api for a given gem. But that's a different issue, and a non-trivial feature to do as well as apidock does. (But now I'm stuck going to apidock when I need to compare-and-contrast rails features; going to api.rubyonrails.org when I want the links to github that it's got (yep, they've got em); and going to rubydoc.info for everything non-rails cause, well, you've got it, and in a very usable website. In my dreams, there's one site that does all that, for all gems, not just rails! )

[jrochkind]

I understand that is may end up a technical challenge for you, that's your call if it's not worth it of course, it's your resources.(for real, respect).

And yeah, I agree that success would need to be reasonably predictable to keep gem authors from complaining. But it'd probably be okay if it entirley predictable based on gem authors following reasonable and documented conventions:

1. List your repo in rubygems.org metadata (yeah, you've got to go log in to rubygems and list it manually)
2. use semver for your tags. (You can use something else too if you want, as long as server tags are there).

You do both these things, you get github links! If not, not! I think that would keep gem authors satisfied.

But if there's no good way to get the repo from rubygems.org metadata, forget it, yeah. And again, I can respect if there aren't the resources to do it now. (But could just do it for new generations, not go back and re-do existing ones. But yeah, you'd need some kind of 'database', even if that's just dropping a dotfile in the filesystem somewhere, depending on how your back end filesystem works, not familiar with it. But yes, I understand this is non-trivial and may not be a priority. I'm just saying it would be sweet!)

[jrochkind]

(It's definitely documented avail from rubygems.org api, but you probably know that better than me. http://guides.rubygems.org/rubygems-org-api/#gem Understand that many authors may not fill that out. Don't think anyone would complain if it was clearly documented 'fill that out if you want auto-generated rubydoc.info github links'. Might even lead to more people filling it out, if it was popular. )

[mpapis]

hmm, this might be related rubygems/rubygems.org#724

[kbrock]

thanks for looking into this stuff and working on that last 10% to making rubydocs more usable.