New HexDocs home page

[josevalim]

Today's home page is fully done in the client, we should probably mirror that here.

Overall, the main feature we want to have is full-text search across (at least) one or more packages. You can add packages by typing # which will trigger an autocomplete, such as #phoenix_live_view. You can then press : to autocomplete a given version, otherwise the latest is picked from the hex.pm API. If the search only has the package information, such as #ecto or #ecto:1.0.0, and no search term, we go to the particular package (+ version).

Anything else we could do when it comes to search? /cc @ericmj @ruslandoga @wojtekmach

I will work on a mockup to the new homepage. The mix hex.search and ExDoc can come later.

[josevalim]

Once we submit the search, we should go to a page like this: https://claude.site/artifacts/3324d578-85b1-411c-accd-c002f37f301b

I am currently in touch with Livebook's designer so he can mockup the new home page plus search. @paulo-valim made himself available to implement the designs as HTML + Tailwind. We already have endpoints in Hex for:

1. Package autocompletion
2. Docs search (coming with Expose the typesense server via search.hexdocs.pm #47)
3. And another for creating short URLs

We will just need someone to implement the front-end bits. I will post all future updates here.

[josevalim]

@ruslandoga I am wondering: should we also index the most recent versions of packages as phoenix-latest, so it is easy for us to always query the latest? In the same way that hexdocs already tracks that hexdocs.pm/phoenix means latest?

[josevalim]

Another potential feature we want to borrow from ExDoc is using title:, doc: and type: to scope search. Here are all features available in ExDoc search:

• Multiple words (such as foo bar) are searched as OR
• Use * anywhere (such as fo*) as wildcard
• Use + before a word (such as +foo) to make its presence required
• Use - before a word (such as -foo) to make its absence required
• Use : to search on a particular field (such as field:word). The available fields are title, doc and type
• Use WORD^NUMBER (such as foo^2) to boost the given word
• Use WORD~NUMBER (such as foo~2) to do a search with edit distance on word

[josevalim]

@ruslandoga @ericmj @wojtekmach @supersimple I just want to share with you the new designs for the home page:

And the package search functionality:

We will work on converting those to HTML, then I will let you know once that's ready :)

[supersimple]

Looks good. I know at least one person that will be happy ;)

[lpil]

Hello everyone! @ghivert may be able to help out here with the frontend with Gleam+Lustre.

P.S. Love the design there, the homepage especially looks really nice.

[ghivert]

Hi everyone! @lpil told me about a potential rewrite with Lustre in Gleam.
I’d be happy to help on the subject and implement the frontend part!

If I understand everything correctly, the backend part is already written, so the main task is only to implement the mockup as a real web app? Is there some requirements like SSR, or anything else?

In case you have a Figma/Sketch/whatever format for the mockups, maybe it could directly translate into a Lustre + Tailwind?

[josevalim]

Oh! Being to able to keep it within the BEAM community would be fantastic, thanks for chiming in @lpil and @ghivert!

If I understand everything correctly, the backend part is already written, so the main task is only to implement the mockup as a real web app? Is there some requirements like SSR, or anything else?

Yes, the backend is existing Hex APIs and the newly added Search API. Our goal is also to serve a static page, so no SSR. We probably want the homepage (the first screen) to be indexable though but I guess that should be doable. And I was told most engines index using a JS engine anyway?

In case you have a Figma/Sketch/whatever format for the mockups, maybe it could directly translate into a Lustre + Tailwind?

We do have one. We also have someone who can do the translation from Figma mockup to HTML + Tailwind. Or would you rather tackle it too?

This is very exciting!

[ghivert]

I'm really happy to be able to help!

Yes, the backend is existing Hex APIs and the newly added Search API. Our goal is also to serve a static page, so no SSR. We probably want the homepage (the first screen) to be indexable though but I guess that should be doable. And I was told most engines index using a JS engine anyway?

What we could do is performing some SSG in that case: no full-fledges SSR, but the HTML statically served contains the initial render. So it could be nicely indexed by search engines, and client-side runtimes will then take care of rendering the content after that (with hydration). I think it makes sense at least for the homepage to be pre-rendered, to improve search engine indexing.
Also, Google is now indexing web sites with a JS engine enabled. I'm not sure Bing is doing it too however.

I think the SSR question is also related to the support of the Open Graph Protocol for Hex. Without SSR, it's not possible to have proper Open Graph tags. With OG tags, we could provide nice displays on some applications, like Discord or Slack when linking HexDocs URL. I'm not sure it's really useful, and it will be more work.

I think it's possible to start with a first implementation entirely client-side, and potentially migrate server-side later if needed (if things like OG tags are required).

We do have one. We also have someone who can do the translation from Figma mockup to HTML + Tailwind. Or would you rather tackle it too?

To be fully transparent on that point, Tailwind is not my favorite tech. If you have someone who's expert with (or at least used to), I think it can be a way to gain some time, but they're too much busy, I can tackle it too. 🙂

Talking about time, do you have some deadline for a working implementation?

[josevalim]

To be fully transparent on that point, Tailwind is not my favorite tech. If you have someone who's expert with (or at least used to), I think it can be a way to gain some time, but they're too much busy, I can tackle it too. 🙂

We are open to alternatives, if you have any mind. :) I suggested Tailwind because it is most likely others will be familiar with it. @paulo-valim made himself available to translate to Tailwind but that will probably only happen within a month. We don't have a deadline per se, given it is all open source.

[josevalim]

Correction, @paulo-valim has already started working on HTML + Tailwind, so we should have something as base sooner than later. :)

[ghivert]

I'm on the pure CSS side of the force! I consider CSS simpler than any framework, but as any take, it's highly debatable 😉

If @paulo-valim started working on it, I'd suggest to wait for him to finish, and then implement the HTML in the Lustre code. That will be less work UI-related and more time to properly implement the flows!

Thanks for your answers about the deadlines.
I can start taking a look this week-end. And experimenting with the search API if it's available?

[josevalim]

I can start taking a look this week-end. And experimenting with the search API if it's available?

Perfect. We will work on getting the API descriptions for you before the weekend (hopefully tomorrow!).

[ghivert]

I can start taking a look this week-end. And experimenting with the search API if it's available?

Perfect. We will work on getting the API descriptions for you before the weekend (hopefully tomorrow!).

Thanks ! No worry though, if you need some time, I can still postpone experimentation (I thought API descriptions were already written). Do not crush for me! 🙂

[wojtekmach]

Hey everyone, some updates:

1. Currently hexdocs.pm homepage search autocomplete hits hexdocs.pm/sitemap.xml which is inefficient as it contains irrelevant data for what we require for new search. We're adding an endpoint https://hexdocs.pm/package_names.csv which contains sorted list of package names, e.g.:

package1
package2

etc.

2. We're working on exposing Typesense which will be available at search.hex.pm search.hexdocs.pm.

3. We can now create short links for hexdocs.pm:

$ mix hex.user key generate
Username: wojtekmach
Account password: ***
Generating key...
***
$ curl -H "authorization: $HEX_API_KEY" https://hex.pm/api/short_url --json '{"url":"https://hexdocs.pm/#test"}'
{"url":"https://hex.pm/l/EmTm6"}

This will be relevant when based on list of selected packages we'll generate a "short ink" for https://hexdocs.pm/#{search}.

I'll post here once 1. and 2. are done. Hopefully this is enough information to iterate on the frontend that will consume those endpoints, let me know otherwise!

[josevalim]

Oh, the short links API requires a user token? So let's remove this functionality from the proof of concept for now. So overall:

1. A listing of all packages will come from a CSV file, with packages sorted one per line separated by \n

2. Once you find a package, you can get its versions from https://hex.pm/api/packages/PACKAGE_NAME

3. The search endpoint will be exposed as Expose the typesense server via search.hexdocs.pm #47 - for now you can use https://search.staging.hexdocs.pm/?q= and it points to a TypeSense collection

4. We won't have the share feature for now. However, we probably don't need to? If whenever we add a package+version to the sidebar, we update the URL, we are good to go. We can do ?p[phoenix]=1.5.6&p[gleam]=1.3.4&q=hello+world and so on

Please let us know if you have any questions @ghivert!

[ghivert]

Hi!
I'm really sorry for being so long to answer, I had some personal & professional issues to handle urgently. 😳 But I'm back on track.

I started playing with the different API. I implemented a quick PoC with Lustre, that you can find here.

That first draft is made of two pages:

1. The first one would be the home page, able to provide an autocomplete on packages, and can query the different package versions if needed.
2. The second page is able to get the results from the TypeSense API, by searching for a term. It is also able to search for specific packages in the search (it's tested and working).

Because a demo is always nice, here's two short videos to illustrate the possible actions:

Details

https://github.com/user-attachments/assets/878ecc3f-0da1-460e-ae99-598eb2e4308b
https://github.com/user-attachments/assets/805e04f8-ac9e-4f8d-99f3-510f07f1efff

---

Because I was unsure of the user flows we desired for HexDocs homepage, I prefered to stay simple and implement the different parts of the logic to be able to reuse them later with the different screens and flow.
Because paulo-valim is making the HTML, I also stuck with simple plain HTML. Of course, we can expect better screens in the end. 😄 I just wanted to advance fast on the subject to check if that's what you're looking for.

Nowadays the code is living in a separate repo for the PoC, but I suppose it should also integrate HexDocs repo directly? In that case I think it's wiser to let you tell me where to integrate and how, but I can also take a look and find an appropriate location. 😄

---

Everything here is a bit rough and needs more polishing. Here's how I understand the user flows from the two screenshots above:

1. User connects on the home page, search for a package, autocomplete shows up, and clicking on one redirects to corresponding HexDocs page.
2. User connects on the search page, inputs packages if needed, then type a term, and we trigger a search request, trying to find the best corresponding matches. User can also type some special characters in the search bar to activated advanced search (# to specify where to search, etc.)

Is it OK? Is there anything I'm missing?

[ruslandoga]

👋 @ghivert

Just a note: https://github.com/ghivert/hexdocs-search seems to be private :) Or maybe it's something else but it returns 404 for me right now.

[ghivert]

Oh ! Thanks for warning ! I changed the visibility !

[josevalim]

Hi @ghivert! First off, no need to apologize, we are all working on our free time, so any contributions and updates are really welcome! ❤️ The demos are nice and it is an exciting preview of what is to come!

I will talk a bit about the flows but that's what I have on my mind and I am very open to suggestions.

Home page

The home page should work as a search engine. You join the website, the input is focused, and you start typing. If you type #, we will assume you want to autocomplete a package. If you add a : after the package name, we will show its versions. For example:

1. If I type "gleam", nothing shows up.
2. If I type "gleam #", then package autocompletion appears
3. If I type "gleam #lustre:", then version autocompletion for that package appears

Once you type something, you can submit the search by clicking the button or pressing enter. Doing so takes you to search page.

Search page

The search page converts the query from the previous page into something a bit more structured. Any package that you added to your input goes to the sidebar. If no version is specified, we automatically pick the latest. New packages can be added at any time, which triggers a new search. If you do a search with no packages, we should probably show a warning asking them to add packages to filter the results.

Structure

I believe the application is a SPA. As you perform queries and add packages, they should automatically change the URL, so we can easily share results with someone else.

WDYT?

Just one note, I am a Gleam noob, what is the best way to run the project? Perhaps we can add the steps in the README? I tried what is there and it didn't work. Thanks and have a great weekend!