Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: inbrowser.link support #14

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Conversation

dennis-tra
Copy link
Contributor

@dennis-tra dennis-tra commented Jan 15, 2025

@SgtPooki

I also had a look, took your commits and debugged quite a bit. First of all, when access http://inbrowser.link:443/ipns/filecoin.io I get a 400 Bad Request in Safari and with curl while it works just fine with Chrome. However, I noticed that the headless chrome seems to also have trouble resolving the http version of the site. I temporarily changed the URL construction code to always use https. We will need to be smarter though because https won't work with the local kubo instance. Maybe we can just hard-code inbrowser.link and use https in this case 🤷‍♂️

Then the page gets loaded successfully and also the service workers get registered 👍 However, now I have trouble running the measurement javascript because I get: Execution context was destroyed.

I suppose this is due to the redirection from the service worker. I might evaluate some JS before the redirection and some after? If I add a sleep in there it works. I would like to wait for an event though.

Lastly, what will we eventually measure? Is it the page load time after the service workers have redirected? I would have thought we want to measure the timing from accessing http://inbrowser.link:443/ipns/filecoin.io until we end up https://filecoin-io.ipns.inbrowser.link/ and have rendered the page. Because of the execution context interruption I'm not sure that this is possible.

I'll stop here for now and let you chime in 👍

@dennis-tra dennis-tra force-pushed the add-inbrowser-link-dennis branch from 508ca85 to 037ba69 Compare January 15, 2025 14:02
@SgtPooki
Copy link

I suppose this is due to the redirection from the service worker. I might evaluate some JS before the redirection and some after? If I add a sleep in there it works. I would like to wait for an event though.

I was hoping go-rod would allow us to listen for a service worker registration event, but I had to run some javascript in the page to get that sort of access. We should be able to wait until the service worker is registered.

Lastly, what will we eventually measure? Is it the page load time after the service workers have redirected? I would have thought we want to measure the timing from accessing http://inbrowser.link:443/ipns/filecoin.io until we end up https://filecoin-io.ipns.inbrowser.link/ and have rendered the page. Because of the execution context interruption I'm not sure that this is possible.

I think measuring from the time the site is first hit to the time the site is fully rendered is ideal. That is what we want to measure for sure... and the closest comparison to all other gateways.

The only way I can think of to make the measurement of inbrowser.link accurate is to accumulate web-vitals from the initial landing, the redirect, and the final reload. But that feels really hacky. I think we may need to make some changes to the service-worker-gateway in order to facilitate tiros' testing, but I'm not a big fan of that idea either.

@dennis-tra
Copy link
Contributor Author

dennis-tra commented Jan 16, 2025

I was hoping go-rod would allow us to listen for a service worker registration event, but I had to run some javascript in the page to get that sort of access. We should be able to wait until the service worker is registered.

Just to clarify, that's what I'm doing in this PR in probe.go#L268. So it is possible 👍

I think measuring from the time the site is first hit to the time the site is fully rendered is ideal. That is what we want to measure for sure... and the closest comparison to all other gateways.

The only way I can think of to make the measurement of inbrowser.link accurate is to accumulate web-vitals from the initial landing, the redirect, and the final reload. But that feels really hacky. I think we may need to make some changes to the service-worker-gateway in order to facilitate tiros' testing, but I'm not a big fan of that idea either.

We could measure the time until the redirect to https://filecoin-io.ipns.inbrowser.link/ happens on the application side and then add that time to the web-vitals values from the final page load? Feels also hacky ...

@SgtPooki
Copy link

We could measure the time until the redirect to https://filecoin-io.ipns.inbrowser.link/ happens on the application side and then add that time to the web-vitals values from the final page load? Feels also hacky ...

Can we listen for the page's "context destroyed" state? Because that's likely when the reload happens.. It would be nice to listen more directly for that reload though.

@dennis-tra
Copy link
Contributor Author

dennis-tra commented Jan 21, 2025

@SgtPooki I think I found an event that we can listen for. If you check out the latest commit here, this will be the output:

time="2025-01-21T17:47:15+01:00" level=debug msg="Starting telemetry endpoint" addr="localhost:6666"
time="2025-01-21T17:47:15+01:00" level=info msg="Starting Tiros run..."
time="2025-01-21T17:47:15+01:00" level=info msg="Initializing database client" host=localhost name=tiros_test port=5432 ssl=disable user=tiros_test
time="2025-01-21T17:47:15+01:00" level=debug msg="Created temporary directory" dir=/var/folders/5v/11vhc3kx5yxc2kr1whd5hyxc0000gn/T/tiros3510657117
time="2025-01-21T17:47:15+01:00" level=info msg="Inserting Run..."
time="2025-01-21T17:47:15+01:00" level=info msg="Starting run!" settleTimes="[0]" times=1 websites="[specs.ipfs.tech]"
time="2025-01-21T17:47:15+01:00" level=info msg="Awaiting Provider or Probe result..."
time="2025-01-21T17:47:15+01:00" level=info msg="Searching for providers done!"
time="2025-01-21T17:47:15+01:00" level=info msg="Awaiting Provider or Probe result..."
time="2025-01-21T17:47:15+01:00" level=info msg="Letting the IPFS implementation settle for 0s\n"
time="2025-01-21T17:47:15+01:00" level=info msg="Start probing specs.ipfs.tech IPFS"
time="2025-01-21T17:47:15+01:00" level=debug msg="Connecting to browser..." url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:15+01:00" level=debug msg="Initialize incognito browser" url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:15+01:00" level=debug msg="Clearing browser cookies" url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:15+01:00" level=debug msg="Opening new page" url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:15+01:00" level=debug msg="Attaching javascript" url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:15+01:00" level=debug msg="Attaching redirect listener" url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:15+01:00" level=debug msg="Attaching navigation listener" url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:15+01:00" level=debug msg=Navigating... timeout=15s url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:15+01:00" level=debug msg="Waiting for onload event ..." timeout=15s url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
Navigated to: https://inbrowser.link/?helia-sw=/ipns/specs.ipfs.tech (Navigation)
time="2025-01-21T17:47:15+01:00" level=debug msg="Waiting for network idle event ..." timeout=15s url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:15+01:00" level=debug msg="Waiting for service worker registration ..." timeout=15s url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:15+01:00" level=debug msg="Register [] service workers\n" timeout=15s url="https://inbrowser.link/?helia-sw=/ipns/specs.ipfs.tech"
Redirect detected: https://specs-ipfs-tech.ipns.inbrowser.link/
Navigated to: https://specs-ipfs-tech.ipns.inbrowser.link/ (Navigation)
Navigated to: https://inbrowser.link/ (Navigation)
Redirect detected: https://specs-ipfs-tech.ipns.inbrowser.link/
time="2025-01-21T17:47:17+01:00" level=debug msg="Running polyfill JS ..." attached=true url="https://inbrowser.link/?helia-sw=/ipns/specs.ipfs.tech"
Navigated to: https://specs-ipfs-tech.ipns.inbrowser.link/ (Navigation)
time="2025-01-21T17:47:17+01:00" level=debug msg="Running measurement ..." attached=true url="https://inbrowser.link/?helia-sw=/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:23+01:00" level=debug msg="Getting Performance Entry measurement ..." url="https://inbrowser.link/?helia-sw=/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:23+01:00" level=debug msg="Closing page..." url="https://inbrowser.link/?helia-sw=/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:23+01:00" level=debug msg="Closing browser connection..." url="https://inbrowser.link/?helia-sw=/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:23+01:00" level=info msg="Probed website specs.ipfs.tech" error="<nil>" fcp=286 lcp=408.89999997615814 status=200 ttfb=164.69999998807907 tti=173
time="2025-01-21T17:47:23+01:00" level=info msg="Handling probe result" url="https://inbrowser.link:443/ipns/specs.ipfs.tech"
time="2025-01-21T17:47:23+01:00" level=info msg="Awaiting Provider or Probe result..."
time="2025-01-21T17:47:23+01:00" level=info msg="Probing websites done!"
time="2025-01-21T17:47:23+01:00" level=info msg="Stopped Tiros run."
time="2025-01-21T17:47:23+01:00" level=info msg="Tiros stopped."

I'm listening for these events:

	go p.page.EachEvent(func(e *proto.NetworkRequestWillBeSent) {
		if e.Request.URL == "https://specs-ipfs-tech.ipns.inbrowser.link/" {
			fmt.Printf("Redirect detected: %s\n", e.Request.URL)
			p.navigated <- struct{}{}
		}
	})()

The second time I see a redirect to https://specs-ipfs-tech.ipns.inbrowser.link/ I start the measurement. This consistently works. However, why are there two redirections?

Unfortunately, I only had like half an hour to look into this. I'll continue throughout the week 👍

@dennis-tra dennis-tra force-pushed the add-inbrowser-link-dennis branch from 40c2dca to 971ad41 Compare January 23, 2025 11:44
// mustNavigateServiceWorker listens to navigation and request events to
// validate the navigation process. Events are matched against an expected
// pattern to confirm a successful operation.
func (p *probe) mustNavigateServiceWorker() {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SgtPooki check out this method. It's a bit convoluted but it works 👍

@dennis-tra
Copy link
Contributor Author

@SgtPooki I just updated this PR with a working version of the service worker measurement. I promoted the service worker configuration on the same level as HTTP and IPFS measurements. This made the most sense to me.

Then there's a new method called mustNavigateServiceWorker that listens for navigation and requestWillBeSent events. I observed that in an incognito window when opening https://inbrowser.link/ipns/specs.ipfs.tech I get the following redirects:

So what I'm tracking in this method is the navigation event to https://inbrowser.link/ and then the requestWillBeSent for the final https://specs-ipfs-tech.ipns.inbrowser.link/. The latter event is also the offset that I add to the final measurements because I presume that this is the timestamp that web-vitals will measure against. The time from the first navigation to this timestamp (which I called finalRequestStarted in the code) is the time that lidel asked for I think (I'm not persisting that time yet).

Please have a look if this makes sense and I could work on a deployment perhaps next week.

@SgtPooki
Copy link

SgtPooki commented Jan 23, 2025

The second time I see a redirect to https://specs-ipfs-tech.ipns.inbrowser.link/ I start the measurement. This consistently works. However, why are there two redirections?

This is because the SW/main thread on the first hit to /ipns/... redirects to the subdomain, and if the SW isn't registered there, the _redirects file sends you to {root Domain}?helia-sw={splat}, which then triggers the registration logic.. and then once that SW is registered, it redirects/reloads the actual subdomain that is then captured by the newly registered service worker.

It may be beneficial for us to take another look at how we're registering the SW on the subdomain here so that there's not so much bouncing, but that was how we decided to get things working initially.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants