Browser support

[sce9sc]

Is there any plans to fix and support on browser?

[zacharygriffee]

I have been using hypercore and much of the hyper-ecosystem extensively in browser with minimal issues. Easiest solution for me earilier on was to use browserify. But now, I use rollup .

Edit: But rollup comes with a bit more complexity, so I'd suggest doing browserify way.

[sce9sc]

Thank you for your reply.

I tried what you mentioned and below I have an example running on browser using browserify

const RAM = require("random-access-memory")
const Corestore = require('corestore')
const RAW = require('random-access-web-storage')

const { pipeline } = require('streamx')


start()

async function start () {
    try{
    const store1 = new Corestore(RAW(localStorage))
    const store2 = new Corestore(RAM)
  
    await store1.ready()
    await store2.ready()

    console.log('11111')


    const core1 = store1.get({ name: 'core-1' })
    const core2 = store1.get({ name: 'core-2' })
    await core1.ready()  
    await core2.ready() 



    await core1.append('hello')
    await core2.append('world')

        const r1 = await core1.get(0)
    const r2 = await core2.get(0)

    console.log('r1',r1)
    console.log('r2',r2)


    console.log('22222', core1.key, core2.key )
  
    const core3 = store2.get({ key: core1.key })
    const core4 = store2.get({ key: core2.key })

    await core3.ready()  
    await core4.ready()    

    const s1 = store1.replicate(true)
    const s2 = store2.replicate(false)

    console.log('3333', s1,s2)
  
    // s1.pipe(s2).pipe(s1)

    pipeline(s1,s2,s1,(e)=>{console.log('errorrrr', e)})
    console.log('4444444 pipeline')

    const r3 = await core3.get(0)

    const r4 = await core4.get(0)

    console.log('55555555')

    console.log('r3',r3)
    console.log('r4',r4)

  return store1
    }catch(e){
        console.log('eeeeeee',e)
    }
}



module.exports = start

The problem is that I get

Noise handshake failed
    at NoiseSecretStream._onhandshakert (bundle.js?t=1699883966724:403:33)
    at NoiseSecretStream._open (bundle.js?t=1699883966724:451:32)
    at WritableState.updateNonPrimary (bundle.js?t=1699883966724:21823:14)
    at WritableState.update (bundle.js?t=1699883966724:21798:72)
    at WritableState.updateWriteNT (bundle.js?t=1699883966724:22147:10)

when trying to replicate .

[zacharygriffee]

Yeah, I could never get any noise streams to pipe to each other through browser. So, to do any kind of replicating, I use the 'dht relay socket' library.

[zacharygriffee]

Just to go on, I think the reason is, only so many cryptography functions from sodium library (e.g. noise algos) works in browser (javascript) where it has to be 'handled' by a microservice thus the relay. And I also believe that is why its so hard for p2p to get 'browser surface'. Check out the failed p2p browser beaker.

[sOniri]

I found the problem . In @hyperswarm/secret-stream if you go to lib/handshake.js in line 12

this.noise = new Noise(pattern, isInitiator, keyPair, { curve })

I think if you remove the curve it will work

[zacharygriffee]

MY expertise in cryptography is deep enough to understand and implement, but still would hope someone more specialized in cyber security would make the nitty gritty parts of it. But removing that noise would most likely remove it's ability to replicate to common peers (anyone that doesn't have that version that removes the curve). And I'm pretty sure you'll remove the 'append only' concept to the hypercore as well.

[jermsam]

I get this error when on the line that tries to call input.ready and output.ready when I do this:

import Corestore from 'corestore';
import random from 'random-access-idb';

const fileName = 'cool';
const storage = random(fileName);

const store = new Corestore(storage);

 const input = store.get({name: 'input'});
  const output = store.get({name: 'output', valueEncoding: 'json'});
  goodbye(async () => {
    await Promise.all([input.close(), output.close()]);
  });

  await Promise.all([input.ready(), output.ready()]); // Error:  this.storage.truncate is not a function  #oplog.js:171:22
  

[zacharygriffee]

Old version of random-access-idb... I decided to create my own repo with mods I've used.

Here is mine: https://github.com/zacharygriffee/random-access-idb

I also found another group trying to update it as well here:

https://github.com/random-access-storage/random-access-idb

The one you get from npm e.g. "npm i random-access-idb" is old.

[zacharygriffee]

I did want to mention, as I just thought about eating dinner, during 'development' when testing out these tools, I utilize cdn esm.run (e.g. cdn.jsdelivr.net/npm/hyperswarm/+esm) to then import either by importmap or directly in type=module script element. Just load a relay on your machine with port forwarding to that relay. This doesn't even need an http server to run***, just plop right into an HTML file. Saves me the bundling until production.

*** The html file doesn't need http to run. The relay will need http for upgrade.

<body>
<script type="importmap">
{
    "imports": {
        "hypercore": "https://esm.run/hypercore",
        "ram": "https://esm.run/random-access-memory",
        "b4a": "https://esm.run/b4a",
        "dht": "https://esm.run/@hyperswarm/dht-relay",
        "ws-stream": "https://esm.run/@hyperswarm/dht-relay/ws",
        "hyperswarm": "https://esm.run/hyperswarm"
    }
}
</script>

<script type="module">
    import DHT from "dht";
    import Stream from "ws-stream";
    import Hyperswarm from "hyperswarm";
    import b4a from "b4a";

    const socket = new WebSocket('ws:///** your relay link**/"');
    const dht = new DHT(new Stream(true, socket));
    const swarm = new Hyperswarm({dht});

    swarm.on("connection", socket => {
        // replicate your core
    });

    swarm.join(b4a.fill(b4a.alloc(32), "piccard"));
</script>
</body>

[sOniri]

@zacharygriffee Very nice example .

I was wondering since you are using Hyperswarm Relay

eg npm install -g @hyperswarm/dht-relay

dht-relay # [--port 49443] [--host 0.0.0.0] [--cert <path fullchain.pem>] [--key <path privkey.pem>]

Do you have any idea how this can scale (horizontally)?

[zacharygriffee]

Do you have any idea how this can scale (horizontally)?

This is what I've done in attempt to scale horizontally.

1: I've implemented the relay in a digital ocean, and a few computers at home.
2: Create electron app you can distribute with the relay embedded. They can select how many connections they want to relay, known nodes, bootstraps, and add some throttles and what not. Similar to bittorrent settings panel/controls.
3: Reward their contribution through gas (Ethereum/Bitcoin validator like), and reputation (the p2p spirit).
4: Utilize rest api for some supporting services to get the swarm growth and have a source of truth like Warrant API or an autobased hyperbee controlled by your staff.
5: Have an 'always up' micro service (digital ocean or home computer) that holds an updated list of relays so that each time the browser user loads their page they will get the 'random relay'. I wouldn't even distribute a relay if it can't keep at least 90% uptime. The dht relay doesn't support mutable/immutable records but your microservice can post it for other microservices in your cluster.

As a note @hyperswarm/dht-relay github does say its not ready for production use, but I've been getting pretty good results with it.

Hope this helps. I know its not a simple direct answer. But this may help you devise a stack to work with the browser limitations. If you got any thoughts as well please share.

[sOniri]

If you got any thoughts as well please share.

Sorry for the late reply but I had to do some tests and found some issues . One of them is the fact that I could not make a private swarm. Secondly I wanted to be able for my internal network to create a swarm and that swarm does not have an internet connection. Thirdly I want to be able to connect both my internal and external networks.

To put the above in more detail . I have 3 or more desktop apps that need to connect internally using the home network. Also I have mobile devices that need also to be able to connect to those desktop apps via either a relay hosted on VPS or via an internal relay hosted on one of the desktop apps. The mobile + desktop connectiviy should happen either when any of them loose internet connectivity but connected to the same local network.

What I found was that if you do not have internet connectivity even on your local bootstrap dht-relay server you cannot connect. Another problem also is that mobile devices when they have limited internet connectivity from the local wifi they tend to switch to mobile carrier and thus loosing the connectiviy to the local network.

Thus I found that the best approach is to create your own relay server for the internal home network and connect to both hyperswarm and own relay-server.

Also from some tests that I've made 99.9 percent when using the hyperswarm from my local network the connections made between the desktop apps where all through relay servers and thus if you did not have any internet you could not join them.

Overall I found that, for internal home network use, Desktop apps and Mobile devices should connect together using use a local relay-server(your own not the one provided), together with the hyperswarm network or hyperswarm dht-relay.

Another interesting thing is that I could not find anywhere how to build my own boostrap server like 'node1.hyperdht.org' together with the relay. If anyone has an idea please share. ( and I don't mean the example that is included in the hyperdht repo. )

Thanks @zacharygriffee for your reply

[zacharygriffee]

Have to have internet.

Sadly I haven't found a way around that, and I have even raised issue at the @holepunchto/hyperswarm-dht-relay here at the beginning of the year. Also mentioned here. I felt that a private network independent of the internet would be invaluable for SOOO many things. I didn't dig too hard into the code with that, maybe I need to do it. But the plate full blah blah.

Self hosted swarm / bootstrap

I have some personal test cases buried on another laptop, however, for now, have you checked the test cases on HyperDHT as well? HyperDHT 'testnet' implements a 'private' network for their tests:

https://github.com/holepunchto/hyperdht/blob/main/testnet.js

Line 8, 13, 15

Aside from the testnet, the few things I found to be important in my tests was:

• The DHT.bootstrapper() convenience creates a dht node with an empty array of bootstrap nodes: [] . I ran a redundant bootstrapper node on another computer, so I had each other in that list of bootstrap nodes.
• Your other nodes need to point to the bootstrapper and the other known nodes you have so they can gossip options.bootstrap: [host:port, host:port] .
• The DHT.bootstrapper() does this, but make sure options.ephemeral is set to false.
• I needed port forwarding on the router the bootstrapper node is behind.
• I needed port forwarding on the router the relay node is behind as well.
• I had to make sure firewall wasn't blocking on both the bootstrapper and relay.

If all this doesn't work I can dig into my other laptop where I had a successful one. But it was from a older version of the dht but it should still be applicable.

mobile devices when they have limited internet connectivity from the local wifi they tend to switch to mobile carrier and thus loosing the connectiviy to the local network.

I have actually not tested network switching of a mobile phone. That is something that will be on my moderately important TODO list and report back. Cause, that is important for me as well. I have not had any problems switching networks with a laptop for example but have not tried mobile phone.

Note to mods: I know this discussion is weakly related to the hypercore problem and strongly related to the hyperDHT. But I feel really hard to find exactly where to raise these issues when they're so interconnected other than discord

[zacharygriffee]

Another thing. This may go without saying but thought worth the mention. Regarding network switching. During testing, I have it set where the 'webpage' I have connecting to the services has two different ip to connect to.

1: The local area network ip: ws://192.168.x.x:port/path
2: The public ip with the 'port forwarding set': ip ws://yourPublicIp:portforwardedPort/path

So when the socket closes because of the network switch, it will attempt both ip's. So when you loose connection to the local it will try the public.

[zacharygriffee]

I use a jst-like-templating with custom seperators so I can add to the url a custom relay address with testing as well.

const {
    concurrency = Math.max((navigator.hardwareConcurrency || 6) - 2, 4),
    relayAddress = "/*=address*/" || "ws://127.0.0.1:[port]/hyper",
    userSeedKey = "identity",
    pageInsertionElementName = "root",
    notificationInsertionElementName = "toast",
    entropyCollectionDuration = 20000
} = window.location.href
    .split("?")
    .pop()
    .split("&")
    .reduce((acc, exp) => {
        const [key, value = true] = exp.split("=");
        if (Number(value)) {
            acc[key] = Number(value);
        } else if (value === "false") {
            acc[key] = false;
        } else if (value === "true") {
            acc[key] = true;
        } else {
            acc[key] = value;
        }
        return acc;
    }, {});

[zacharygriffee]

Also regrding the bootstrapper node, I found examples here specifically targeting the bootstrapper (private swarm) at dht-rpc