Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support p2p retrieval by default #130

Merged
merged 36 commits into from
Nov 21, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
6a2cffa
feat: add support for p2p retrieval
2color Nov 5, 2024
5de4bff
deps: add missing deps explicitly
2color Nov 6, 2024
7bf29f3
refactor: simplify naming
2color Nov 6, 2024
ae3df4d
deps: make helia prod dep
2color Nov 7, 2024
e92b3c0
feat: simplify config and default to helia p2p
2color Nov 8, 2024
6889a6d
chore: remove unneeded dep
2color Nov 8, 2024
4aa11df
Merge branch 'main' into add-p2p
2color Nov 18, 2024
b30c55d
fix: build works
SgtPooki Nov 18, 2024
21374d4
Merge branch 'main' into add-p2p
SgtPooki Nov 18, 2024
a80278e
chore: remove unused dep
SgtPooki Nov 18, 2024
7a52f02
test: tests no longer hang
SgtPooki Nov 18, 2024
ae3d952
fix: interop tests
SgtPooki Nov 18, 2024
3de5678
fix: gateway conformance runs again
SgtPooki Nov 18, 2024
2a1afc9
chore: skip some gateway conformance tests
SgtPooki Nov 18, 2024
5f85aaf
fix: single method for getting peerId
SgtPooki Nov 18, 2024
74e27ac
fix: type error and remove identify push
2color Nov 19, 2024
e6bccdf
Revert "fix: type error and remove identify push"
2color Nov 19, 2024
af90f4d
fix: reuse dns in libp2p and helia
2color Nov 19, 2024
8dbe937
feat: config for libp2p with browser override
2color Nov 19, 2024
4ab9c99
fix: type errors
2color Nov 19, 2024
cace07f
fix: libp2p option customization fix
SgtPooki Nov 19, 2024
fe9ab4c
chore: fix unnecessary code
SgtPooki Nov 19, 2024
0916ccf
fix: eslint rule forces style that breaks types
SgtPooki Nov 19, 2024
310dd08
fix: ipns is prod dep now
SgtPooki Nov 19, 2024
30fb274
deps: bump delegated routing client
2color Nov 20, 2024
5f2f127
test: add direct-retrieval test
SgtPooki Nov 20, 2024
c8f6709
test: fix browser testing
SgtPooki Nov 20, 2024
0cc392b
fix: use matching index signature for Libp2pServices
SgtPooki Nov 20, 2024
1b876d7
chore: fix dep-check
SgtPooki Nov 20, 2024
06894f8
chore: fix ipns tests and dep-check
SgtPooki Nov 20, 2024
a21ddf1
Merge branch 'main' into add-p2p
SgtPooki Nov 20, 2024
088686e
Merge branch 'main' into add-p2p
2color Nov 21, 2024
053d245
fix: build by using relative path
2color Nov 21, 2024
226bf23
chore: simplify types
2color Nov 21, 2024
654565d
chore: remove unused dep
2color Nov 21, 2024
dc07e56
fix: use caret so that the sibling dep is used
2color Nov 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion packages/verified-fetch/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,7 @@
"dependencies": {
"@helia/block-brokers": "^4.0.0",
"@helia/car": "^4.0.0",
"@helia/delegated-routing-v1-http-api-client": "^4.1.2",
"@helia/http": "^2.0.0",
"@helia/interface": "^5.0.0",
"@helia/ipns": "^8.0.0",
Expand All @@ -155,16 +156,21 @@
"@libp2p/interface": "^2.1.3",
"@libp2p/kad-dht": "^14.0.1",
"@libp2p/peer-id": "^5.0.5",
"@libp2p/webrtc": "^5.0.16",
"@libp2p/websockets": "^9.0.11",
"@libp2p/webtransport": "^5.0.16",
"@multiformats/dns": "^1.0.6",
"cborg": "^4.2.4",
"hashlru": "^2.3.0",
"helia": "^5.1.0",
"interface-blockstore": "^5.3.1",
"interface-datastore": "^8.3.1",
"ipfs-unixfs-exporter": "^13.6.1",
"it-map": "^3.1.1",
"it-pipe": "^3.0.1",
"it-tar": "^6.0.5",
"it-to-browser-readablestream": "^2.0.9",
"libp2p": "^2.2.1",
"lru-cache": "^10.2.2",
"multiformats": "^13.3.0",
"progress-events": "^1.0.1",
Expand All @@ -185,7 +191,6 @@
"blockstore-core": "^5.0.2",
"browser-readablestream-to-it": "^2.0.7",
"datastore-core": "^10.0.2",
"helia": "^5.0.0",
"ipfs-unixfs-importer": "^15.3.1",
"ipns": "^10.0.0",
"it-all": "^3.0.6",
Expand Down
107 changes: 90 additions & 17 deletions packages/verified-fetch/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -594,10 +594,16 @@
* 4. `AbortError` - If the content request is aborted due to user aborting provided AbortSignal. Note that this is a `AbortError` from `@libp2p/interface` and not the standard `AbortError` from the Fetch API.
*/

import { trustlessGateway } from '@helia/block-brokers'
import { bitswap, trustlessGateway } from '@helia/block-brokers'
import { createDelegatedRoutingV1HttpApiClient } from '@helia/delegated-routing-v1-http-api-client'
import { createHeliaHTTP } from '@helia/http'
import { delegatedHTTPRouting, httpGatewayRouting } from '@helia/routers'
import { delegatedHTTPRouting, httpGatewayRouting, libp2pRouting } from '@helia/routers'
import { webRTCDirect } from '@libp2p/webrtc'
import { webSockets } from '@libp2p/websockets'
import { webTransport } from '@libp2p/webtransport'
import { dns } from '@multiformats/dns'
import { createHelia, libp2pDefaults } from 'helia'
import { createLibp2p } from 'libp2p'
import { VerifiedFetch as VerifiedFetchClass } from './verified-fetch.js'
import type { GetBlockProgressEvents, Helia } from '@helia/interface'
import type { ResolveDNSLinkProgressEvents } from '@helia/ipns'
Expand Down Expand Up @@ -639,6 +645,27 @@
gateways: string[]
routers?: string[]

/**
* Whether to enable WebTransport providers.
*
* @default false
*/
webSockets?: boolean

/**
* Whether to enable WebTransport providers.
*
* @default false
*/
webTransport?: boolean

/**
* Whether to enable WebRTC providers.
*
* @default false
*/
webRTC?: boolean

SgtPooki marked this conversation as resolved.
Show resolved Hide resolved
/**
* In order to parse DNSLink records, we need to resolve DNS queries. You can
* pass a list of DNS resolvers that we will provide to the @helia/ipns
Expand Down Expand Up @@ -789,21 +816,67 @@
*/
export async function createVerifiedFetch (init?: Helia | CreateVerifiedFetchInit, options?: CreateVerifiedFetchOptions): Promise<VerifiedFetch> {
if (!isHelia(init)) {
init = await createHeliaHTTP({
blockBrokers: [
trustlessGateway({
allowInsecure: init?.allowInsecure,
allowLocal: init?.allowLocal
})
],
routers: [
...(init?.routers ?? ['https://delegated-ipfs.dev']).map((routerUrl) => delegatedHTTPRouting(routerUrl)),
httpGatewayRouting({
gateways: init?.gateways ?? ['https://trustless-gateway.link']
})
],
dns: createDns(init?.dnsResolvers)
})
if (init?.webTransport ?? init?.webSockets ?? init?.webRTC ?? false) {
const libp2pConfig = libp2pDefaults()

libp2pConfig.transports = []

if (init?.webTransport === true) {
libp2pConfig.transports.push(webTransport())
}
if (init?.webSockets === true) {
libp2pConfig.transports.push(webSockets())
}
if (init?.webRTC === true) {
libp2pConfig.transports.push(webRTCDirect())
}

libp2pConfig.peerDiscovery = [] // disable default bootstrap peers
libp2pConfig.addresses = {} // disable default listen addresses

const routers = init?.routers ?? ['https://delegated-ipfs.dev']
for (let index = 0; index < routers.length; index++) {
const routerUrl = routers[index]
libp2pConfig.services[`delegatedRouting${index}`] = () => createDelegatedRoutingV1HttpApiClient(routerUrl)
}

libp2pConfig.dns = createDns(init?.dnsResolvers)

const libp2p = await createLibp2p(libp2pConfig)

init = await createHelia({
libp2p,
blockBrokers: [
trustlessGateway({
allowInsecure: init?.allowInsecure,
allowLocal: init?.allowLocal
}),
bitswap()
],
routers: [
httpGatewayRouting({
gateways: init?.gateways ?? ['https://trustless-gateway.link']
}),
libp2pRouting(libp2p)
]
})

Check warning on line 862 in packages/verified-fetch/src/index.ts

View check run for this annotation

Codecov / codecov/patch

packages/verified-fetch/src/index.ts#L820-L862

Added lines #L820 - L862 were not covered by tests
} else {
init = await createHeliaHTTP({
blockBrokers: [
trustlessGateway({
allowInsecure: init?.allowInsecure,
allowLocal: init?.allowLocal
})
],
routers: [
...(init?.routers ?? ['https://delegated-ipfs.dev']).map((routerUrl) => delegatedHTTPRouting(routerUrl)),
httpGatewayRouting({
gateways: init?.gateways ?? ['https://trustless-gateway.link']
})
],
dns: createDns(init?.dnsResolvers)
})
}
}

const verifiedFetchInstance = new VerifiedFetchClass({ helia: init }, options)
Expand Down
Loading