Skip to content

Commit

Permalink
remove pagination, update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
asabya committed Nov 28, 2023
1 parent c6ab2ec commit 98f3520
Show file tree
Hide file tree
Showing 3 changed files with 114 additions and 30 deletions.
85 changes: 68 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
## w3kipedia
## zwi-fave

[w3kipedia](https://github.com/onepeerlabs/w3kipedia) was originally a try to participate for the [WAM](https://www.wearemillions.online/) hackathon for a better wikipedia on swarm.

Now [w3kipedia-fave](https://github.com/onepeerlabs/w3kipedia-fave) is a modified version of w3kipedia that uses FaVe to store, index and search the content.
This project uses FaVe and fairdatasociety/huggingface-vectorizer to index, store and search zwi files content.

It has two components

Expand All @@ -14,8 +12,6 @@ It has two components

Server will start a http-server that can fetch content from FaVe and display them in the web browser.



### How to serve?

```
Expand All @@ -25,7 +21,7 @@ go build

Docker:
```
docker build -f Dockerfile.server --tag w3ki-server .
docker build -f Dockerfile.server --tag zwi-fave-server .
```

#### Help :
Expand All @@ -43,7 +39,7 @@ Usage of ./server:

Docker:
```
docker run w3ki-server -h
docker run zwi-fave-server -h
```


Expand All @@ -58,20 +54,75 @@ Docker:
```
docker run \
-p 8080:8080 \
w3ki-server -fave=<FAVE_API_ENDPOINT> -collection=<COLLECTION_NAME>
zwi-fave-server -fave=<FAVE_API_ENDPOINT> -collection=<COLLECTION_NAME>
```

This will start a local http-serve which will serve wikipedia content on port `:8080`.
This will start a local http-serve which will serve wikipedia content on port `:8080`.

#### How Indexer works:
this project [uses code](https://github.com/akhenakh/gozim/blob) that is [MIT licensed](https://github.com/akhenakh/gozim/blob/master/LICENSE)

Indexer uses [FaVe](https://github.com/fairDataSociety/FaVe) to store content and index them for doing semantic search. It is sanitizing the Article content and vectorizing it to do FaVe Nearest Neighbour Search.
### Docker Compose

#### How Server works:
Best way to run the server is to use a single docker-compose file that will start vectorizer, FaVe and the server.

Server lists all the items in the server with "text/html" mimetype.
Copy the following docker-compose file and save it as `docker-compose.yml` in a directory.
```
version: '3'
services:
vectorizer:
command:
- --model-name
- sentence-transformers/all-mpnet-base-v2
image: fairdatasociety/huggingface-vectorizer:latest
ports:
- 9876:9876
restart: on-failure:0
fave:
command:
- --host
- 0.0.0.0
- --port
- '1234'
- --write-timeout
- 1500m
- --read-timeout
- 1500m
image: fairdatasociety/fave:latest
ports:
- 1234:1234
restart: on-failure:0
environment:
BEE_API: <BEE_URL>
RPC_API: <RPC_ENDPOINT_FOR_ENS>
STAMP_ID: 0
USER: <USER_NAME>
PASSWORD: <PASSWORD>
POD: <POD_NAME>
VECTORIZER_URL: http://vectorizer:9876
VERBOSE: true
zwi-fave:
command:
- -collection
- <COLLECTION_NAME>
- -port
- '8526'
- -fave
- http://fave:1234/v1
image: fairdatasociety/zwi-fave:latest
ports:
- 8526:8526
restart: on-failure:0
```

For showing the content it checks "GET" request on "/wiki/XXXX". it reads the "XXXX" relative url, then finds the entry for that item in the FaVe/FairOS-dfs document store,
downloads the content and sends it back as response.
Change the environment variables to match your setup.
```
BEE_API: <BEE_URL> # Bee api endpoint (Make sure to use local ip (e.g. 192.168.x.x), not "localhost" or "127.0.0.1")
RPC_API: <RPC_ENDPOINT_FOR_ENS> # This is for fairOS-dfs to authenticate with ENS
USER: <USER_NAME> # FairOS-dfs user name
PASSWORD: <PASSWORD> # FairOS-dfs password
POD: <POD_NAME> # FairOS-dfs pod name to store content
COLLECTION_NAME: <COLLECTION_NAME> # Collection name to store content on FaVe
```

this project [uses code](https://github.com/akhenakh/gozim/blob) that is [MIT licensed](https://github.com/akhenakh/gozim/blob/master/LICENSE)
Then run `docker-compose up` to start the server. The server should be available on port `:8526`.
13 changes: 0 additions & 13 deletions cmd/server/templates/searchResult.html
Original file line number Diff line number Diff line change
Expand Up @@ -70,19 +70,6 @@
{{end}}
</div> <!-- /container -->

<div id="footer">
<div class="container">
<ul class="pager">
<li class="previous">
<a href="/search/?search_data={{ .Query }}&page={{ .PreviousPage }}">← Previous</a>
</li>
<li class="next">
<a href="/search/?search_data={{ .Query }}&page={{ .NextPage }}">Next →</a>
</li>
</ul>
</div>
</div>

<script src="/static/js/jquery.min.js"></script>
<script src="/static/js/bootstrap.min.js"></script>

Expand Down
46 changes: 46 additions & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
version: '3'
services:
vectorizer:
command:
- --model-name
- sentence-transformers/all-mpnet-base-v2
image: fairdatasociety/huggingface-vectorizer:latest
ports:
- 9876:9876
restart: on-failure:0

fave:
command:
- --host
- 0.0.0.0
- --port
- '1234'
- --write-timeout
- 1500m
- --read-timeout
- 1500m
image: fairdatasociety/fave:latest
ports:
- 1234:1234
restart: on-failure:0
environment:
BEE_API: <BEE_URL>
RPC_API: <RPC_ENDPOINT_FOR_ENS>
STAMP_ID: <STAMP_ID>
USER: <USER_NAME>
PASSWORD: <PASSWORD>
POD: <POD_NAME>
VECTORIZER_URL: http://vectorizer:9876
VERBOSE: true
zwi-fave:
command:
- -collection
- <COLLECTION_NAME>
- -port
- '8526'
- -fave
- http://fave:1234/v1
image: fairdatasociety/zwi-fave:latest
ports:
- 8526:8526
restart: on-failure:0

0 comments on commit 98f3520

Please sign in to comment.