integrate search engine

* add initial dependencies

* add findpapers #2

+ reduce gitignore

* add shell to ignore

* Feature/frontend (#7)

* update gitignore & add poetry.lock file

* initial structure of the app

* initial structure of the app #1

* add checkbox for the available databases #1

* add datepicker and make the selection of database to true by default

* modify the placement of date picker; add star textbox for APIkeys #1

* first version of search string textbox & join button #1

* update the list of join types and add the search button #1

The join list is updated making None as the default. If the join is None, then the join type is not added to the search string

* slider for the number of results #1

* check if query is empty when button clicked and modify the final query string

* first version of CI file #4

* Update test.yml

* add .coveragerc and update the CI file

* add download button & checkbox for showing results as df #1

* add separate download buttons for csv and bib files #1

* add coverage badge part in the CI file

* update the branch name

* comment out the codecov part in CI file

* delete unwanted comment

* add the features/interface branch of findpapers as submodule

* update gitignore

* uninstall findpapers

* update the findpaper submodule

* Removed submodule findpapers

* add findpapers as submodule in src folder

* add deps of findpapers submodule

* update the branch name

* Updated submodule src/findpapers

* adj integration findpapers #2

* update lock file

* integrate findpapers search with streamlit and save the search in downloads

* adj main for findpapers #11

* del footer

* adj consider keys #11

* make the index param in CSV false

* update variable name

* add json support #11

* add prisma information #15

* update lock file

* append join_type to the list separately

* clean the entire search string  #11

* remove the last added character #11

* add option to insert plain search string directly #11

* add barchart #15

- delete search module

* add ris download #18

* update deps

* refactoring app structure #11 #22

* add search options #23

+ page prefix

* add download btns

* adj start page text

* add prisma viz as page #15

* add manual screening #25

* update deps

* adj gitignore

* refine user infos

* add demo

* adj url demo

.coveragerc

@@ -0,0 +1,10 @@
+# .coveragerc to control coverage.py
+[run]
+omit =
+    */site-packages/*
+    */distutils/*
+    tests/*
+    /usr/*
+
+[html]
+directory = htmlcov

.github/workflows/test.yml

@@ -0,0 +1,71 @@
+name: Test
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.8]
+
+    steps:
+      #----------------------------------------------
+      #       check-out repo and set-up python
+      #----------------------------------------------
+      - name: Check out repository
+        uses: actions/checkout@v2
+      - name: Set up python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+      #----------------------------------------------
+      #  -----  install & configure poetry  -----
+      #----------------------------------------------
+
+      - name: Install Poetry
+        uses: snok/[email protected]
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+
+      #----------------------------------------------
+      #       load cached venv if cache exists
+      #----------------------------------------------
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v2
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
+
+      #----------------------------------------------
+      # install dependencies if cache does not exist
+      #----------------------------------------------
+      - name: Install dependencies
+        run: |
+          poetry install
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+      #----------------------------------------------
+      #              run test suite
+      #----------------------------------------------
+      - name: Run pytest
+        run: PYTHONPATH=src/ poetry run python -m coverage run -m pytest
+
+      - name: Run Coverage
+        run: PYTHONPATH=src/ poetry run python -m coverage report -m;
+
+      - name: Generate html Report
+        run: PYTHONPATH=src/ poetry run python -m coverage html
+      #----------------------------------------------
+      #              coverage badge
+      #----------------------------------------------
+      - uses: codecov/codecov-action@v2
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          fail_ci_if_error: true
+          verbose: true
+

.gitignore

@@ -1,129 +1,29 @@
-# Byte-compiled / optimized / DLL files
+# Byte-compiled
 __pycache__/
-*.py[cod]
-*$py.class
-
-# C extensions
-*.so
-
-# Distribution / packaging
-.Python
-build/
-develop-eggs/
-dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-pip-wheel-metadata/
-share/python-wheels/
-*.egg-info/
-.installed.cfg
-*.egg
-MANIFEST
-
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
-
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
-
-# Unit test / coverage reports
-htmlcov/
-.tox/
-.nox/
-.coverage
-.coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*.cover
-*.py,cover
-.hypothesis/
-.pytest_cache/
-
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
-*.log
-local_settings.py
-db.sqlite3
-db.sqlite3-journal
-
-# Flask stuff:
-instance/
-.webassets-cache
-
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-target/
-
-# Jupyter Notebook
-.ipynb_checkpoints
-
-# IPython
-profile_default/
-ipython_config.py
-
-# pyenv
-.python-version
 
 # pipenv
 #   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
 #   However, in case of collaboration, if having platform-specific dependencies or dependencies
 #   having no cross-platform support, pipenv may install dependencies that don't work, or not
 #   install all needed dependencies.
-#Pipfile.lock
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow
-__pypackages__/
+poetry.lock
 
-# Celery stuff
-celerybeat-schedule
-celerybeat.pid
-
-# SageMath parsed files
-*.sage.py
-
-# Environments
-.env
-.venv
-env/
-venv/
-ENV/
-env.bak/
-venv.bak/
+# Unit test / coverage reports
+.pytest_cache/
+*/tests/data
 
-# Spyder project settings
-.spyderproject
-.spyproject
+# docker
+Dockerfile
 
-# Rope project settings
-.ropeproject
+# shell scripts
+*.sh
 
-# mkdocs documentation
-/site
+# vscode
+.vscode
 
-# mypy
-.mypy_cache/
-.dmypy.json
-dmypy.json
+# notes
+notes.md
 
-# Pyre type checker
-.pyre/
+# coverage
+.coverage
+htmlcov/

README.md

@@ -1,2 +1,45 @@
-# set-you-free
-gather and preprocess data for structured literature search
+Automated search across multiple databases and preprint servers to save your time 
+during structured literature search and review.
+
+# Features
+* Coherent structure across multiple databases and preprint servers
+* Cross-reference search based on the references of your findings
+* Export to advanced structured literature engines such as Ryyan or [cadima](https://www.cadima.info/)
+* Excludes most duplicates across databases
+* Manual selection based on publication details (see PRISMA)
+
+# Demo
+
+![example image](https://github.com/ChristianGerloff/set-you-free/blob/feature/findpapers-integration/assets/demo.gif)
+
+
+# Requirements
+* python > 3.8
+* poetry
+
+# How to start
+
+Navigate to the repo folder and start poetry shell
+```
+poetry shell
+```
+
+Install the dependencies
+```
+poetry install
+```
+
+Start the application via streamlit
+```
+streamlit run src/home.py
+```
+
+## Authors
+
+Christian Gerloff, Leon Lotter, Kashyap Maheshwari
+
+## How to cite
+If you use `Set You Free` please cite (see Zenodo):
+
+
+Gerloff C., Lotter L., & Maheshwari K. (2020). Set You Free: Automated Structured Literature Search.

pyproject.toml

@@ -0,0 +1,18 @@
+[tool.poetry]
+name = "set-you-free"
+version = "0.1.0"
+description = "Structured Literature Search "
+authors = ["Christia Gerloff <[email protected]>"]
+
+[tool.poetry.dependencies]
+python = ">=3.8.0,<4.0.0"
+streamlit = "^1.11.0"
+findpapers = {git = "https://github.com/ChristianGerloff/findpapers.git", branch = "develop"}
+graphviz = "^0.20.1"
+matplotlib-venn = "^0.11.7"
+streamlit-aggrid = "^0.2.3"
+rispy = "^0.7.1"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"

src/home.py

@@ -0,0 +1,28 @@
+import streamlit as st
+
+from pathlib import Path
+from utils.site_config import set_page_title, set_page_style
+
+@st.cache
+def get_home_md() -> str:
+    """Returns home
+
+    Returns:
+        str -- The home as a string of MarkDown
+    """
+    url = Path(__file__).resolve().parent / "pages" / "md" / "home.md"
+    with open(url, mode="r") as file:
+        readme_md_contents = "".join(file.readlines())
+    return readme_md_contents.split("\n", 3)[-1]
+
+set_page_title()
+
+
+# add sidebar
+st.sidebar.markdown("# Home 🎈")
+
+# load page
+with st.spinner("Loading Home ..."):
+    st.title("Set You Free")
+    home = get_home_md()
+    st.markdown(home)