Code

Pipeline: Ingestion pipeline#96

Code

Merged

bassner merged 181 commits intomainls1intum/Pyris:mainfrom feature/Ingestion_pipelinels1intum/Pyris:feature/Ingestion_pipelineCopy head branch name to clipboardJun 21, 2024

+879-121

Commits

Commits on Feb 14, 2024

• Add new pipeline DTOs

  MichaelOwenDyer

  committed
  403f118

  Copy full SHA for 403f118

• Apply autoformatter

  MichaelOwenDyer

  committed
  e7c74f2

  Copy full SHA for e7c74f2

Commits on Feb 15, 2024

• Have DTOs extend BaseModel

  MichaelOwenDyer

  committed
  26e86ac

  Copy full SHA for 26e86ac

• Add data package

  MichaelOwenDyer

  committed
  6997315

  Copy full SHA for 6997315

Commits on Feb 19, 2024

• update retrieval interface and requirements

  yassinsws

  committed
  128ea40

  Copy full SHA for 128ea40

Commits on Feb 20, 2024

• Merge branch 'main' into feature/datastore

  yassinsws

  authored
  0818109

  Copy full SHA for 0818109

Commits on Feb 21, 2024

• Use cloud cluster for weaviate for now for the hackathon

  Show description for 70ed83f
  yassinsws

  committed
  70ed83f

  Copy full SHA for 70ed83f

• Merge remote-tracking branch 'origin/feature/datastore' into feature/datastore

  yassinsws

  committed
  7acf809

  Copy full SHA for 7acf809

• fix splitting function.

  yassinsws

  committed
  2c0793a

  Copy full SHA for 2c0793a

Commits on Feb 22, 2024

• Add content_service, data ingester and vector repository subsystems

  yassinsws

  committed
  b4cb05d

  Copy full SHA for b4cb05d

• Merge branch 'main' into feature/datastore

  yassinsws

  authored
  2f3882f

  Copy full SHA for 2f3882f

• fix lintin

  yassinsws

  committed
  05490f2

  Copy full SHA for 05490f2

• Merge remote-tracking branch 'origin/feature/datastore' into feature/datastore

  yassinsws

  committed
  e08aac6

  Copy full SHA for e08aac6

• add a return statement to unzip

  yassinsws

  committed
  a29a44b

  Copy full SHA for a29a44b

Commits on Mar 6, 2024

• Add image recognition for Ollama, GPT4V and image generation for Dall-E

  Hialus

  committed
  0c96395

  Copy full SHA for 0c96395

Commits on Mar 17, 2024

• Solved requirements problem ( removed olama for now as weaviate needs a httpx version >= 0.26 and ollama needs a version >= 0.25.2 and 0.26<,

  Show description for e9874b9
  yassinsws

  committed
  e9874b9

  Copy full SHA for e9874b9

• fixed requirements file

  yassinsws

  committed
  3a186c9

  Copy full SHA for 3a186c9

• fixed message interpretation function in the llm class

  yassinsws

  committed
  379550b

  Copy full SHA for 379550b

Commits on Mar 18, 2024

• Added detail parameter to image_interpretation model

  Show description for 0f6e576
  yassinsws

  committed
  0f6e576

  Copy full SHA for 0f6e576

• renamed pyris_image to iris_image

  yassinsws

  committed
  a4186c3

  Copy full SHA for a4186c3

• Update app/content_service/Ingestion/lectures_ingestion.py

  Show description for 9f2848e
  

  yassinsws

  and

  coderabbitai[bot]

  authored
  9f2848e

  Copy full SHA for 9f2848e

• Update app/content_service/Retrieval/abstract_retrieval.py

  Show description for 224a701
  

  yassinsws

  and

  coderabbitai[bot]

  authored
  224a701

  Copy full SHA for 224a701

• Update app/content_service/Ingestion/repository_ingestion.py

  Show description for 93a2f44
  

  yassinsws

  and

  coderabbitai[bot]

  authored
  93a2f44

  Copy full SHA for 93a2f44

• Update app/content_service/Ingestion/lectures_ingestion.py

  Show description for bca6377
  

  yassinsws

  and

  coderabbitai[bot]

  authored
  bca6377

  Copy full SHA for bca6377

• Update app/content_service/Retrieval/lecture_retrieval.py

  Show description for 6e9525d
  

  yassinsws

  and

  coderabbitai[bot]

  authored
  6e9525d

  Copy full SHA for 6e9525d

• erase old lecture download files

  yassinsws

  committed
  bc97236

  Copy full SHA for bc97236

• Add a function to get lectures from Artemis

  yassinsws

  committed
  b0291b1

  Copy full SHA for b0291b1

Commits on Mar 24, 2024

• Update app/content_service/get_lecture_from_artemis.py

  Show description for 7211386
  

  yassinsws

  and

  coderabbitai[bot]

  authored
  7211386

  Copy full SHA for 7211386

• refractor tutor pipeline

  yassinsws

  committed
  738e7a0

  Copy full SHA for 738e7a0

Commits on Mar 25, 2024

• Lecture content first draft ready for review

  yassinsws

  committed
  303f6d4

  Copy full SHA for 303f6d4

• Black

  yassinsws

  committed
  1dfd03a

  Copy full SHA for 1dfd03a

Commits on Mar 29, 2024

• Black prompt and update database link

  yassinsws

  committed
  f3b3c93

  Copy full SHA for f3b3c93

Commits on Mar 31, 2024

• Lecture chat pipeline works just fine

  yassinsws

  committed
  34bc567

  Copy full SHA for 34bc567

• Black

  yassinsws

  committed
  67d6bd0

  Copy full SHA for 67d6bd0

• flake8

  yassinsws

  committed
  2fe64ab

  Copy full SHA for 2fe64ab

• Add Image support to our llm

  yassinsws

  committed
  4ef1672

  Copy full SHA for 4ef1672

• flake8

  yassinsws

  committed
  d15c6e6

  Copy full SHA for d15c6e6

• black

  yassinsws

  committed
  0f57336

  Copy full SHA for 0f57336

• black

  yassinsws

  committed
  bcc54c2

  Copy full SHA for bcc54c2

Commits on Apr 1, 2024

• Added method to delete objects and collections from the data base, adjusted the lecture units dto

  yassinsws

  committed
  22a96ab

  Copy full SHA for 22a96ab

• Initial commit for the ingestion pipeline

  yassinsws

  committed
  981a453

  Copy full SHA for 981a453

Commits on Apr 2, 2024

• Merge branch 'main' into feature/llm/image_support

  Hialus

  authored
  a0b1676

  Copy full SHA for a0b1676

Commits on Apr 3, 2024

• Merge remote-tracking branch 'origin/feature/llm/image_support' into feature/Ingestion_pipeline

  Show description for 1b372ec
  yassinsws

  committed
  1b372ec

  Copy full SHA for 1b372ec

• black

  yassinsws

  committed
  7c48731

  Copy full SHA for 7c48731

Commits on Apr 6, 2024

• Integrate Lecture Pipeline and Tutor chat Pipeline

  yassinsws

  committed
  5c94f8d

  Copy full SHA for 5c94f8d

• Requirements cannot work with ollama ( version too old )

  yassinsws

  committed
  5261e94

  Copy full SHA for 5261e94

• Merge branch 'main' of https://github.com/ls1intum/Pyris into feature/Lecture-Pipeline

  Show description for 67226d2
  yassinsws

  committed
  67226d2

  Copy full SHA for 67226d2

Commits on Apr 7, 2024

• Merge branch 'feature/Lecture-Pipeline' of https://github.com/ls1intum/Pyris into feature/Ingestion_pipeline

  Show description for deaebf0
  yassinsws

  committed
  deaebf0

  Copy full SHA for deaebf0

• Save Work, ingestion is implemented

  yassinsws

  committed
  ea7291c

  Copy full SHA for ea7291c

• lecture ingestion Pipeline implemented and ready for review

  yassinsws

  committed
  0cbc8ca

  Copy full SHA for 0cbc8ca

• there is no image support in completion

  yassinsws

  committed
  fa4e705

  Copy full SHA for fa4e705

• Fix Linters

  yassinsws

  committed
  53edf86

  Copy full SHA for 53edf86

• Update app/content_service/Retrieval/abstract_retrieval.py

  Show description for 57b0d72
  

  yassinsws

  and

  coderabbitai[bot]

  authored
  57b0d72

  Copy full SHA for 57b0d72

Commits on Apr 22, 2024

• fixed issues with ingestion pipeline

  yassinsws

  committed
  f567809

  Copy full SHA for f567809

• Fix linter

  yassinsws

  committed
  f8aef7e

  Copy full SHA for f8aef7e

• Fix linter

  yassinsws

  committed
  7a6270b

  Copy full SHA for 7a6270b

• Fix Ingestion Pipeline, ready for review

  yassinsws

  committed
  bc69e96

  Copy full SHA for bc69e96

Commits on Apr 23, 2024

• Ingestion Pipeline tested with a new instance of the vector database

  yassinsws

  committed
  bea9fcf

  Copy full SHA for bea9fcf

Commits on Apr 24, 2024

• change the database

  yassinsws

  committed
  12b33f9

  Copy full SHA for 12b33f9

• Merge remote-tracking branch 'origin/main' into feature/Ingestion_pipeline

  yassinsws

  committed
  d46f7e7

  Copy full SHA for d46f7e7

Commits on Apr 25, 2024

• Solve merge Conflict and update Pr

  yassinsws

  committed
  b4acb1d

  Copy full SHA for b4acb1d

• Update Image Support

  yassinsws

  committed
  412d5a7

  Copy full SHA for 412d5a7

• Merge branch 'main' into feature/llm/image_support

  yassinsws

  authored
  2ac1d48

  Copy full SHA for 2ac1d48

• Fix Requirements, ollama should be deleted because it's using an old installer that does not work with weaviate

  yassinsws

  committed
  58ac585

  Copy full SHA for 58ac585

• Merge With Latest version of main

  yassinsws

  committed
  892bd6f

  Copy full SHA for 892bd6f

• Merge With Latest version of main

  yassinsws

  committed
  e05206c

  Copy full SHA for e05206c

• Merge remote-tracking branch 'origin/main' into feature/Ingestion_pipeline

  yassinsws

  committed
  1ca6b8e

  Copy full SHA for 1ca6b8e

• Merge remote-tracking branch 'origin/feature/llm/image_support' into feature/Ingestion_pipeline

  yassinsws

  committed
  c72f8ad

  Copy full SHA for c72f8ad

• Fix Warning

  yassinsws

  committed
  5d29c93

  Copy full SHA for 5d29c93

Commits on Apr 26, 2024

• Readjusted the image generation and recognition PR

  yassinsws

  committed
  2e09692

  Copy full SHA for 2e09692

• Image interpretation tested works fine

  yassinsws

  committed
  fe76c80

  Copy full SHA for fe76c80

• Black

  yassinsws

  committed
  ec964c3

  Copy full SHA for ec964c3

Commits on Apr 27, 2024

• Merge remote-tracking branch 'origin/feature/llm/image_support' into feature/Ingestion_pipeline

  yassinsws

  committed
  f961717

  Copy full SHA for f961717

• Update app/content_service/Retrieval/repositories_retrieval.py

  Show description for 1171b25
  

  yassinsws

  and

  coderabbitai[bot]

  authored
  1171b25

  Copy full SHA for 1171b25

• Update app/llm/external/openai_dalle.py

  Show description for ab73df5
  

  yassinsws

  and

  coderabbitai[bot]

  authored
  ab73df5

  Copy full SHA for ab73df5

Commits on Apr 28, 2024

• Added status update and delete data from database

  yassinsws

  committed
  c7518ee

  Copy full SHA for c7518ee

• black

  yassinsws

  committed
  1bd1b85

  Copy full SHA for 1bd1b85

Commits on May 1, 2024

• Skip was not working when the Stages are done

  yassinsws

  committed
  764931e

  Copy full SHA for 764931e

Commits on May 3, 2024

• Update code

  yassinsws

  committed
  6c60225

  Copy full SHA for 6c60225

• Update code

  yassinsws

  committed
  a9c77c1

  Copy full SHA for a9c77c1

• Flake8

  yassinsws

  committed
  69c791a

  Copy full SHA for 69c791a

• Update and merge main with datastore PR

  yassinsws

  committed
  c7f53ee

  Copy full SHA for c7f53ee

• Erase drafts of lecture_ingestion and repository_ingestion, because it does not make sense to implement them here.

  yassinsws

  committed
  aa247b8

  Copy full SHA for aa247b8

• refractor code

  yassinsws

  committed
  4dd3b3d

  Copy full SHA for 4dd3b3d

• refractor code

  yassinsws

  committed
  f06e884

  Copy full SHA for f06e884

• refractor code

  yassinsws

  committed
  008a9e5

  Copy full SHA for 008a9e5

• Merge

  yassinsws

  committed
  b4d0120

  Copy full SHA for b4d0120

• Black Flake8

  yassinsws

  committed
  42ce267

  Copy full SHA for 42ce267

• return get client

  yassinsws

  committed
  fcbddc5

  Copy full SHA for fcbddc5

Commits on May 4, 2024

• implement request changes

  yassinsws

  committed
  4bd9cd2

  Copy full SHA for 4bd9cd2

Commits on May 5, 2024

• implement request changes

  yassinsws

  committed
  7021ba5

  Copy full SHA for 7021ba5

• modify lecute_unit_dto

  yassinsws

  committed
  bc75592

  Copy full SHA for bc75592

• make class into enum

  yassinsws

  committed
  0ac2712

  Copy full SHA for 0ac2712

• make class into enum

  yassinsws

  committed
  b50ea25

  Copy full SHA for b50ea25

• merge datastore pr changes

  yassinsws

  committed
  dfa3063

  Copy full SHA for dfa3063

• merge datastore pr changes

  yassinsws

  committed
  eb5f446

  Copy full SHA for eb5f446

• Merge branch 'main' into feature/datastore

  yassinsws

  authored
  586aa1e

  Copy full SHA for 586aa1e

• Clean PR

  yassinsws

  committed
  5133adc

  Copy full SHA for 5133adc

• Clean PR

  yassinsws

  committed
  5abd811

  Copy full SHA for 5abd811

• add typed dict

  yassinsws

  committed
  3e77483

  Copy full SHA for 3e77483

• add typed dict

  yassinsws

  committed
  1da1d5e

  Copy full SHA for 1da1d5e

• Erase content_service

  yassinsws

  committed
  4699fed

  Copy full SHA for 4699fed

• Erase content_service

  yassinsws

  committed
  ea32c7b

  Copy full SHA for ea32c7b

• fix lecture_schema

  yassinsws

  committed
  6bc383a

  Copy full SHA for 6bc383a

• fix lecture_schema

  yassinsws

  committed
  19e2c6c

  Copy full SHA for 19e2c6c

• fix lecture_schema

  yassinsws

  committed
  b0e6f1d

  Copy full SHA for b0e6f1d

Commits on May 6, 2024

• fix status update bug

  yassinsws

  committed
  f56c288

  Copy full SHA for f56c288

• fix status update bug

  yassinsws

  committed
  2e25f9d

  Copy full SHA for 2e25f9d

• MERGE MAIN AND DATASTORE PIPELINE

  Show description for db9d67a
  yassinsws

  committed
  db9d67a

  Copy full SHA for db9d67a

• MERGE MAIN AND DATASTORE PIPELINE

  Show description for dfdb5e5
  yassinsws

  committed
  dfdb5e5

  Copy full SHA for dfdb5e5

• MERGE MAIN AND DATASTORE PIPELINE

  Show description for 719aec2
  yassinsws

  committed
  719aec2

  Copy full SHA for 719aec2

• Add an exponential Backoff window for the embeddings, to get past the rate limit error while importing with big batches of data in the vector database

  yassinsws

  committed
  27c91d7

  Copy full SHA for 27c91d7

• Add an exponential Backoff window for the embeddings, to get past the rate limit error while importing with big batches of data in the vector database

  yassinsws

  committed
  9a50a79

  Copy full SHA for 9a50a79

• Add an exponential Backoff window for the embeddings, to get past the rate limit error while importing with big batches of data in the vector database

  yassinsws

  committed
  d9dd8bc

  Copy full SHA for d9dd8bc

• Add an exponential Backoff window for the embeddings, to get past the rate limit error while importing with big batches of data in the vector database

  yassinsws

  committed
  9a1679d

  Copy full SHA for 9a1679d

Commits on May 7, 2024

• Add classes used in code

  yassinsws

  committed
  53b13d8

  Copy full SHA for 53b13d8

• replace import all classes only with the classes needed

  yassinsws

  committed
  1b477ff

  Copy full SHA for 1b477ff

• replace import all classes only with the classes needed

  yassinsws

  committed
  dca1493

  Copy full SHA for dca1493

• Merge branch 'main' into feature/datastore

  yassinsws

  authored
  fd11add

  Copy full SHA for fd11add

• Update requirements.txt

  yassinsws

  committed
  3930890

  Copy full SHA for 3930890

• rename db to database

  yassinsws

  committed
  5a70b5c

  Copy full SHA for 5a70b5c

• Merge remote-tracking branch 'origin/feature/datastore' into feature/Ingestion_pipeline

  yassinsws

  committed
  ef1b4ef

  Copy full SHA for ef1b4ef

• Make batch import done by only one thread.

  Show description for c9587c5
  yassinsws

  committed
  c9587c5

  Copy full SHA for c9587c5

Commits on May 10, 2024

• Change the way we add image interpretation to the ingestion

  yassinsws

  committed
  7b40f7c

  Copy full SHA for 7b40f7c

• Change the way we add image interpretation to the ingestion

  yassinsws

  committed
  8e3d710

  Copy full SHA for 8e3d710

Commits on May 20, 2024

• Merge remote-tracking branch 'origin/main' into feature/Ingestion_pipeline

  Show description for a8a3c4f
  yassinsws

  committed
  a8a3c4f

  Copy full SHA for a8a3c4f

Commits on May 26, 2024

• Minor Changes in ingestion pipeline

  yassinsws

  committed
  b410cf3

  Copy full SHA for b410cf3

• Minor Changes in ingestion pipeline

  yassinsws

  committed
  32f33be

  Copy full SHA for 32f33be

• return the Basic request handler because needed for ingestion

  yassinsws

  committed
  6062f39

  Copy full SHA for 6062f39

• fix wrong use of prompt param

  bassner

  committed
  67c3ee6

  Copy full SHA for 67c3ee6

• fix formatting

  bassner

  committed
  d1e9874

  Copy full SHA for d1e9874

• use gpt 3 for lang detection

  bassner

  committed
  0eadd32

  Copy full SHA for 0eadd32

• remove redundant alias

  bassner

  committed
  d37851a

  Copy full SHA for d37851a

• merge and update course language method

  yassinsws

  committed
  f4c8c3c

  Copy full SHA for f4c8c3c

• merge and update course language method

  yassinsws

  committed
  46e157a

  Copy full SHA for 46e157a

• minor changes on message_content_dto and image generation

  yassinsws

  committed
  b89703e

  Copy full SHA for b89703e

• Correct change llm to 3.5

  yassinsws

  committed
  730305f

  Copy full SHA for 730305f

Commits on May 27, 2024

• Apply suggestions from code review

  Show description for 0f354ae
  

  bassner

  and

  coderabbitai[bot]

  authored
  0f354ae

  Copy full SHA for 0f354ae

• Add futur Todos

  yassinsws

  committed
  190587c

  Copy full SHA for 190587c

• Merge remote-tracking branch 'origin/feature/Ingestion_pipeline' into feature/Ingestion_pipeline

  yassinsws

  committed
  4751070

  Copy full SHA for 4751070

• Black

  yassinsws

  committed
  dd9fed4

  Copy full SHA for dd9fed4

• Black

  yassinsws

  committed
  dbaef87

  Copy full SHA for dbaef87

• requirements.txt merge

  yassinsws

  committed
  e620fcb

  Copy full SHA for e620fcb

• remove print statements

  yassinsws

  committed
  f5f1738

  Copy full SHA for f5f1738

• Add local weaviate database with docker file

  yassinsws

  committed
  d9c458e

  Copy full SHA for d9c458e

• Linters

  yassinsws

  committed
  ed7b2e1

  Copy full SHA for ed7b2e1

Commits on May 30, 2024

• Add base_url field to ingestion

  yassinsws

  committed
  dd21a1a

  Copy full SHA for dd21a1a

• Add timor Review

  yassinsws

  committed
  ca35123

  Copy full SHA for ca35123

• Linter

  yassinsws

  committed
  50bd537

  Copy full SHA for 50bd537

Commits on May 31, 2024

• change 8000 to 8001

  yassinsws

  committed
  abdbc21

  Copy full SHA for abdbc21

• TIMOR REVIEW

  yassinsws

  committed
  7728d03

  Copy full SHA for 7728d03

• env variable from application.yml

  yassinsws

  committed
  753cdbb

  Copy full SHA for 753cdbb

Commits on Jun 1, 2024

• timor last comment, make volume configurable with an environment variable

  yassinsws

  committed
  7a80e5f

  Copy full SHA for 7a80e5f

• timor last comment, make volume configurable with an environment variable

  yassinsws

  committed
  f854fd4

  Copy full SHA for f854fd4

• Linter

  yassinsws

  committed
  6192932

  Copy full SHA for 6192932

Commits on Jun 3, 2024

• make fields that can be null on artemis optional

  yassinsws

  committed
  8b1345f

  Copy full SHA for 8b1345f

• black

  yassinsws

  committed
  ed51977

  Copy full SHA for ed51977

Commits on Jun 4, 2024

• Get weaviate production ready

  yassinsws

  committed
  ccbffbb

  Copy full SHA for ccbffbb

• Get weaviate production ready

  yassinsws

  committed
  b89a58f

  Copy full SHA for b89a58f

• Get weaviate production ready

  yassinsws

  committed
  5f6b8b7

  Copy full SHA for 5f6b8b7

Commits on Jun 7, 2024

• Merge branch 'main' into feature/Ingestion_pipeline

  yassinsws

  authored
  b0df9f2

  Copy full SHA for b0df9f2

• Light-Weight Ingestion Pipeline (#119)

  yassinsws

  authored
  58258c9

  Copy full SHA for 58258c9

Commits on Jun 10, 2024

• Update example_application.yml

  Show description for 092a0ec
  

  kaancayli

  and

  Hialus

  authored
  092a0ec

  Copy full SHA for 092a0ec

Commits on Jun 11, 2024

• Max chunks get extended

  yassinsws

  committed
  d22c856

  Copy full SHA for d22c856

• Merge branch 'main' into feature/Ingestion_pipeline

  yassinsws

  authored
  ca11b9e

  Copy full SHA for ca11b9e

• Merge with ingestion Pipeline

  yassinsws

  committed
  16e0e49

  Copy full SHA for 16e0e49

• Fix typo in pyris-dev

  yassinsws

  committed
  930050d

  Copy full SHA for 930050d

• flake8

  yassinsws

  committed
  d503ba2

  Copy full SHA for d503ba2

Commits on Jun 13, 2024

• Ingestion with images

  yassinsws

  committed
  e4c7575

  Copy full SHA for e4c7575

• Ingestion with images

  yassinsws

  committed
  a6b70ac

  Copy full SHA for a6b70ac

• keep error handling

  yassinsws

  committed
  e5b34f0

  Copy full SHA for e5b34f0

• keep error handling

  yassinsws

  committed
  c60921b

  Copy full SHA for c60921b

• erase full ingestion checks for now

  yassinsws

  committed
  9e34fc2

  Copy full SHA for 9e34fc2

• erase full ingestion checks for now

  yassinsws

  committed
  28d99cf

  Copy full SHA for 28d99cf

• Add log when course finished

  yassinsws

  committed
  4843299

  Copy full SHA for 4843299

• Add log when course finished

  yassinsws

  committed
  dc205ed

  Copy full SHA for dc205ed

• erase unsused import

  yassinsws

  committed
  847913d

  Copy full SHA for 847913d

• Merge branch 'main' into feature/Ingestion_pipeline

  bassner

  authored
  692a10a

  Copy full SHA for 692a10a

Commits on Jun 15, 2024

• Add check if course was ingested and only use the lecture pipeline if it was already ingested

  yassinsws

  committed
  d5d926e

  Copy full SHA for d5d926e

Commits on Jun 18, 2024

• make chunks size 512

  yassinsws

  committed
  5732e7c

  Copy full SHA for 5732e7c

• give only 5 best chunks to tutor pipeline

  yassinsws

  committed
  cba0561

  Copy full SHA for cba0561