Document sources are wrong in the answers

[sunank200]

The answers from the Slackbot return the sources and these sources are wrong most of the time and are very irrelevant.

More details here: https://astronomer.slack.com/archives/C05QJA9LTR9/p1698954492896499

For example: this message on slack has query hello and it is returning various sources

[shillion]

Good example here
The sources referenced seem much less relevant than (say) the Airflow docs.

[vatsrahul1001]

@shillion correct sources/Related docs are not relevant sometimes

[phanikumv]

Might disable this feature if not fixed in time for the Nov 15 release, being discussed here

[sunank200]

We ingested airflow docs which has improved the reference sources.

[sunank200]

We also tried Hybrid search with Cohere re-ranking. This did not work well and answers as well as sources degraded. We will be continuing on this after Nov 28th release

[Lee-W]

@sunank200 The last time you mentioned waiting for our next step. Have we received a response? If not, should we move this to the backlog?

[phanikumv]

David will be working on this, as discussed in the sprint planning call

[phanikumv]

@davidgxue are we making progress on this ticket?

cc @shillion @kaxil

[davidgxue]

Closing this issue as it is duplicate/covered by another issue #133