Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

US: Understand search results #10

Open
ridoo opened this issue Oct 25, 2023 · 3 comments
Open

US: Understand search results #10

ridoo opened this issue Oct 25, 2023 · 3 comments
Labels
Milestone

Comments

@ridoo
Copy link
Member

ridoo commented Oct 25, 2023

As an end user, I want to see the field(s) my search term matched a value, in order to make better sense of my results

@ridoo ridoo converted this from a draft issue Oct 25, 2023
@ridoo ridoo added the end user label Oct 25, 2023
@ridoo
Copy link
Member Author

ridoo commented Oct 26, 2023

Think to include this in the API response

@ridoo ridoo moved this from Backlog to Ready in 52°North Innovation Prize Oct 26, 2023
@ridoo ridoo moved this from Ready to Backlog in 52°North Innovation Prize Oct 26, 2023
@ridoo ridoo added this to the must have milestone Oct 26, 2023
@ridoo ridoo moved this from Backlog to Ready in 52°North Innovation Prize Jun 17, 2024
@ridoo
Copy link
Member Author

ridoo commented Sep 9, 2024

Als erster Schritt würde schon reichen, dem Nutzer anzuzeigen, welche Tokens aus dem verwendeten Suchkontext, die mit dem höchsten Ranking waren, und welche Suchanfrage am Ende verwendet wurde (condensed query).

@simeonwetzel
Copy link
Collaborator

First step implemented:

  • Added a custom explainer based on SHAP
  • Problem: SHAP is relatively slow. I decided to not include it in the chatbot workflow, but implemented it as separate API-endpoint that one can pass the query and the documents (search results from a previous search in LangChain Document format) and the words with highest relevance (according to the text-embedding) model are retrieved.

Example Usage:

curl -X POST http://localhost:8000/explain_results \
-H "Content-Type: application/json" \
-d '{
    "index_name": "geojson",
    "query": "hospital building data in Dresden",
    "documents": [
        {
            "page_content": "Name: Innere Medizin - Haus 81\n\nAddress: Fetscherstraße, 74, Dresden, 01307, DE\nbuilding: hospital\nbuilding:levels: 3\ncn_ukd:token: U0081\nname: Innere Medizin - Haus 81\n_osm_type: way\ndescription: A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
            "metadata": {
                "_osm_type": "way",
                "addr:city": "Dresden",
                "addr:country": "DE",
                "addr:housenumber": "74",
                "addr:postcode": "01307",
                "addr:street": "Fetscherstraße",
                "building": "hospital",
                "building:levels": "3",
                "cn_ukd:token": "U0081",
                "description": "A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
                "feature": "{\n  \"type\": \"Polygon\",\n  \"coordinates\": [\n    [\n      [\n        13.781329,\n        51.055514\n      ],\n      [\n        13.781659,\n        51.055528\n      ],\n      [\n        13.781609,\n        51.055989\n      ],\n      [\n        13.781406,\n        51.05598\n      ],\n      [\n        13.781372,\n        51.055979\n      ],\n      [\n        13.781278,\n        51.055975\n      ],\n      [\n        13.781293,\n        51.055841\n      ],\n      [\n        13.781329,\n        51.055514\n      ]\n    ]\n  ]\n}",
                "name": "Innere Medizin - Haus 81",
                "type": "hospital",
                "url": "url"
            },
            "type": "Document"
        },
        {
            "page_content": "Name: Haus D (24)\n\nAddress: Stendaler Straße, 24, Dresden, 01109, DE\nbuilding: hospital\nbuilding:levels: 2\nname: Haus D (24)\nroof:levels: 0\nsource: HiRes aerial imagery\n_osm_type: way\ndescription: A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
            "metadata": {
                "_osm_type": "way",
                "addr:city": "Dresden",
                "addr:country": "DE",
                "addr:housenumber": "24",
                "addr:postcode": "01109",
                "addr:street": "Stendaler Straße",
                "building": "hospital",
                "building:levels": "2",
                "description": "A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
                "feature": "{\n  \"type\": \"Polygon\",\n  \"coordinates\": [\n    [\n      [\n        13.783085,\n        51.120414\n      ],\n      [\n        13.783115,\n        51.120452\n      ],\n      [\n        13.783167,\n        51.120486\n      ],\n      [\n        13.783209,\n        51.120468\n      ],\n      [\n        13.783221,\n        51.120481\n      ],\n      [\n        13.783406,\n        51.120403\n      ],\n      [\n        13.783389,\n        51.12039\n      ],\n      [\n        13.783426,\n        51.12037\n      ],\n      [\n        13.783417,\n        51.120342\n      ],\n      [\n        13.783377,\n        51.120316\n      ],\n      [\n        13.783344,\n        51.120326\n      ],\n      [\n        13.783332,\n        51.120314\n      ],\n      [\n        13.783267,\n        51.120339\n      ],\n      [\n        13.783243,\n        51.120337\n      ],\n      [\n        13.78321,\n        51.120343\n      ],\n      [\n        13.783192,\n        51.120358\n      ],\n      [\n        13.783192,\n        51.120372\n      ],\n      [\n        13.783085,\n        51.120414\n      ]\n    ]\n  ]\n}",
                "name": "Haus D (24)",
                "roof:levels": "0",
                "source": "HiRes aerial imagery",
                "type": "hospital",
                "url": "url"
            },
            "type": "Document"
        },
        {
            "page_content": "Name: Haus 31\n\nAddress: Fiedlerstraße, 23, Dresden, 01307, DE\nbuilding: hospital\ncn_ukd:token: U0031\nname: Haus 31\n_osm_type: way\ndescription: A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
            "metadata": {
                "_osm_type": "way",
                "addr:city": "Dresden",
                "addr:country": "DE",
                "addr:housenumber": "23",
                "addr:postcode": "01307",
                "addr:street": "Fiedlerstraße",
                "building": "hospital",
                "cn_ukd:token": "U0031",
                "description": "A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
                "feature": "{\n  \"type\": \"Polygon\",\n  \"coordinates\": [\n    [\n      [\n        13.78272,\n        51.055751\n      ],\n      [\n        13.782908,\n        51.05575\n      ],\n      [\n        13.782905,\n        51.055544\n      ],\n      [\n        13.78285,\n        51.055545\n      ],\n      [\n        13.78285,\n        51.055513\n      ],\n      [\n        13.782849,\n        51.055476\n      ],\n      [\n        13.783345,\n        51.055474\n      ],\n      [\n        13.783344,\n        51.055357\n      ],\n      [\n        13.782684,\n        51.055359\n      ],\n      [\n        13.782683,\n        51.055477\n      ],\n      [\n        13.782729,\n        51.055476\n      ],\n      [\n        13.782729,\n        51.055517\n      ],\n      [\n        13.782657,\n        51.055516\n      ],\n      [\n        13.78262,\n        51.055516\n      ],\n      [\n        13.782621,\n        51.055363\n      ],\n      [\n        13.782405,\n        51.055363\n      ],\n      [\n        13.782403,\n        51.055442\n      ],\n      [\n        13.782447,\n        51.055442\n      ],\n      [\n        13.782445,\n        51.0555\n      ],\n      [\n        13.782446,\n        51.055548\n      ],\n      [\n        13.782404,\n        51.055548\n      ],\n      [\n        13.782405,\n        51.055745\n      ],\n      [\n        13.782603,\n        51.055747\n      ],\n      [\n        13.782604,\n        51.055605\n      ],\n      [\n        13.782604,\n        51.055567\n      ],\n      [\n        13.782716,\n        51.055568\n      ],\n      [\n        13.782718,\n        51.055588\n      ],\n      [\n        13.782718,\n        51.055598\n      ],\n      [\n        13.78272,\n        51.055751\n      ]\n    ]\n  ]\n}",
                "name": "Haus 31",
                "type": "hospital",
                "url": "url"
            },
            "type": "Document"
        }
]
}

Results:

[
    {
        "page_content": "Name: Innere Medizin - Haus 81\n\nAddress: Fetscherstraße, 74, Dresden, 01307, DE\nbuilding: hospital\nbuilding:levels: 3\ncn_ukd:token: U0081\nname: Innere Medizin - Haus 81\n_osm_type: way\ndescription: A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
        "metadata": {
            "_osm_type": "way",
            "addr:city": "Dresden",
            "addr:country": "DE",
            "addr:housenumber": "74",
            "addr:postcode": "01307",
            "addr:street": "Fetscherstraße",
            "building": "hospital",
            "building:levels": "3",
            "cn_ukd:token": "U0081",
            "description": "A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
            "feature": "{\n  \"type\": \"Polygon\",\n  \"coordinates\": [\n    [\n      [\n        13.781329,\n        51.055514\n      ],\n      [\n        13.781659,\n        51.055528\n      ],\n      [\n        13.781609,\n        51.055989\n      ],\n      [\n        13.781406,\n        51.05598\n      ],\n      [\n        13.781372,\n        51.055979\n      ],\n      [\n        13.781278,\n        51.055975\n      ],\n      [\n        13.781293,\n        51.055841\n      ],\n      [\n        13.781329,\n        51.055514\n      ]\n    ]\n  ]\n}",
            "name": "Innere Medizin - Haus 81",
            "type": "hospital",
            "url": "url",
            "relevant_words": [
                [
                    "U0081",
                    0.04021817938803536
                ],
                [
                    "Haus",
                    0.03670811340064961
                ],
                [
                    "Dresden,",
                    0.036137163368211524
                ],
                [
                    "cn_ukd:token:",
                    0.03556632531448108
                ],
                [
                    "which",
                    0.028685500317960164
                ],
                [
                    "Useamenity=hospitalfor",
                    0.026842378458952345
                ],
                [
                    "Haus",
                    0.023134704056613227
                ],
                [
                    "hospital",
                    0.0181867753360835
                ],
                [
                    "building:",
                    0.017998147659422505
                ],
                [
                    "building",
                    0.015051868759361361
                ],
                [
                    "hospital.",
                    0.015021690339013027
                ],
                [
                    "description:",
                    0.014225017746953887
                ],
                [
                    "Innere",
                    0.01332157938117844
                ],
                [
                    "building:levels:",
                    0.011109900067221928
                ],
                [
                    "the",
                    0.010730932161399023
                ],
                [
                    "Innere",
                    0.0104195896345709
                ],
                [
                    "-",
                    0.004880422377064338
                ],
                [
                    "hospital",
                    0.0031463343570015873
                ],
                [
                    "-",
                    0.003114987882891871
                ]
            ]
        },
        "type": "Document"
    },
    {
        "page_content": "Name: Haus D (24)\n\nAddress: Stendaler Straße, 24, Dresden, 01109, DE\nbuilding: hospital\nbuilding:levels: 2\nname: Haus D (24)\nroof:levels: 0\nsource: HiRes aerial imagery\n_osm_type: way\ndescription: A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
        "metadata": {
            "_osm_type": "way",
            "addr:city": "Dresden",
            "addr:country": "DE",
            "addr:housenumber": "24",
            "addr:postcode": "01109",
            "addr:street": "Stendaler Straße",
            "building": "hospital",
            "building:levels": "2",
            "description": "A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
            "feature": "{\n  \"type\": \"Polygon\",\n  \"coordinates\": [\n    [\n      [\n        13.783085,\n        51.120414\n      ],\n      [\n        13.783115,\n        51.120452\n      ],\n      [\n        13.783167,\n        51.120486\n      ],\n      [\n        13.783209,\n        51.120468\n      ],\n      [\n        13.783221,\n        51.120481\n      ],\n      [\n        13.783406,\n        51.120403\n      ],\n      [\n        13.783389,\n        51.12039\n      ],\n      [\n        13.783426,\n        51.12037\n      ],\n      [\n        13.783417,\n        51.120342\n      ],\n      [\n        13.783377,\n        51.120316\n      ],\n      [\n        13.783344,\n        51.120326\n      ],\n      [\n        13.783332,\n        51.120314\n      ],\n      [\n        13.783267,\n        51.120339\n      ],\n      [\n        13.783243,\n        51.120337\n      ],\n      [\n        13.78321,\n        51.120343\n      ],\n      [\n        13.783192,\n        51.120358\n      ],\n      [\n        13.783192,\n        51.120372\n      ],\n      [\n        13.783085,\n        51.120414\n      ]\n    ]\n  ]\n}",
            "name": "Haus D (24)",
            "roof:levels": "0",
            "source": "HiRes aerial imagery",
            "type": "hospital",
            "url": "url",
            "relevant_words": [
                [
                    "building",
                    0.04721152093956271
                ],
                [
                    "building:",
                    0.042570680345710085
                ],
                [
                    "which",
                    0.03622307518319122
                ],
                [
                    "A",
                    0.03256417478316661
                ],
                [
                    "Dresden,",
                    0.02562999006347844
                ],
                [
                    "hospital.",
                    0.024987433534905946
                ],
                [
                    "hospital",
                    0.02425938400302163
                ],
                [
                    "the",
                    0.024123832403124823
                ],
                [
                    "Useamenity=hospitalfor",
                    0.02222813824676195
                ],
                [
                    "Address:",
                    0.021957302935452387
                ],
                [
                    "hospital",
                    0.02014690109865475
                ],
                [
                    "name:",
                    0.01870588234300493
                ],
                [
                    "0",
                    0.017804558184802918
                ],
                [
                    "forms",
                    0.016616498640807713
                ],
                [
                    "grounds.",
                    0.015623788349146975
                ],
                [
                    "a",
                    0.013288836799905421
                ],
                [
                    "imagery",
                    0.00921730500650579
                ],
                [
                    "way",
                    0.00047317725782688725
                ]
            ]
        },
        "type": "Document"
    },
    {
        "page_content": "Name: Haus 31\n\nAddress: Fiedlerstraße, 23, Dresden, 01307, DE\nbuilding: hospital\ncn_ukd:token: U0031\nname: Haus 31\n_osm_type: way\ndescription: A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
        "metadata": {
            "_osm_type": "way",
            "addr:city": "Dresden",
            "addr:country": "DE",
            "addr:housenumber": "23",
            "addr:postcode": "01307",
            "addr:street": "Fiedlerstraße",
            "building": "hospital",
            "cn_ukd:token": "U0031",
            "description": "A building which forms part of a hospital. Useamenity=hospitalfor the hospital grounds.",
            "feature": "{\n  \"type\": \"Polygon\",\n  \"coordinates\": [\n    [\n      [\n        13.78272,\n        51.055751\n      ],\n      [\n        13.782908,\n        51.05575\n      ],\n      [\n        13.782905,\n        51.055544\n      ],\n      [\n        13.78285,\n        51.055545\n      ],\n      [\n        13.78285,\n        51.055513\n      ],\n      [\n        13.782849,\n        51.055476\n      ],\n      [\n        13.783345,\n        51.055474\n      ],\n      [\n        13.783344,\n        51.055357\n      ],\n      [\n        13.782684,\n        51.055359\n      ],\n      [\n        13.782683,\n        51.055477\n      ],\n      [\n        13.782729,\n        51.055476\n      ],\n      [\n        13.782729,\n        51.055517\n      ],\n      [\n        13.782657,\n        51.055516\n      ],\n      [\n        13.78262,\n        51.055516\n      ],\n      [\n        13.782621,\n        51.055363\n      ],\n      [\n        13.782405,\n        51.055363\n      ],\n      [\n        13.782403,\n        51.055442\n      ],\n      [\n        13.782447,\n        51.055442\n      ],\n      [\n        13.782445,\n        51.0555\n      ],\n      [\n        13.782446,\n        51.055548\n      ],\n      [\n        13.782404,\n        51.055548\n      ],\n      [\n        13.782405,\n        51.055745\n      ],\n      [\n        13.782603,\n        51.055747\n      ],\n      [\n        13.782604,\n        51.055605\n      ],\n      [\n        13.782604,\n        51.055567\n      ],\n      [\n        13.782716,\n        51.055568\n      ],\n      [\n        13.782718,\n        51.055588\n      ],\n      [\n        13.782718,\n        51.055598\n      ],\n      [\n        13.78272,\n        51.055751\n      ]\n    ]\n  ]\n}",
            "name": "Haus 31",
            "type": "hospital",
            "url": "url",
            "relevant_words": [
                [
                    "hospital.",
                    0.04978396384931288
                ],
                [
                    "Haus",
                    0.04336857627039959
                ],
                [
                    "Dresden,",
                    0.03996199106464996
                ],
                [
                    "building",
                    0.037107618465903006
                ],
                [
                    "hospital",
                    0.029316511465435235
                ],
                [
                    "forms",
                    0.023264507322540103
                ],
                [
                    "building:",
                    0.017957887459665554
                ],
                [
                    "31",
                    0.017263015324133706
                ],
                [
                    "Fiedlerstraße,",
                    0.017134354914571746
                ],
                [
                    "DE",
                    0.01672150589513922
                ],
                [
                    "Haus",
                    0.01624414927058191
                ],
                [
                    "Useamenity=hospitalfor",
                    0.01457707736054114
                ],
                [
                    "description:",
                    0.011783178587512013
                ],
                [
                    "Address:",
                    0.008827190483165295
                ],
                [
                    "a",
                    0.008450949070797967
                ],
                [
                    "which",
                    0.007315367943198004
                ],
                [
                    "cn_ukd:token:",
                    0.0002555877375971543
                ]
            ]
        },
        "type": "Document"
    }
]

@simeonwetzel simeonwetzel moved this from In Progress to Done in 52°North Innovation Prize Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Development

No branches or pull requests

2 participants