Cosine_Similarity_Plagerism_Checker.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Cosine_Similarity - Plagerism Checker.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZxyshVJaL9oE"
      },
      "source": [
        "Import"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8xaR52ZeLpQu"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yQK1rpU_gPhI"
      },
      "source": [
        "Load file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "izn48HX9gSDJ"
      },
      "source": [
        "my_text ='''TfidfVectorizer converts a collection of raw documents to a matrix of Tfidf features. Text analysis is a major application field in machine learning algorithms.The text must first to numerical feature vectors because the computer wants to process numeric data.\n",
        "Sklearn accomplishes natural language text processing by: tokenizing the strings and assigning an integer for each possible token; counting the occurrences of each token in each document; and normalising and weighting with diminishing importance the tokens that occur in the majority of text documents. When transforming the text data into numeric data, each individual token occurrence frequency is treated as a feature.\n",
        "A corpus of documents can be represented as one row of data for each document and one column, or feature, for each token.\n",
        "Vectorization is the general process of turning a collection of text documents into numerical feature vectors. This specific strategy is called Bag of Words representation. Documents are described by word occurrences while ignoring the relative position of the words in the document'''"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yMq7Sh9Qjh6D"
      },
      "source": [
        "sk_text = '''Text Analysis is a major application field for machine learning algorithms. However the raw data, a sequence of symbols cannot be fed directly to the algorithms themselves as most of them expect numerical feature vectors with a fixed size rather than the raw text documents with variable length.\n",
        "\n",
        "In order to address this, scikit-learn provides utilities for the most common ways to extract numerical features from text content, namely:\n",
        "\n",
        "tokenizing strings and giving an integer id for each possible token, for instance by using white-spaces and punctuation as token separators.\n",
        "\n",
        "counting the occurrences of tokens in each document.\n",
        "\n",
        "normalizing and weighting with diminishing importance tokens that occur in the majority of samples / documents.\n",
        "\n",
        "In this scheme, features and samples are defined as follows:\n",
        "\n",
        "each individual token occurrence frequency (normalized or not) is treated as a feature.\n",
        "\n",
        "the vector of all the token frequencies for a given document is considered a multivariate sample.\n",
        "\n",
        "A corpus of documents can thus be represented by a matrix with one row per document and one column per token (e.g. word) occurring in the corpus.\n",
        "\n",
        "We call vectorization the general process of turning a collection of text documents into numerical feature vectors. This specific strategy (tokenization, counting and normalization) is called the Bag of Words or “Bag of n-grams” representation. Documents are described by word occurrences while completely ignoring the relative position information of the words in the document'''"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ypT6KrFisQ-y"
      },
      "source": [
        "Corpus"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IVTBfWNIsTJg"
      },
      "source": [
        "corpus = [my_text, sk_text]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EqYSlFk_hXBt"
      },
      "source": [
        "TdIdf Vectorizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        },
        "id": "yftIa0Lxskbs",
        "outputId": "002b7f32-5e26-4075-fd4c-17cf1fec2b9f"
      },
      "source": [
        "#see:- https://kanoki.org/2018/12/27/text-matching-cosine-similarity/\n",
        "\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "vectorizer = TfidfVectorizer()\n",
        "trsfm=vectorizer.fit_transform(corpus)\n",
        "pd.DataFrame(trsfm.toarray(),columns=vectorizer.get_feature_names(),index=['my_doc','sk_doc'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>accomplishes</th>\n",
              "      <th>address</th>\n",
              "      <th>algorithms</th>\n",
              "      <th>all</th>\n",
              "      <th>an</th>\n",
              "      <th>analysis</th>\n",
              "      <th>and</th>\n",
              "      <th>application</th>\n",
              "      <th>are</th>\n",
              "      <th>as</th>\n",
              "      <th>assigning</th>\n",
              "      <th>bag</th>\n",
              "      <th>be</th>\n",
              "      <th>because</th>\n",
              "      <th>by</th>\n",
              "      <th>call</th>\n",
              "      <th>called</th>\n",
              "      <th>can</th>\n",
              "      <th>cannot</th>\n",
              "      <th>collection</th>\n",
              "      <th>column</th>\n",
              "      <th>common</th>\n",
              "      <th>completely</th>\n",
              "      <th>computer</th>\n",
              "      <th>considered</th>\n",
              "      <th>content</th>\n",
              "      <th>converts</th>\n",
              "      <th>corpus</th>\n",
              "      <th>counting</th>\n",
              "      <th>data</th>\n",
              "      <th>defined</th>\n",
              "      <th>described</th>\n",
              "      <th>diminishing</th>\n",
              "      <th>directly</th>\n",
              "      <th>document</th>\n",
              "      <th>documents</th>\n",
              "      <th>each</th>\n",
              "      <th>expect</th>\n",
              "      <th>extract</th>\n",
              "      <th>feature</th>\n",
              "      <th>...</th>\n",
              "      <th>sklearn</th>\n",
              "      <th>spaces</th>\n",
              "      <th>specific</th>\n",
              "      <th>strategy</th>\n",
              "      <th>strings</th>\n",
              "      <th>symbols</th>\n",
              "      <th>text</th>\n",
              "      <th>tfidf</th>\n",
              "      <th>tfidfvectorizer</th>\n",
              "      <th>than</th>\n",
              "      <th>that</th>\n",
              "      <th>the</th>\n",
              "      <th>them</th>\n",
              "      <th>themselves</th>\n",
              "      <th>this</th>\n",
              "      <th>thus</th>\n",
              "      <th>to</th>\n",
              "      <th>token</th>\n",
              "      <th>tokenization</th>\n",
              "      <th>tokenizing</th>\n",
              "      <th>tokens</th>\n",
              "      <th>transforming</th>\n",
              "      <th>treated</th>\n",
              "      <th>turning</th>\n",
              "      <th>using</th>\n",
              "      <th>utilities</th>\n",
              "      <th>variable</th>\n",
              "      <th>vector</th>\n",
              "      <th>vectorization</th>\n",
              "      <th>vectors</th>\n",
              "      <th>wants</th>\n",
              "      <th>ways</th>\n",
              "      <th>we</th>\n",
              "      <th>weighting</th>\n",
              "      <th>when</th>\n",
              "      <th>while</th>\n",
              "      <th>white</th>\n",
              "      <th>with</th>\n",
              "      <th>word</th>\n",
              "      <th>words</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>my_doc</th>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.167032</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.083516</td>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.083516</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.083516</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.167032</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.125274</td>\n",
              "      <td>0.208790</td>\n",
              "      <td>0.250548</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.167032</td>\n",
              "      <td>...</td>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.250548</td>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.459338</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.125274</td>\n",
              "      <td>0.167032</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.083516</td>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.058689</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.041758</td>\n",
              "      <td>0.083516</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sk_doc</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.069159</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.207477</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.069159</td>\n",
              "      <td>0.138318</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.069159</td>\n",
              "      <td>0.069159</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.103739</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.069159</td>\n",
              "      <td>0.069159</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.138318</td>\n",
              "      <td>0.172898</td>\n",
              "      <td>0.103739</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.103739</td>\n",
              "      <td>...</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.138318</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.484114</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.103739</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.103739</td>\n",
              "      <td>0.172898</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.069159</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.069159</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.034580</td>\n",
              "      <td>0.0486</td>\n",
              "      <td>0.138318</td>\n",
              "      <td>0.069159</td>\n",
              "      <td>0.069159</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2 rows × 153 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "        accomplishes  address  algorithms  ...      with      word     words\n",
              "my_doc      0.058689   0.0000    0.041758  ...  0.041758  0.041758  0.083516\n",
              "sk_doc      0.000000   0.0486    0.069159  ...  0.138318  0.069159  0.069159\n",
              "\n",
              "[2 rows x 153 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GQfgdBBmtod1"
      },
      "source": [
        "Cosine similarity"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dv-7SiOvtrrN",
        "outputId": "f2b62104-de04-469f-cea0-f07a8e90ff2e"
      },
      "source": [
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "\n",
        "cosine_similarity(trsfm[0:1], trsfm)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1.        , 0.83172947]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    }
  ]
}