Future Work on the NLP Apps Notebook

[antmarakis]

Recently in the nlp_apps notebook I added a section on the Federalist Papers. What I did was write a simple workflow from start to finish. There is a lot of work to be done still and I am opening this to community contributions, since I believe it is a great way to get started with the applications notebooks.

A few ways you can improve the section:

• DONE - One big issue with the Naive Bayes Classifier in this problem is that the multiplication of probabilities causes underflow (all the probability multiplications result in 0.0). That happens because examples are long texts. To avoid this, we are currently using the decimal module of Python. I believe this problem can be solved more elegantly using the logarithm of probabilities instead of probabilities. So instead of multiplying the probabilities, we add their logarithms.

• Do some pre-processing. Currently I only added a sample pre-processing step (removing one common word from each paper). I would like to see some other pre-processing tasks + analysis of the text. Which are the most common words for each author? Is it worth it if we removed the most popular words?

• Right now we are using unigram word models. There are other options available too. I would like to explore this in the notebooks. Maybe an author likes using two words together. Maybe another spells some words a bit differently. I would like to see different models used/explored in the notebook, to let the readers know that they shouldn't rely on just one model all the time. We can even combine models together.

• At the end of the notebook I note that the dataset is lopsided. We have way more information on Hamilton than the other two. Maybe it is worth adding some more writings from Jay/Madison to balance this out. I think it would be interesting to see if we could improve the results by using external data. This could come after the current section, so that we could compare the results.

• Finally, maybe we can take a step back and try and classify all the Federalist papers, not just the disputed ones. Add a new section where we use external data to train our model and then try and classify the papers.

---

This is a big undertaking, and it doesn't need to happen on the particular problem. If you have a problem in mind, you can instead use the above ideas to tackle your own problems! Sentiment Analysis is trending right now, so maybe this is a place to explore some of the above.

All in all, I think this is a good project to chip in every once in a while and I hope it will serve as an introduction to the repository. Or maybe it will sound interesting to GSoC students who might choose to tackle this.

In any case, feel free to post here with ideas + if you want to start working on something.

[Dimkoim]

Hello @MrDupin! I have worked in NLP problems and I can give you some help here. :)

[antmarakis]

That's great! You can work on them either on GSoC or on your own. Whenever you want to get started, don't forget to post a comment here explaining in short what you want to do.

[codingblazes]

@MrDupin With reference to point 3,I would like to include support for both N-gram and Bag of Words models.I am mentioning the same in my proposal.

[antmarakis]

@ashwinnalwade Awesome! Good luck with your application!

[prasadgujar]

Hello, Reference to the 2nd point . We can do some preprocessing task like ranking words/sentence in paper with reference to frequency of word/sentence occurence and uniqueness of each word by considering ranking factor like the length of word/sentence.

[dsaw]

Hello, I would like to work on the first point on implementing log addition if no one is doing it yet.
We can retain the decimal library Naive Bayes implementation and add logarithmic version with it. It might be good to compare both approaches in the notebook.

[ad71]

Sure @dsaw you can get started, nobody is working on this yet.

[reachtarunhere]

@MrDupin I did some work related to language identification during my GSoC (the year before yours). This follows from an example in the book. If anyone is keen on taking this up - here is the notebook: https://github.com/reachtarunhere/aima-python/blob/lang-id/lang-id.ipynb

If not, I would be happy to clean it up and get it merged myself.

[antmarakis]

@reachtarunhere I only skimmed through it, but it looks good. Maybe the current GSoC students can take it up?

[ad71]

@MrDupin I would have loved to take it up but I have no knowledge of NLP. I'll definitely come back to this once I get acquainted with the topic.

[reachtarunhere]

@ad71 or anyone else taking this up - feel free to get in touch if you need any help/clarifications. In case I miss this here (lots of notifs) you can always drop an email :)

[llucifer97]

hi @MrDupin I have some experience in nlp would like to solve the issue if this issue is still open
I would be working on case 2 , would like to use different methods like bag of model.

[antmarakis]

@llucifer97: You can definitely work on the issue! Simply fork the repository, make your changes to the notebook, and submit a Pull Request.

If you need any help, feel free to ask (although I am swamped with work at the moment so I am not sure I will be able to provide much help).

[aditya-hari]

Hello @MrDupin.

Would I be right in assuming that this isn't an issue that someone completely new to opensource and AI-related topics can handle? I am well versed with Python but have never dabbled with this sort of thing.

Apologies if this is a very trivial question.

[antmarakis]

@aditya-hari: Hmmm, I'm not sure... I don't think any past experience with open-source is necessary, but I believe some understanding of the AI concepts covered in this notebook is a must. If you want you can take it on, but it seems to me that it will be difficult to make much progress efficiently, mainly because not only do you have to execute the concepts, but also showcase them in a manner in which others can understand them.

It is up to you, but I first suggest you read up on NLP and some basic AI material in order to be better equipped to tackle this challenge.

[aditya-hari]

Is encouraging to know that past experience isn't necessary.

Could you kindly guide me to some resources which can help me get up to speed? Or point me to some other issues which would be easier to start off with?

Would really appreciate any help.

[antmarakis]

Hi @aditya-hari, sorry for taking so long to get back to you.

The way this repository works no longer involves issues (there are exceptions though). The main work remaining is to simply add to the notebooks.

For NLP, I am afraid there's not much I can say. Personally, I was just looking around the internet for information, I was never taught anything in a class. I suggest you snoop around and try to find a university course or something that is open to the public. Then you can follow its structure, complementing your studying with googling.

[vaibhavshukla182]

Hi, @MrDupin
I have never contributed to open source before this so pardon me if i say anything trivial...but can you please tell me that how could I contribute to your projects, I do have some NLP experience, have implemented a research paper to code on text analysis, so can you please tell me that what contribution I could do.

[antmarakis]

@vaibhavshukla182: Hi! Since most of the algorithms in this project have been implemented, I suggest you turn to providing examples for the notebooks. Maybe some text analysis, or a tutorial on NLP techniques, or stuff of this nature. Remember that this project is educational, so the more examples we have the better. Just don't forget to include some instructions on how your examples work.

Also, I believe you can implement algorithms outside the AIMA book as well. But only if they have short implementations and easy testing under pytest. If you have any questions, feel free to ask!

[vaibhavshukla182]

Can I add some data preprocessing and data visualisation techniques?
Or something like nlp for fuzzy string matching, or a comparison of various methods for text classification and which one is better in which situation.

[antmarakis]

@vaibhavshukla182: This sounds great! For the nlp stuff though, don't use any external libraries. We want to keep all the code right in front of the student, with no middle-man. Feel free to use visualization libraries (preferably matplotlib) for the viz stuff.

[rishabhdash]

@MrDupin I too want to work on the visualisation technique. I was thinking of implementing t-SNE for dimensionality reduction. I know this will not bring in great results for a big dataset. Also i want to add tf-idf and word2vec in addition to BOW and n grams. I can also work on with different algos on the same dataset like KNN and time split(temporal) data KNN and so on.

[rishabhdash]

Also i think i might have to add some additional dataset on which i can do some super wised classification and do a polarity based t-sne visualisation as well as for KNN. Please see this for dataset recommendation:
aimacode/aima-data#10 (comment)

[antmarakis]

@cursed4ever I think for now TF.IDF would be a nice addition, if you want to get started with it.

[antmarakis]

@hackerashish25 What external library are you planning on using? Quite a while ago I wrote an implementation for tf-idf, and it did not require any libraries, if I remember correctly.

[Ask149]

Also, I believe you can implement algorithms outside the AIMA book as well. But only if they have short implementations and easy testing under pytest. If you have any questions, feel free to ask!

Hello @MrDupin, I want to contribute to aima-python. It seems most of the algorithms have already been implemented. Can I propose and work on some AI optimization algorithms? ( If adding new algorithms outside AIMA book is valid) I was suggesting to implement algorithms like Stochastic Gradient Descent, Particle Swarm Optimization, etc. I would also like to work on other notebook completions if the above algorithms implementation is not necessary for the project. Thank You!

[antmarakis]

Hello @MrDupin, I want to contribute to aima-python. It seems most of the algorithms have already been implemented. Can I propose and work on some AI optimization algorithms? ( If adding new algorithms outside AIMA book is valid) I was suggesting to implement algorithms like Stochastic Gradient Descent, Particle Swarm Optimization, etc. I would also like to work on other notebook completions if the above algorithms implementation is not necessary for the project. Thank You!

@Ask149: I am not familiar with Particle Swarm, but SGD is something that can be implemented here, sure!

[JayantSravan]

Hey @MrDupin,
I want to contribute to aima-python. I am interested and experienced in the fields of NLP, genetic algorithms, particle swarm optimizations, reinforcement learning and information retrieval. Any pointers you could give me on what project should I take up? I want to do it as my GSoC project.

[ShaswatLenka]

Hello @MrDupin , Can I take on "sentiment analysis" for the apps notebook if no one is working on it? Also I would like to implement customisable vectorisers as utility functions that can come handy in many situations.

[aasthasood]

Hello @MrDupin, I am interested in starting with sentiment analysis and would like to contribute to the NLP Apps notebook. Any pointers you could give me on project selection, as I want to do this for my GSoC project.

[antmarakis]

Hello all (@JayantSravan, @ShaswatLenka, @aasthasood). Thank you for your interest in this project.

For sentiment analysis, you can pick a project and work on it. A popular one is the movie reviews dataset. I would prefer if you picked something else (you can research "sentiment analysis datasets" on Google). Pick a relatively small dataset and work on it.

Remember though that you cannot use any external libraries for the training, you have to do it yourselves to showcase the algorithm.

[0xrushi]

Hi @MrDupin, can you repost what else is left to be done i see log,visualization,preprocessing is already completed, can i go with sentiment analysis ?

[antmarakis]

@rushic24: Sure, you can do solve some sentiment analysis problems. Sounds good!

[Kaustav97]

Hi @MrDupin , has anyone taken up the sentiment analysis example? If not, I can contribute since I have prior work on IMDB dataset, which I can probably re-use. Also, I can take up the explanation of the 'Question Answering' section in nlp.ipynb? Thanks.

[0xrushi]

@rushic24: Sure, you can do solve some sentiment analysis problems. Sounds good!

@MrDupin i had written twitter sentiment analysis few weeks back, can that be put here ?

[antmarakis]

@Kaustav97: Sure, you can do Sentiment Analysis and/or Question Answering. Just remember to only use as basic Python as possible to explain the algorithms.

@rushic24: That sounds great as well.

[thesagarsehgal]

@MrDupin Can I add Soundex based phonetic search in nlp_apps notebook? I don't think I would be needing any external dataset for this.

[thesagarsehgal]

@MrDupin I was planning to add Text Classification, using Naive Bayes Classification into different classes like if "apple" in a statement refers to apple inc. company statement or apple fruit or any other categories. or can you please specify any other category in which we can classify sentences which would be more easy to understand. Can I proceed further with adding it in the nlp_apps.ipynb? Also, I had 1 doubt, should I add a dataset for this, or should I work out an example using just 4-5 example sentences of each class. And then predict the class of the new sentence.

[antmarakis]

@sagar-sehgal: You can add Soundex, if the implementation is simple enough (I have not heard of the algorithm). Also, you can work on classification, even though it would be best if you used datasets already in aima-data.

[thesagarsehgal]

@MrDupin Soundex in a phonetic coding algorithm which gives a code of a specific length to every word as pronounced in English. It helps in searching through the documents even if we have made a spelling mistake (because of the difference in the way of pronounciation) since the Soundex code would still remain the same. For eg:- "Robert" and "Rupert" both have Soundex code as "R163".

For Classifiaction, instead of using a dataset I was planning to define 2-3 sentences of each class in the notebook itself.

Thank You!

[antmarakis]

@sagar-sehgal: Soundex sounds good, although it all depends on the implementation. If it is short and there is a nice description to go with it, it's all fine.

For the Classification task, if defining 2-3 sentences for each class in the notebook works, then it is great!

[thesagarsehgal]

@MrDupin As I had mentioned earlier, I was working on Text Classification. Since Text Classification is a broad topic and many other topics Sentiment Analysis, Topic Labelling, etc. come under it.

So I planned to go with Topic Labelling on whether apple word in a sentence refers to fruit or company class. I planned to go with Naive Bayes classification and wrote everything completely and then found that similar work has already been done under Author Identification. It is so because Authorship Identification can also be seen as a Text Classification task b/w 2 Authors.

Here, in this case, we would just be having the same application, but trained on different data. So, we could achieve similar results, even without any coding. We would just have to change the data in the dataset directory and then run the application.

So, after all this should I still push the PR of Text Classification?

[antmarakis]

You can still push your work on Topic Labeling, as long as you explain what is going on. The solution may be pretty much the same, but the application is different, so having an example on it won't hurt. Reusing the same code to solve a different problem is even better, showcasing that there is an underlying structure to a lot of NLP problems.

[thesagarsehgal]

@MrDupin I wrongly referred the concept as of Topic Labelling. Classifying the word apple as referring to the company or fruit is Word Sense Disambiguation and not Topic Labelling. Although both of them are applications under Text Classification and follow the same procedure. I am really sorry for referring the wrong name.

[antmarakis]

The PR looks fine, thanks! I have made some comments.

[memogamd21]

In the language recognition problem, there might be a solution for very long words to be segmented such as lebensversicherungsgesellschaftsangestellter to better perform on the language classifier. This word for example is in german, so if you use word segmentation, it might help perform better in the language recognition specially that the two classes contain german language which already has very long words

[gamechanger98]

can we use some more proper classifier to identify the author?

[thesagarsehgal]

@gamechanger98 Naive Bayes is a Machine Learning Classifier and it has been coded without using any other external library. Other classifiers do exist in NLP but they use Deep Learning Techniques and need external libraries. But here all the code has to be coded without using an external library. If you code up a better Machine Learning Classifier, then it can also be included.

[thesagarsehgal]

@memogamd21 Yes, that seems to be a nice idea. It can increase performance. You can code that up and if it increases the accuracy then it would be really nice and can be added.
But I have a doubt. For applying word segmentation on a test sentence, we need to know the language initially, since the segmentation model for both the languages would be different. How can we handle that?

[memogamd21]

@sagar-sehgal well, we can use the maximum matching algorithm in which we can compare the word we have with the longest ever word in our two corpora (german and english one) and if it matches one of them, the language can be classified easily. I mean like we can make the long words list as a list of tuples (word, language it belongs to) and then from that detect the lanuage :)