Updates from last 9 months

.gitignore

@@ -1,9 +1,7 @@
 .idea/
 *.pyc
 
-people/friends/
-people/new/
-people/VU/
+people/**/*.bin
 tmp/
 
 test/news/tmp
@@ -14,7 +12,11 @@ tokens.json
 
 log.txt
 *.trig
+!.gitignore
 
 pepper/.DS_Store
 .DS_Store
 venv/
+
+
+papers/

HOWTO.md

@@ -0,0 +1,57 @@
+Setup
+-----
+
+### Run on Laptop
+1. Start Docker (and make sure you've pulled ```bamos/openface```)
+2. Start GraphDB (and make sure there is a repository called ```leolani```)
+3. Start ```object_detection.py``` within [pepper_tensorflow](https://github.com/cltl/pepper_tensorflow)
+4. Set ```APPLICATION_BACKEND``` to ```pepper.ApplicationBackend.SYSTEM``` in ```pepper/config.py```
+5. Start any Application in ```pepper/apps/..```
+6. Done
+
+### Run on Robot
+1. Start Docker (and make sure you've pulled ```bamos/openface```)
+2. Start GraphDB (and make sure there is a repository called ```leolani```)
+3. Start ```object_detection.py``` within [pepper_tensorflow](https://github.com/cltl/pepper_tensorflow)
+4. Set ```APPLICATION_BACKEND``` to ```pepper.ApplicationBackend.NAOQI``` in ```pepper/config.py```
+5. Set ```NAOQI_IP``` and ```NAOQI_PORT``` in accordance with robot's address
+6. Start any Application in ```pepper/apps/..```
+7. Done
+
+Pepper Troubleshooting
+----------------------
+> No Pepper-laptop connection can be established
+
+1. Make sure Pepper and the laptop are on the same network
+2. Verify Pepper has access to network (by pressing belly-button)
+3. Make sure ```NAOQI_IP``` and ```NAOQI_PORT``` are set correctly in ```pepper/config.py```
+
+> Pepper cannot connect to network
+
+1. Connect Pepper to network using ethernet cable
+2. Press belly-button to obtain IP and update ```pepper/config.py -> NAOQI_IP``` accordingly
+3. Go to robot web page (by entering IP in browser)
+4. Go to network settings and connect to wifi
+    - If unlisted, reboot robot (and wifi). Make sure wifi is online before robot is.
+5. Shutdown robot, remove ethernet cable, and boot again. It now should work...
+
+> Problems with speech audio
+
+1. Start an application with StatisticsComponent and look at the STT (Speech to Text) activity
+2. If no signal (i.e ```STT [..........]```):
+    1. Check if the NAOqi or System Microphone is used: ```pepper/config.py -> NAOQI_USE_SYSTEM_MICROPHONE```
+    2. Make sure external mic, if used, is switched on and sensitive enough (use OS settings)
+    3. Make sure Pepper mic, if used, is not broken?
+3. If signal is low (i.e ```STT [|||.......]``` < ```pepper/config.py -> VOICE_ACTIVITY_DETECTION_THRESHOLD```):
+    1. Make sure you talk loud enough (noisy fans in Peppers head make it difficult)
+    2. Make sure you talk in the right microphone (i.e. ```pepper/config.py -> NAOQI_USE_SYSTEM_MICROPHONE```)
+    3. Make sure the external mics volume is high enough!
+4. If signal is too high (i.e. ```STT [||||||||||]```) all the time:
+    1. Peppers own mics cannot handle very loud/noisy environments, like fairs
+    2. Use a microphone attached to the laptop, instead! 
+        - don't forget to set ```pepper/config.py -> NAOQI_USE_SYSTEM_MICROPHONE = True```
+5. Microphone should process audio at 16 kHz  (i.e. Statistics: ```Mic 16.0 kHz```), if not:
+    1. Lower ```CAMERA_RESOLUTION``` and/or ```CAMERA_FRAME_RATE``` in order to meet performance requirements
+
+
+

README.md

@@ -1,17 +1,102 @@
-# VU Amsterdam - CLTL - Pepper / Nao Framework
+# VU Amsterdam - CLTL - Robot Framework
 
-This is the main repository for Pepper/Nao Applications created as part of the Computational Lexicology & Terminology Lab (CLTL) at the Vrije Universiteit (VU), Amsterdam.
+Repository for Robot Applications created as part of the [Computational Lexicology & Terminology Lab (CLTL)](http://www.cltl.nl) at the Vrije Universiteit, Amsterdam.
+
+| **This is the original Python 2 version of the Leolani platform. This version was intended for the robot use as it supports the NAOqi backend. However, development on this repository has stopped due to the [sunsetting of Python 2](https://www.python.org/doc/sunset-python-2/). For the latest version of this software please see the [Python 3 version of the repository](https://github.com/leolani/pepper).** |
+|---|
 
 ![Pepper Robot Leolani](https://github.com/cltl/pepper/blob/develop/docs/images/pepper.png)
 
-## CLTL/Pepper Features
- - A Platform-Agnostic, Object-Oriented Framework for creating Robot Applications
- - The necessary Machine Learning for enabling Human-Robot conversation
- - Language Understanding through Grammars
- - Knowledge Representation of Learned Facts through RDF Graph
- - Realtime visualisation in web browser (WIP)
+### Features
+ - A framework for creating interactive Robot Applications using Python, to enable:
+   - Human-Robot conversation using Speech-to-Text and Text-to-Speech
+   - Recognising friends by face and learning about them and the world through conversation
+   - Recognising and positioning the people and objects in its enviroment.
+ - Natural Language Understanding through Syntax Trees (Grammars)
+ - Knowledge Representation of all learned facts through a RDF Graph: the robot's Brain!
+ - Curiosity based on Knowledge Gaps and Conflicts resulting from learned facts
+ - Realtime visualisation in web browser
+
+## Getting started
+Check out our [WIKI](https://github.com/cltl/pepper/wiki) for information on [how it works](https://github.com/cltl/pepper/wiki/2.-How-it-works) and [how to set up](https://github.com/cltl/pepper/wiki/1.-Set-up).
+
+Check out our [API Reference](https://cltl.github.io/pepper/) and [Sample Applications](https://github.com/cltl/pepper/tree/develop/apps/examples)!
+
+
+### Prerequisites
+
+* NAOqi Python SDK (version 2.5.10) from the [Softbank robotics download page](https://www.softbankrobotics.com/emea/en/support/pepper-naoqi-2-9/downloads-softwares/former-versions?category=108)
+* [Python 2.7.10](https://www.python.org/downloads/release/python-2710/) 
+* An application on [Wolfram Alpha](https://products.wolframalpha.com/api/) with API access
+* A project on the [Google Cloud Platform](https://cloud.google.com/speech-to-text/docs/quickstart-client-libraries) supporting Text-To-Speech and Speech-To-Text APIs
+* [Docker Engine](https://docs.docker.com/engine/install/)
+* [OpenFace](http://cmusatyalab.github.io/openface/) 
+* Pepper's sister-project: [pepper_tensorflow](https://github.com/cltl/pepper_tensorflow), which includes the Tensorflow services within Python 3.6.
+* [GraphDB](http://graphdb.ontotext.com/) with a repository named `leolani`
+
+Detailed instructions on [how to set up](https://github.com/cltl/pepper/wiki/1.-Set-up).
+
+
+## More information
+More information on the Make Robots Talk project at CLTL can be found on its [website](http://makerobotstalk.nl)
+
+###  Citation
+
+Please cite (at least one of) the following papers if you use this software in your research:
+
+[Leolani: a reference machine with a theory of mind for social communication](https://arxiv.org/abs/1806.01526)
+```
+@inproceedings{vossen2018leolani,
+      title={Leolani: a reference machine with a theory of mind for social communication},
+      author={Vossen, Piek and Baez, Selene and Bajc̆eti{\'c}, Lenka and Kraaijeveld, Bram},
+      booktitle={International conference on text, speech, and dialogue},
+      pages={15--25},
+      year={2018},
+      organization={Springer},
+}
+```
+
+[A communicative robot to learn about us and the world](http://www.dialog-21.ru/media/4636/vossenpplusetal-050.pdf)
+```
+@inproceedings{vossen2019communicative,
+      title={A communicative robot to learn about us and the world},
+      author={Vossen, Piek and Santamaria, Selene Baez and Bajc̆eti{\'c}, Lenka and Ba{\v{s}}i{\'c}, Suzana and Kraaijeveld, Bram},
+      booktitle={2019 Annual International Conference on Computational Linguistics and Intellectual Technologies, Dialogue 2019},
+      pages={728--743},
+      year={2019}
+}
+```
+
+[Leolani: A robot that communicates and learns about the shared world](http://ceur-ws.org/Vol-2456/paper47.pdf)
+```
+@inproceedings{vossen2019leolani,
+      title={Leolani: A robot that communicates and learns about the shared world},
+      author={Vossen, Piek and Baez, Selene and Baj{\v{c}}eti{\'c}, Lenka and Ba{\v{s}}i{\'c}, Suzana and Kraaijeveld, Bram},
+      booktitle={2019 ISWC Satellite Tracks (Posters and Demonstrations, Industry, and Outrageous Ideas), ISWC 2019-Satellites},
+      pages={181--184},
+      year={2019},
+      organization={CEUR-WS}
+}
+```
+
+[Modelling context awareness for a situated semantic agent](https://link.springer.com/chapter/10.1007/978-3-030-34974-5_20)
+```
+@inproceedings{vossen2019modelling,
+      title={Modelling context awareness for a situated semantic agent},
+      author={Vossen, Piek and Baj{\v{c}}eti{\'c}, Lenka and Baez, Selene and Ba{\v{s}}i{\'c}, Suzana and Kraaijeveld, Bram},
+      booktitle={International and Interdisciplinary Conference on Modeling and Using Context},
+      pages={238--252},
+      year={2019},
+      organization={Springer}
+}
+```
+
+### License
 
-## Get started
-Check out our [WIKI](https://github.com/cltl/pepper/wiki) for information on [how it works](https://github.com/cltl/pepper/wiki/How-it-works) and [how to Install the package](https://github.com/cltl/pepper/wiki/Installation).
+Distributed under the MIT License. See [`LICENSE`](https://github.com/cltl/pepper/blob/develop/LICENCE) for more information.
 
-Check out our [API Reference](https://cltl.github.io/pepper/) for detailed information on the framework.
+### Authors
+* [Bram Kraaijeveld](https://github.com/as-the-crow-flies)
+* [Selene Báez Santamaría](https://github.com/selBaez)
+* [Lenka Bajc̆etic](https://github.com/lenkaB)
+* [Piek Vossen](https://github.com/piekvossen)

apps/eliza_app.py

@@ -0,0 +1,95 @@
+"""Example Application that answers questions posed in natural language using Wikipedia"""
+
+from pepper.framework import *              # Contains Application Building Blocks
+from pepper.knowledge import Wikipedia      # Class to Query Wikipedia using Natural Language
+from pepper import config                   # Global Configuration File
+from pepper.responder import eliza
+from pepper.knowledge import sentences, animations
+
+
+
+SPEAKER_NAME_THIRD = "Dear patient"
+SPEAKER_NAME = "Dear patient"
+SPEAKER_FACE = "HUMAN"
+DEFAULT_SPEAKER = "Human"
+NAME = "Sigmund Freud"
+
+MIN_ANSWER_LENGTH = 4
+# Override Speech Speed for added clarity!
+config.NAOQI_SPEECH_SPEED = 80
+
+
+class ElizaApplication(AbstractApplication,         # Every Application Inherits from AbstractApplication
+                           StatisticsComponent,         # Displays Performance Statistics in Terminal
+                           SpeechRecognitionComponent,  # Enables Speech Recognition and the self.on_transcript event
+                           TextToSpeechComponent):      # Enables Text to Speech and the self.say method
+
+    SUBTITLES_URL = "https://bramkraai.github.io/subtitle?text={}"
+
+
+    def __init__(self, application):
+        """Greets New and Known People"""
+        self.name_time = {}  # Dictionary of <name, time> pairs, to keep track of who is greeted when
+
+        super(ElizaApplication, self).__init__(application)
+
+        IntroductionIntention(self).speech()
+        sleep(2.5)
+
+
+    def show_text(self, text):
+        text_websafe = text
+        # text_websafe = urllib.quote(''.join([i for i in re.sub(r'\\\\\S+\\\\', "", text) if ord(i) < 128]))
+        self.backend.tablet.show(self.SUBTITLES_URL.format(text_websafe))
+
+    def on_transcript(self, hypotheses, audio):
+        """
+        On Transcript Event.
+        Called every time an utterance was understood by Automatic Speech Recognition.
+
+        Parameters
+        ----------
+        hypotheses: List[ASRHypothesis]
+            Hypotheses about the corresponding utterance
+        audio: numpy.ndarray
+            Utterance audio
+        """
+
+        # Choose first ASRHypothesis and interpret as question
+        for h in hypotheses:
+            question = h.transcript
+            self.show_text(question)
+
+            answer = eliza.analyze(question)
+
+            if answer:
+                print("You said:", question)
+                # Tell Answer to Human
+                self.show_text(answer)
+                self.say(answer)
+                break
+
+
+class IntroductionIntention(AbstractIntention, ElizaApplication):
+
+
+    def speech(self):
+        # 1.1 - Welcome
+        self.say("Hello. Welcome to my clinic", animations.BOW)
+        self.say("My name is "+NAME)
+        self.say("I am your best friend and personal therapist", animations.MODEST)
+        self.say("How do you feel today?", animations.FRIENDLY)
+        sleep(3.5)
+
+
+if __name__ == "__main__":
+
+    # Get Backend from Global Configuration File
+    backend = config.get_backend()
+
+
+    # Create Application with given Backend
+    application = ElizaApplication(backend)
+
+    # Run Application
+    application.run()

apps/examples/greeting.py

@@ -8,6 +8,8 @@
 
 class GreetingApplication(AbstractApplication,          # All Applications inherit from AbstractApplication
                           FaceRecognitionComponent,     # We need Face Recognition to Greet People by Name
+                          StatisticsComponent,
+                          SpeechRecognitionComponent,
                           TextToSpeechComponent):       # We need Text to Speech to actually greet people
 
     GREET_TIMEOUT = 15  # Only Greet people once every X seconds