Pepp Talk: User Guide

Note: The install script takes roughly 30 minutes to run, so it's best to do something else as you're waiting. Also, if the install script does not have the correct permissions use the command chmod 777 install.sh then execute the install script with ./install.sh

Set the Absolute path for the PeppTalkProduct folder

This process is outlined in the Pepp Talk cannot find my .template file! section.

Setup the RESTful Apache Server

This is an essentail step that will setup the Apache RESTful server and connect it with the UnQLite database enabling a user to store data through REST requests to the server. The Setup guide for Apache can be found in the Preparing Your Database with Apache section.

Write a Web Seed

Go to Understanding the Database Processing API section to understand the web seed.

Writing a Web Scraping Seed

Using Python to Write Seeds

PeppTalkProduct/Seeds/Examples

Example Seed

Setup a Cron job

Setting up a Cron job can be found in the Making Repeated Calls to the Database (cron job) section of the user guide.

Write an IoT-based Seed

Creating a IoT Seed is very similar to the writing a Web Seed. The process is outlined in the Writing an OpenHab Seed section. The example IoT Seed can be found in the PeppTalkProduct/Seeds/Examples folder.

Create a new User Template

The User Template is where a user creates the text that they would like to hear in their Pepp Talk report. The User Templates section outlines how to create your very own user templates and has an Example Template to use as reference.

Run your newly created Pepp talk

Now that we have created all of the components of our Pepp Talk it is now time to run the Pepp Talk. This process is outlined in the Running Pepp Talk section of the user guide. (Note: if the report_generator.sh script does not have the correct permissions use the command chmod 777 report_generator.sh)

Dependencies

Inside the PeppTalkProduct folder is an install.sh Shell Script. Running this script will install all our dependencies for you. However, this may install unnecessary files; for example, our script runs sudo apt-get upgrade, which may install unnecessary files onto your system. If you are concerned with which files are installed on your system, please view the script and select the dependencies you wish to install. Please note however, that our product may not work if you do not install everything in the script.

Do note that if the Linux environment is not running on a Raspberry Pi, sudo rpi-update will fail. This is OK as long as you are not running the Linux environment on a Raspberry Pi. If sudo rpi-update fails and you are running on a Raspberry Pi, please refer to your Raspberry Pi's official documentation on this issue.

Below are the dependencies the script installs. Small descriptions are provided in case apt-get or pip fails and you have to install manually.

sudo apt-get install default-jdk

Installs the default Java Development Kit, used by openHab sudo apt-get install python 2.7.6

Installs Python 2.7.6, which most of our scripts run on sudo apt-get install python-pip

Installs Python Pip, used to install dependencies further along on this list sudo apt-get install apache2

Installs Apache 2 which the Database runs on sudo apt-get install python-dev

Installs Python development kit used to run the Database sudo apt-get install libapache2-mod-wsgi

Installs a library dependency for Apache 2 sudo pip install bottle

Installs Bottle, used to handle REST calls from Python sudo pip install UnQLite

Installs UnQLite, the Database we are using sudo apt-get install curl

Installs curl, which helps with HTTP calls in the command line sudo pip install requests

Installs requests which allows you to make HTTP requests from Python sudo pip install datetime

Installs datetime, which turns timestamps into a date sudo apt-get install Cython

Installs Cython, which UnQLite requires to access C bindings sudo apt-get install festival

Installs Festival, which turns text files (aka Pepp Text) into .wav files (aka Pepp Talk) sudo apt-get install vlc

Installs VLC Media Player, which is used to play the .wav files

We are no longer using Beautiful Soup 4, a web scraping tool. You may still want to install Beautiful Soup 4 for use in your own Seeds. To do so, use:

sudo pip install beautifulsoup4

Preparing Your Database with Apache

The PeppTalk Database wrapper is designed to run as a webservice.
You can use your favorite way of hosting Python WSGI files to host it.
Here we describe the way to deploy it so that it is configured to work with the example applications we have provided.
These instructions are designed for a Debian based box (like Raspbian or Ubuntu)

The PeppTalk Database wrapper requires several dependencies. Note if you executed the install.sh script then the application dependencies have already been installed.

Configuring Apache

We will make minimal changes to apache from its default configuration.

Check if apache is running by typing curl localhost: 80 Running this command should display a default web page. For example, you might see the following:

<html>
    <body>
        <h1>It works!</h1>
        <p>This is the default web page for this server.</p>
        <p>The web server software is running but no content has been added, yet.</p>
    </body>
</html>

Navigate to the apache configuration directory. This command should by default be cd /etc/apache2/. This folder has a bunch of conf files and other things. We will change ports.conf and the provided default site. If you run your own server, you will probably want to set up the page as a seperate site. You can use the test editor of your choice to edit ports.conf such as nano make sure to edit this file using sudo because the file is read only.
Edit ports.conf. Find the lines that says:

NameVirtualHost *:80
    Listen 80

and replace 80 with 43508. You can use any port you wish, as long as you keep the other componets of Pepp Talk up to date. If NameVirtualHost does not exist add it in with port 43508.

Navigate into etc/apache2/sites-enabled. Use your prefered text editor to edit the default site. It is probably called 000-default. Again make sure to edit this file with sudo. In that file, within the virtual host tags, add this in the file:

WSGIScriptAlias /PeppTalk /var/www/pepptalk/app.wsgi

<Directory /var/www/pepptalk>
    WSGIProcessGroup test
    WSGIApplicationGroup %{GLOBAL}
    Order deny,allow
    Allow from all
</Directory>

Also change the tag <VirtualHost *:80> at the top of the file by replacing 80 with 43508

This will serve the pepptalk app WSGI to the /PeppTalk/ extension of your server URL.

Return to the main apache2 directory. Open up apache2.conf in a text editor (remember to use sudo). Near the bottom of apache2.conf, above Include sites-enabled/ add the following line to the file:

WSGIDaemonProcess test

Now that apache is configured to serve the WSGI, we need to place the python package and create the WSGI file to serve it.

Placing Python Packages

cd /var/www
sudo mkdir pepptalk
sudo chmod 777 pepptalk because we need to make sure the apache user is able to access our files.
Copy the DatabaseWrapper folder provided in the pepp talk download into /var/www/pepptalk.
Note you might have to be a super user to add this folder to pepptalk so find where the database wrapper is and run
sudo mv -r DatabaseWrapper /var/www/pepptalk
sudo chmod 777 DatabaseWrapper
Now that the package is in place, we just need to create the WSGI file that will interact with Mod_WSGI

Creating WSGI File

Create the file app.wsgi in the pepptalk folder. Add the following:

import sys 
sys.path.insert(0,"/var/www/pepptalk/DatabaseWrapper")
from pepp_wsgi import application

sudo chmod 777 app.wsgi
sudo service apache2 restart
This restarts your server (which we want, since we have edited some config files).
To test that the your listening server is running correctly run the following code snippet:
```
curl -H "Content-Type: application/json" -X POST -d '{"name":"weather"}' localhost:43508/PeppTalk/Seeds/
```
If no errors are returned from the listening server, then the listening server has been setup correctly.

Congratulations, your listening server is now configured!

You may start your Apache server at anytime using

sudo service apache2 start

and stop your Apache server using

sudo service apache2 stop

Running Pepp Talk

In order to run Pepp Talk, you must have information inside the Database, populated using Seeds, and a User Template file. For more information on creating Seeds, see the Seeds section. For more information on User Templates, see the User Templates section. Once you have data in the database, locate the .template file you would like to use to generate your Pepp Talk, and pass its path into the report_generator.sh Shell Script. This path can be a local path or an absolute path. An example of passing in a template to the script is:

./report_generator.sh Template/Weather.template

VLC will open and play your Pepp Talk .wav file after your Pepp Talk is generated (and then VLC will close after the .wav file finishes playing), and if you have a default audio output device connected you should be hearing your Pepp Talk as well. If a .wav file does not play, an error may have occurred, but check that you have a default audio output device connected.

After running Pepp Talk, the .txt and .wav outputs will be in the same location as the User Template.

Admin Mode: Accessing the Database Directly

Inside the DatabaseWrapper folder is the wrapper.py script. Inside this script is a bunch of methods to access the database directly. In order to use these methods, import this class in Pythonuse the methods in wrapper.py to do your bidding to the Database. See wrapper.py for details.

NOTE: Do not attempt to directly modify the Database while the Apache server is running. It is also important to ensure that the user running python has permissions to edit the database file. It defaults to being owned by the Apache User.

Seeds

Seeds are formatting scripts written to gather data from various sources to be fed into a Pepp Talk. These Seeds can be written in any language and communicate with the database through the REST API.

The Seeds Folder

The Seeds folder contains an example folder with scripts that pull from various data sources like weather, finances, IoT devices, and news. These scripts gather the data, format it, and send it to the database. These serve as good examples of how to create a Seed for the Pepp Talk system in Python.

Picking a Programming Language

Pepp Talk is designed to have Seeds written in various different languages. We recommend using Python, as our example scripts are written in Python.

Understanding the Database Processing API "REST"

Add Data from the Seed with Name $Name$ to the database

If $Name$ does not exist, this method will fail with a 303 response telling the user to create the method first.

We are adding data $D1$ to Seed $Name$ , which was retrieved at time $T1$ . The data, $D1$ is a nested JSON document that the user must know the structure of to create the report generator.

Post /PeppTalk/Seeds/$Name$ returns:

{
    "data": $D1$,
    "name": $Name1$,
    "timestamp": $T1$
}

The database will return the JSON object passed in.

Retrieve a List of Seeds from the Database

If $Name$ , $Name2$ , and $Name3$ are stored in the database, the below will be returned from the call below. No data is expected on the call. If the database is empty, a 204 (No Content) will be returned.

Get /PeppTalk/Seeds returns:

{
    "name": [$Name$, $Name2$, $Name3$]
}

Retrieve the data stored by a Seed

If $D1$ is data associated with Seed $Name1$ , the below will be returned for the call, with $T2$ as the time it was entered into the database. Names is a Seed. A request for a non-existent Seed will respond with a 400 (Bad Request) response code.

Get /PeppTalk/Seeds/$Name1 returns:

{
    "data": $D1$,
    "name": $Name1$,
    "timestamp": $T1$,
    "logtimestamp": $T2$
}

Accessing previous data entries

If you wish to access Seed entries in the database other than the most recent, append to the url that retrieves the data (e.g. /PeppTalk/Seeds/$Name1) a forward slash and a number corresponding to how far back in the database you would like to go (i.e. /0 is most recent, /1 is one back, /4 is four back.).

Here is a full example, getting an entry 2 back (the 3rd one) from Seed Name1:

/PeppTalk/Seeds/$Name1/2

Writing a Web Scraping Seed

In order to gather data from the web, you can use whatever method you want. Some may choose to use web-scraping tools such as beautiful soup, or to rely on a specific website's APIs. Regardless of which method is chosen, the data gathered must be formatted properly before being added to the Pepp Talk database. All items in the Pepp Talk database should be JSON objects. For example, if the data retrieved by a seed named $Name$ has fields $F1$ and $F2$ with values of $V1$ and $V2$ , respectively, then the resulting JSON object should look like:

{
    "data": {$F1$:$V1, $F2$:$V2},
    "name": $Name$,
    "timestamp": $T1$
}

where $T1$ is the time at which the data is being entered into the database.

Writing a openHab Seed

OpenHab also uses a REST API to query data about the states of connected devices. The API is hosted at http://localhost:8080/rest. If you would like to query the state of a specific item name $item1$ , then you simply make a GET request to the url http://localhost:8080/rest/items/$item1$. Since the data placed into the database must be in JSON format, it may be helpful to retrieve the data from openHab in JSON format as well. To do this, simply append ?type=json to the end of the request url. actual state of the item can be retrieved driectly out of this JSON response by accessing the JSON object at the key 'state' The state of the item can alternatively appending 'state' to the end of the request url like so: http://localhost:8080/rest/items/$item1$/state.

Using Python to Write Seeds

In order to make HTTP requests in Python, it is necessary to import the requests package. This package should be installed as a part of the Pepp Talk installation script. If for any reason it is not installed, simply call pip install requests. In order to make a 'GET' request on a specific url, simply call requests.get() on that url. Similarly, you can use requests.post() to post your JSON-formatted data to the database. It is also useful to import the json package to format JSON responses. For example, the following code snippet makes a get request against a url and then formats the response into a dict by calling .json() on it.

{
    url = 'http://localhost:8080/rest/items/lightBulb?type=json'
    response = requests.get(url).json()
}

In the above example. The state of the lightBulb item can be retrieved from the response by accessing the dict with the 'state' key, like so: state = response['state']

In order to send this data to the database, simply format it into a dictionary using the format defined above. json.dumps() will take in a dictionary and convert it into a string, and json.loads() will take in a string and convert it into a JSON object. When your data is properly formatted, make a 'POST' request against the database API url to push it to the databse.

{
    encoded = {"name":"openHab",
            "data":{"itemName":$item1$,
                    "itemState":state},
            "timestamp":$T1$}
    json_object = json.loads(json.dumps(encoded))

    requests.post("http://localhost:43508/PeppTalk/Seeds", json=json_object)

    requests.post("http://localhost:43508/PeppTalk/Seeds/openHab", json=json_object)
}

In the above example, we assume that the seed is named "openHab". The first call to requests.post() creates the Seed entry in the database under the name "openHab" if it hasn't already been created. Note that the Seed name in the database is created based on the entry of the "name" field in the JSON, so it is imperative that this entry is present in the JSON object that is being posted. The second call to requests.post() then adds the json object itself to the database under that seed. Again, note that the name of the seed is now appended to the end of the url, and is the same name that was used in the JSON object itself.

Making Repeated Calls to the Database (cron job)

Cron tasks can be scheduled with crontab -e. The crontab takes form

m h dom mon dow command

The first five columns of the crontab are for scheduling how often tasks are run. See the Linux man pages for cron and crontab for more details.

Be aware that the cron daemon runs as a different user, which could require different permissions.

Here are two example lines you can add to the bottom of the crontab in order to run the provided example weather Seed and report generator once a day. The first line will run the weather seed at 5:55 AM every morning. The second line will run the report generator for the provided weather template at 6:00 AM every day.


55 5 * * * python /path/to/PeppTalkProduct/Seeds/Examples/weather_seed.py
0 6 * * * /path/to/PeppTalkProduct/report_generator.sh /path/to/PeppTalkProduct/Template/Examples/Weather.template

Example Seed


import requests
import json
 
def main():
    #Request url comes from Yahoo Weather API Documentation
    response = requests.get("https://query.yahooapis.com/v1/public/yql?q=select%20item.condition%20from%20weather.forecast%20where%20woeid%20%3D%202383078&format=json").json()

    location = "College Park, MD"
    condition = response['query']['results']['channel']['item']['condition']
    current_temp = condition['temp']

    encoded = {}
    encoded['name'] = 'weather'
    data = {}
    data['location'] = location
    data['temp'] = current_temp
    encoded['data'] = data

    json_object = json.loads(json.dumps(encoded))

    requests.post("http://localhost:43508/PeppTalk/Seeds", json=json_object)

    requests.post("http://localhost:43508/PeppTalk/Seeds/weather", json=json_object)


if __name__ == '__main__':
    main()

User Templates

A User Template is a specially formatted text document that will be translated into a regular text document using the Genshi Templating Engine. It consists of two components: the specially formatted text that the user would like in a report, and a list of the Seeds that this report is pulling information from. User Templates are the instructions that create your very own Pepp Talk A User Template should be saved with the .template extension.

The Templates Folder

The Templates folder is a good place to keep all your User Templates. You do not need to place your User Templates inside this folder (even though we recommend it), as you will be manually passing in the path of a User Template in as a command line argument.

Inside the Templates folder is also a sub-folder called Examples. There you will find several example User Templates that create Pepp Talks using the example Seeds we provided inside the Seeds folder. See The Seeds Folder section for more details on these Seeds.

Formatting a Template for the Report Generator System

Scripts for our tool are written in a strict subset of the Genshi New Text Template language. The first line is expected to be a Genshi comment, delimited by {# ... #} that contains a list of seeds from which data will be retrieved for use in the template. Each variable in the template is expected to be either an identifier contained in that list, or a retrieval from a dictionary with an identifier contained in that list, with a key corresponding to an entry in the database or a dictionary or key nested within that.

For instance, if the first line is

{# ["weather","finance","openHab"] #}

And the Weather Seed has the fields "location" and "forecast", and "forecast" is a nested structure with the field "hi" and "low", valid variable identifiers would include (non-exhaustively) weather, weather["location"], weather["forecast"], and weather["forecast"]["hi"]. It would not include weather["location"]["hi"]. Errors in variable identifiers will be reported at run-time.

Scripts whose first line lack a list of Seeds are equivalent to scripts whose first line contains an empty list, and will only be valid if they contain no variables.

When you design your own Seeds, you will know their structure and the fields you can call.

How to Use One or Several Completed Templates

In order to use a completed template, you must have information in the database, most likely populated by one or more Seeds. You can use a User Template by passing it to the report_generator.sh Shell Script via command line argument. He is one such example.

./report_generator.sh Template/Weather.template

Example Template

The following is an example User Template for the example weather Seed included with Pepp Talk:

{#["weather"]#}

Hi Kevin,

Currently, the temperature is ${weather['temp']} degrees.

Today's forecast is ${weather['condition']} with a high of ${weather['high']} degrees and low of ${weather['low']} degrees.

{% for forecast in weather['forecast'] %}

${forecast['day']}'s forecast is ${forecast['condition']} with a high of ${forecast['high']} degrees and low of ${forecast['low']} degrees.

{% end %}

Troubleshooting

Hearing the Audio Out

If you are having trouble hearing your Pepp Talk, check that your audio device is connected to your Linux environment, and that the audio device is the Default Audio Output Device.

`sudo rpi-update` Fails with "command not found"

If the Linux environment is not running on a Raspberry Pi, sudo rpi-update will fail. This is OK as long as you are not running the Linux environment on a Raspberry Pi. If sudo rpi-update fails and you are running on a Raspberry Pi, please refer to your Raspberry Pi's official documentation on this issue.

Pepp Talk cannot find my `.template` file!

Make sure you have set the absolute path of the PeppTalkProduct folder! Go to the PeppTalkProduct folder and open report_generator.sh with your preferred text editor. Change the ABS variable to be the absolute path to this folder. If you used "$(pwd)" as your absolute path, you may not be able to run cron jobs. If you wish to use cron jobs, please write out the path manually.