Showing posts with label GAE. Show all posts
Showing posts with label GAE. Show all posts

Sep 7, 2013

AJAX with JQuery and Google App Engine: Python

This is an adaption of AJAX with JQuery and PHP. It is a continuation of my Python 2.7 and Google App Engine series, and portions of the code builds upon my earlier work.

A first example...

AJAX stands for Asynchronous JavaScript And XML, and allows us to make our websites seem more dynamic. Let's start with the JavaScript side of things...

Create a file called static.html and add the following:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
       <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
       <title>Ajax With Jquery</title>

       <!-- Grab the JQuery API from the Google servers -->
       <script src="http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js" type="text/javascript" charset="utf-8"></script>

       <!-- Our own JQuery code to do the AJAX-y stuff -->
       <script type="text/javascript" charset="utf-8">

          $(document).ready(function(){
             $('#txtValue').keyup(function(){
                 $('#display').html($(this).val());
             });
          });

       </script>
    </head>

    <body>
       <label for="txtValue">Enter a value : </label>
       <input type="text" name="txtValue" value="" id="txtValue">

        <div id="display"></div>
    </body>
</html>

Create a file called app.yaml and add the following:
application: almightynassar
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /(.*).html
  static_files: \1.html
  upload: (.*).html

The above will simply serve ALL html page requests directly (which is enough for this example).

Now, if you are using Eclipse just right click on the project, select 'Run As' and choose the 'PyDev: Google App Run' option. Depending on how you configured your Google SDK, it will start up a server on your machine on port 8080. Open up a new tab and navigate to http://localhost:8080/static.html. You should now see a text box; when you type into the box it will automatically display the text in the space below.

Something a little more AJAX - The HTML

So our original example just showed how we could manipulate the DOM (Document Object Model) of the browser during run-time, without a page refresh. How can we use this knowledge utilizing the power of Google App Engine? Well, first let us edit static.html to something we can manipulate:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
    <head>
       <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
       <title>Ajax With Jquery</title>
     
       <!-- Grab the JQuery API from the Google servers -->
       <script src="http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js" type="text/javascript" charset="utf-8"></script>

       <!-- Our own JQuery code to do the AJAX-y stuff -->
       <script type="text/javascript" charset="utf-8">
       $(document).ready(function(){
            // This will run when the item of class 'button' is clicked
            $(".button").click(function() {
               
                // Grabs the text input
                var name = $("input#txtValue").val();
        
                var dataString = 'txtValue='+ name;  
                // This creates the AJAX connection
                $.ajax({
                    type: "POST",
                    url: "/tutorial",
                    data: dataString,
                    success: function(data) {
                        $('#display').html(data.text);
                    }
                });
            return false;
            });
        });
       </script>
    </head>

    <body>
         <form name="contact" method="post" action="">
            <label for="txtValue">Enter a value : </label>
            <input type="text" name="txtValue" value="" id="txtValue">
            <input type="submit" name="submit" class="button" id="submit_btn" value="Send" />
         </form>
         <div id="display">
         </div>
   </body>
</html> 

Our JavaScript now sends some data to an external URL (in this case /tutorial) and inserts the returned data into the display div. Now all we need to do is code our Python script....

Our Python webapp

Create a file called tutorial.py and add the following code:

# The webapp2 framework
import webapp2

# The JSON library
import json

# Our JSON handler class
class JsonPage(webapp2.RequestHandler):
    # The POST handler
    def post(self):
        # Our POST Input
        txtinput = self.request.get('txtValue')
       
        # Create an array
        array = {'text': txtinput}
       
        # Output the JSON
        self.response.headers['Content-Type'] = 'application/json'
        self.response.out.write(json.dumps(array))

# URL map the JSON function
app = webapp2.WSGIApplication([('/tutorial', JsonPage)], debug=True)

Now we just need to edit app.yaml so it can serve up our tutorial script:

application: almightynassar
version: 1
runtime: python27
api_version: 1
threadsafe: no

handlers:
- url: /(.*).html
  static_files: \1.html
  upload: (.*).html

- url: /(.*)
  script: tutorial.app

Open up a new tab and navigate to http://localhost:8080/static.html. You should now see a text box with a button called 'Send'. Typing into this box and then clicking 'Send' will transmit the text to our tutorial script and then display the text in the space below.

References:

Jun 26, 2012

Handling HTTP GET requests with webapp2 and Google App Engine: Python

This is a continuation of my Python 2.7 and Google App Engine series. This particular blog post builds upon the code given in my previous posts URL Routing and  Cron and Datastore in Google App Engine: Python, which in turn builds upon my earlier work. If you don't understand parts of the code I highly suggest you browse my earlier blog posts so you can understand some of the design decisions I have made.

A brief overview...

For those who are diving straight in, let me explain the old code and how I will update it:

I have a script feed.py that I have mapped using app.yaml. A cron script (configured by cron.yaml) simply connects to my Twitter account and converts my status updates into an RSS feed. It then stores the RSS feed into a Google Datastore object.

The feed script takes the Datastore object and displays it. We use another script (entity.py) to define the Datastore object.

We will now configure the system so that it can convert multiple twitter accounts into an RSS feed. To display a particular RSS feed we will use a HTTP GET request.

The main application

We will create a file called feed.py. This script will be our controller; it simply gets the HTTP requests and maps them to certain classes. These classes will then call other functions to perform the required tasks.

# The webapp2 framework
import webapp2

# Our datastore interface
from google.appengine.ext import db

# Our entity library
import entity

# Our XML2RSS library
import XML2RSS

# Output the XML in a HTML friendly manner
class Cron(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
        # A try-catch statement
        try:
            XML2RSS.getTweets("almightyolive")
            XML2RSS.getTweets("founding")
            XML2RSS.getTweets("ABCNews24")
            XML2RSS.getTweets("SBSNews")
       
        # Our exception code
        except (TypeError, ValueError):
            self.response.out.write("<html><body><p>Invalid inputs</p></body></html>")

# Fetches an XML document and parses it
class MainPage(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
        # A try-catch statement
        try:
            account = self.request.get('account')
           
            feed = entity.Rss()
            feed_k = db.Key.from_path('Rss', account)
            feed = db.get(feed_k)
           
            # Outputs the RSS
            self.response.out.write(feed.content)

        # Our exception code
        except (TypeError,ValueError):
            self.response.out.write("<html><body><p>Invalid inputs (Type Error)</p></body></html>")
        except:
            self.response.out.write("<html><body><p>Unspecified Error</p></body></html>")

# Create our application instance that maps the root to our
# MainPage handler
app = webapp2.WSGIApplication([('/', MainPage),('/cron', Cron)], debug=True)

The XML2RSS script

As you may have noticed,the feed.py script made reference to an XML2RSS object. This is a separate script that outsources the conversion of XML to RSS into easy-to-call functions. Create a new file called XML2RSS.py and add the following:

# The minidom library for XML parsing
from xml.dom.minidom import parseString

# The URL Fetch library
from google.appengine.api import urlfetch

# Our entity library
import entity

# Detects if it is a URL link and adds the HTML tags
def linkify(text):
    # If http is present in, add the link tag
    if "http" in text:
        text = "&lt;a href='" + text + "'&gt;" + text + "&lt;/a&gt;"
    elif "@" in text:
        text = "&lt;a href='http://twitter.com/#!/" + text.split("@")[1] + "'&gt;" + text + "&lt;/a&gt;"
    elif "#" in text:
        text = "&lt;a href='https://twitter.com/#!/search/%23" + text.split("#")[1] + "'&gt;" + text + "&lt;/a&gt;"
       
    return text

# Output the XML in a HTML friendly manner
def outputRSS(xml, account):
    # The get the states list
    statuses = xml.getElementsByTagName("status")
   
    # Our return string
    outputString = "<?xml version='1.0'?>\n<rss version='2.0'>\n\t<channel>\n\t\t<title>Twitter: " + account + "</title>\n\t\t"
    outputString+= "<link>https://twitter.com/#!/almightyolive</link>\n\t\t<description>The twitter feed for " + account + "</description>"
   
    # Cycled through the states
    for status in statuses:
        #Gets the statuses
        text = status.getElementsByTagName("text")[0].firstChild.data
        date = status.getElementsByTagName("created_at")[0].firstChild.data
        tweet = status.getElementsByTagName("id")[0].firstChild.data
       
        # Insert links into the text
        words = text.split()
       
        for i in range (len(words)):
            words[i] = linkify(words[i])
       
        # Recompile words
        text = " ".join(words)
       
        # Creates our output
        string = "\n\t\t<item>\n\t\t\t<title>" + str(date) + "</title>\n\t\t\t<link>https://twitter.com/AlmightyOlive/status/" + tweet + "</link>\n\t\t\t<description>" + str(text) + "</description>\n\t\t</item>"
        outputString+=string
       
    # Output string
    outputString += "\n\t</channel>\n</rss>"
    return outputString   

# Our RSS storage function
def getTweets(account):
    # Grabs the XML
    url = urlfetch.fetch('https://api.twitter.com/1/statuses/user_timeline.xml?screen_name=' + account + '&count=10&trim_user=true')
           
    # Parses the document
    xml = parseString(url.content)

    # Converts the XML into RSS
    content = outputRSS(xml, account)
   
    # Our RSS storage entity
    rssStore = entity.Rss(key_name='' + account)

    # Elements of our RSS
    rssStore.feed = '' + account
    rssStore.content = content

    # Stores our RSS Feed into the datastore
    rssStore.put()

The pieces to make it all work

If you have been following on from my previous work, then you should already have most of this code. I won't bother explaining it here because it is mostly self-explanatory.

app.yaml:
application: almightynassar
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /cron
  script: feed.app
  login: admin
 
- url: /.*
  script: feed.app

cron.yaml:


cron:
- description: daily summary job
  url: /cron
  schedule: every 1 hours

entity.py:

# Our datastore interface
from google.appengine.ext import db

# Our RSS entity object
class Rss(db.Model):
    feed = db.StringProperty()
    content = db.TextProperty()

And that's it! You now have a fully functional application that just uses the webapp2 framework!

If you navigate to http://localhost:8080/?account=almightyolive you should now see the RSS feed. You can test if your mapping works by navigating to http://localhost:8080/?account=founding; you should see the Founding Institute twitter account instead!


References:

Jun 25, 2012

URL Routing through WebApp2 in Google App Engine: Python

This is a continuation of my Python 2.7 and Google App Engine series. This particular blog post builds upon the code given in my previous post Cron and Datastore in Google App Engine: Python, which in turns builds upon my earlier work. If you don't understand parts of the code I highly suggest you browse my earlier blog posts so you can understand some of the design decisions I have made.

A brief overview...

For those who are diving straight in, let me explain the old code and how I will update it:

I have two scripts (feed.py and cron.py) that I have mapped using app.yaml. The cron script simply connects to my Twitter account and converts my status updates into an RSS feed. It then stores the RSS feed into a Google Datastore object.

The feed script takes the Datastore object and displays it. Both scripts use a third script (entity.py) to define the Datastore object.

Currently the set-up is not thread-safe because I have to use two different scripts to handle my incoming requests. The plan is to replace this set-up with one that is thread-safe. Effectively, we will be using the URL routing functionality provided by the webapp2 framework.

Combining the scripts

The first thing we will do is combine both cron.py and feed.py into one script. The following code should be saved to a file called feed.py:

# The webapp2 framework
import webapp2

# Our datastore interface
from google.appengine.ext import db

# The minidom library for XML parsing
from xml.dom.minidom import parseString

# The URL Fetch library
from google.appengine.api import urlfetch

# Our entity library
import entity

# Detects if it is a URL link and adds the HTML tags
def linkify(text):
    # If http is present in, add the link tag
    if "http" in text:
        text = "&lt;a href='" + text + "'&gt;" + text + "&lt;/a&gt;"
    elif "@" in text:
        text = "&lt;a href='http://twitter.com/#!/" + text.split("@")[1] + "'&gt;" + text
        text+= "&lt;/a&gt;"
    elif "#" in text:
        text = "&lt;a href='https://twitter.com/#!/search/%23" + text.split("#")[1] + "'&gt;" + text + "&lt;/a&gt;"
       
    return text

# Output the XML in a HTML friendly manner
def outputRSS(xml):
    # The get the states list
    statuses = xml.getElementsByTagName("status")
   
    # Our return string
    outputString = "<?xml version='1.0'?>\n<rss version='2.0'>\n\t<channel>"
    outputString+= "\n\t\t<title>Almightyolive Twitter</title>\n\t\t"
    outputString+= "<link>https://twitter.com/#!/almightyolive</link>\n"
    outputString+= "\t\t<description>The twitter feed for the Almighty "
    outputString+= "Olive</description>"
   
    # Cycled through the states
    for status in statuses:
        #Gets the statuses
        text = status.getElementsByTagName("text")[0].firstChild.data
        date = status.getElementsByTagName("created_at")[0].firstChild.data
        tweet = status.getElementsByTagName("id")[0].firstChild.data
       
        # Insert links into the text
        words = text.split()
       
        for i in range (len(words)):
            words[i] = linkify(words[i])
       
        # Recompile words
        text = " ".join(words)
       
        # Creates our output
        string = "\n\t\t<item>\n\t\t\t<title>" + str(date) + "</title>\n"
        string+= "\t\t\t<link>https://twitter.com/AlmightyOlive/status/" + tweet
        string+= "</link>\n\t\t\t<description>" + str(text) + "</description>\n"
        string+= "\t\t</item>"
        outputString+=string
       
    # Output string
    outputString += "\n\t</channel>\n</rss>"
    return outputString   

# Output the XML in a HTML friendly manner
class Cron(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
        # A try-catch statement
        try:
            # Grabs the XML
            url = urlfetch.fetch('https://api.twitter.com/1/statuses/user_timeline.xml?screen_name=almightyolive&count=10&trim_user=true')
           
            # Parses the document
            xml = parseString(url.content)

            content = outputRSS(xml)
            # Our RSS storage entity
            rssStore = entity.Rss(key_name='almightyolive')
           
            # Elements of our RSS
            rssStore.feed = "almightyolive"
            rssStore.content = content

            # Stores our RSS Feed into the datastore
            rssStore.put()
       
        # Our exception code
        except (TypeError, ValueError):
            self.response.out.write("<html><body><p>Invalid inputs</p></body></html>")

# Fetches an XML document and parses it
class MainPage(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
        # A try-catch statement
        try:
            feed = entity.Rss()
            feed_k = db.Key.from_path('Rss', 'almightyolive')
            feed = db.get(feed_k)
           
            # Outputs the RSS
            self.response.out.write(feed.content)

        # Our exception code
        except (TypeError, ValueError):
            self.response.out.write("<html><body><p>Invalid inputs</p></body></html>")

# Create our application instance that maps the root to our
# MainPage handler
app = webapp2.WSGIApplication([('/', MainPage),('/cron', Cron)], debug=True)

The big changes are:
  • We have added a new class called Cron, which included all of that loose code in cron.py
  • We have added a new URL mapping to our WSGI Application. This will hand over any request for '/cron' to our new Cron class

The pieces to make it all work

If you have been following on from my previous work, then you should already have most of this code. The only thing you need to touch is one line in app.yaml, which is to map /cron to our feed webapp.

app.yaml:
application: almightynassar
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /cron
  script: feed.app
  login: admin
 
- url: /.*
  script: feed.app

cron.yaml:


cron:
- description: daily summary job
  url: /cron
  schedule: every 1 hours

entity.py:

# Our datastore interface
from google.appengine.ext import db

# Our RSS entity object
class Rss(db.Model):
    feed = db.StringProperty()
    content = db.TextProperty()

And that's it! You now have a fully functional application that just uses the webapp2 framework!

References:

Jun 15, 2012

Cron and Datastore in Google App Engine: Python

This is a continuation of my Python 2.7 and Google App Engine series. If you are just starting out I suggest you start reading Getting Started and First App. If you are after parsing XML or HTML files please see my posts 'Parsing XML with Google App Engine: Python' or 'Parsing HTML with lxml and Google App Engine: Python'.

This example will use the code from my previous post 'Convert Twitter stream into an RSS feed'. If you don't understand why I did something, just pop on over to that link and see how I came up with the code originally.

I STRONGLY suggest you look up some of the references such as how Google App Engine handles cron, datastore, and some things about entities and keys. Most of the stuff you will need are provided in the references at the end of the post.

What our plan is...

Ok, let me briefly go over what my proposed system will do and how cron and GAE's Datastore fits in.

In a previous blog post I created a web app that would connect to someones twitter feed and convert it into an RSS feed. A problem with this set-up was that there was a massive lag (about 2-3 seconds) while the app downloaded the stream, parsed it, inserted links and outputted an RSS XML file.

To solve this, I will create a cron script that I will run in the background (I will also hide it behind an administration login page so that random users cannot call it randomly). This requires me to store the feed into a persistent object, which Datastore conveniently supplies.

Now for the code....

The RSS object

We will first create a Python object that we will use to define the objects we store into our database. Just create a file called entity.py and add the following code:

# Our datastore interface
from google.appengine.ext import db

# Our RSS entity object
class Rss(db.Model):
    feed = db.StringProperty()
    content = db.TextProperty()

Note that we import the db object; this is our interface to Datastore. If you want to know more about creating entities, I suggest you read the references provided at the end of the post.

The cron script

This script will do everything we did in our previous blog post, except it will store the feed into our RSS entity and into the Datastore. Create a file called cron.py and insert the following code:

# The minidom library for XML parsing
from xml.dom.minidom import parseString

# The URL Fetch library
from google.appengine.api import urlfetch

# Our entity library
import entity

# Detects if it is a URL link and adds the HTML tags
def linkify(text):
    # If http is present in, add the link tag
    if "http" in text:
        text = "&lt;a href='" + text + "'&gt;" + text + "&lt;/a&gt;"
    # If @ is present, turn it into a twitter handle link

    elif "@" in text:
        text = "&lt;a href='http://twitter.com/#!/" + text.split("@")[1] + "'&gt;" + text
        text+= "&lt;/a&gt;"
    # Turn into twitter hash tags

    elif "#" in text:
        text = "&lt;a href='https://twitter.com/#!/search/%23" + text.split("@")[1] + "'&gt;" + text
        text+= "&lt;/a&gt;"

       
    return text

# Output the XML into an RSS feed
def outputRSS(xml):
    # The get the status list
    statuses = xml.getElementsByTagName("status")
   
    # Our return string
    outputString = "<?xml version='1.0'?>\n<rss version='2.0'>\n\t<channel>"
    outputString+= "\n\t\t<title>Almightyolive Twitter</title>\n\t\t"

    outputString+= "<link>https://twitter.com/#!/almightyolive</link>\n"
    outputString+= "\t\t<description>The twitter feed for the Almighty "
    outputString+= "Olive</description>"

   
    # Cycled through the status
    for status in statuses:
        #Gets the statuses
        text = status.getElementsByTagName("text")[0].firstChild.data
        date = status.getElementsByTagName("created_at")[0].firstChild.data
        tweet = status.getElementsByTagName("id")[0].firstChild.data
       
        # Insert links into the text
        words = text.split()
       
        for i in range (len(words)):
            words[i] = linkify(words[i])
       
        # Recompile words
        text = " ".join(words)
       
        # Creates our output
        string = "\n\t\t<item>\n\t\t\t<title>" + str(date) + "</title>\n"
        string+="\t\t\t<link>https://twitter.com/AlmightyOlive/status/" + tweet
        string+= "</link>\n\t\t\t<description>" + str(text) + "</description>\n"
        string+= "\t\t</item>"

        outputString+=string
       
    # Output string
    outputString += "\n\t</channel>\n</rss>"
    return outputString   

# OUR CRON SCRIPT PROPER!
#
# Grabs the XML

url = urlfetch.fetch('https://api.twitter.com/1/statuses/user_timeline.xml?screen_name=almightyolive&count=10&trim_user=true')
           
# Parses the document
xml = parseString(url.content)

content = outputRSS(xml)
# Our RSS storage entity
rssStore = entity.Rss(key_name='almightyolive')

# Elements of our RSS
rssStore.feed = "almightyolive"
rssStore.content = content

# Stores our RSS Feed into the datastore
rssStore.put()

The functions linkify() and outputRSSS() are exactly the same as in the previous blog post (with the addition to linkify to do hashtags). Our biggest difference is the replacing MainPage and the webapp specific stuff with a simple sequential script (which in actuality is not unlike the content of MainPage).

A brief explanation of the entity and datastore code:
  1. Create rssStore object as defined by the Rss object in our entity.py file. Note that we pass a key called 'almightyolive', which is our unique identifier for this object.
  2. Store our object values, especially our feed variable content
  3. Call the put() method on our rssStore object to push it onto the Datastore
And thats it!

The feed app

Now we need to create our front-end to access the RSS feed xml. Create a new file called feed.py and add the following:

# The webapp2 framework
import webapp2

# Our datastore interface
from google.appengine.ext import db

import entity

# Fetches an datastore object and displays it
class MainPage(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
        # A try-catch statement
        try:
            # Create RSS entity

            feed = entity.Rss()
            # Get the key for an RSS entity called almightyolive

            feed_k = db.Key.from_path('Rss', 'almightyolive')
            # Retrieve object from datastore

            feed = db.get(feed_k)
           
            # Outputs the RSS
            self.response.out.write(feed.content)

        # Our exception code
        except (TypeError, ValueError):
            self.response.out.write("<html><body><p>Invalid inputs</p></body></html>")

# Create our application instance that maps the root to our
# MainPage handler
app = webapp2.WSGIApplication([('/*', MainPage)], debug=True)

Pretty simple, huh? Now onto the configuration files....

app.yaml and cron.yaml

Let's start with app.yaml first. Add the following:

application: almightynassar
version: 1
runtime: python27
api_version: 1
threadsafe: no

handlers:
- url: /cron
  script: cron.py
  login: admin
 
- url: /.*
  script: feed.app
Note that this is no longer threadsafe; this is because we defined another handler other than feed.app. Why? Because I added a line for our cron handler: 'login: admin'. This restricts access to the URL to only the administrators of the application.

Now create cron.yaml with the following:

cron:
- description: daily summary job
  url: /cron
  schedule: every 1 hours

And there you have it! Once you upload it, the cron.py script will run every hour (you can run it manually first to populate the database) and then see the RSS feed!

References

Jun 10, 2012

Convert Twitter into RSS Feed with Google App Engine: Python

This is a continuation of my Python 2.7 and Google App Engine series. If you are just starting out I suggest you start reading Getting Started and First App. If you are after parsing XML or HTML files please see my posts 'Parsing XML with Google App Engine: Python' or 'Parsing HTML with lxml and Google App Engine: Python'.


Simple RSS syndication

We are going to assume you have already created a project ( Hint: You just need an app.yaml configuration file and a main.py file). If you don't know, please refer to one of my earlier blog posts (above) or the references (below).

In your main.py file add the following:

# The webapp2 framework
import webapp2

# The minidom library for XML parsing
from xml.dom.minidom import parseString

# The URL Fetch library
from google.appengine.api import urlfetch

# Fetches an XML document and parses it
class MainPage(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
        # A try-catch statement
        try:
            # Grabs the XML
            url = urlfetch.fetch('https://api.twitter.com/1/statuses/user_timeline.xml?screen_name=almightyolive&count=10&trim_user=true')
           
            # Parses the document
            xml = parseString(url.content)
           
            # Outputs the RSS
            self.response.out.write(outputRSS(xml))
           
            # Sets up the webpage
            self.response.out.write("</table></body></html>")

        # Our exception code
        except (TypeError, ValueError):
            self.response.out.write("<html><body><p>Invalid inputs</p></body></html>")

# Output the XML in a HTML friendly manner
def outputRSS(xml):
    # The get the states list
    statuses = xml.getElementsByTagName("status")
   
    # Set up our XML return
    outputString = "<?xml version='1.0'?>\n<rss version='2.0'>\n\t<channel>"
    outputString+= "\n\t\t<title>Almightyolive Twitter</title>\n\t\t"

    outputString+= "<link>https://twitter.com/#!/almightyolive</link>\n"
    outputString+= "\t\t<description>The twitter feed</description>"

   
    # Cycled through the statuses
    for status in statuses:
        #Gets the Text and date and cycles through them
        text = status.getElementsByTagName("text")[0].firstChild.data
        date = status.getElementsByTagName("created_at")[0].firstChild.data
        string = "\n\t\t<item>\n\t\t\t<title>" + str(date) + "</title>\n"
        string+= "\t\t\t<link>https://twitter.com/#!/almightyolive</link>\n\t\t"
        string+= "\t<description>" + str(text) + "</description>\n\t\t</item>"

        outputString+=string
       
    # Output string
    outputString += "\n\t</channel>\n</rss>"
    return outputString   

# Create our application instance that maps the root to our
# MainPage handler
app = webapp2.WSGIApplication([('/*', MainPage)], debug=True)

This is a very simple feed; you won't get any nice links and clicking on a particular item will just take you to the main feed. Now to tweak it just a little....


RSS with links

Now we will add a function that will add appropriate links to our tweets. Note that this is a really, really dumb function: it will apply to ANY instances of 'http' or '@' in a word, so it will accidentally affect emails or tweets about the HTTP protocol. I leave it up to you to fix the code if you don't want these things to happen.

Anyway, replace main.py with the following code:

# The webapp2 framework
import webapp2

# The minidom library for XML parsing
from xml.dom.minidom import parseString

# The URL Fetch library
from google.appengine.api import urlfetch

# Fetches an XML document and parses it
class MainPage(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
        # A try-catch statement
        try:
            # Grabs the XML
            url = urlfetch.fetch('https://api.twitter.com/1/statuses/user_timeline.xml?screen_name=almightyolive&count=10&trim_user=true')
           
            # Parses the document
            xml = parseString(url.content)
           
            # Outputs the RSS
            self.response.out.write(outputRSS(xml))

        # Our exception code
        except (TypeError, ValueError):
            self.response.out.write("<html><body><p>Invalid inputs</p></body></html>")

# Output the XML in a HTML friendly manner
def outputRSS(xml):
    # The get the states list
    statuses = xml.getElementsByTagName("status")
   
    # Our return string
    outputString = "<?xml version='1.0'?>\n<rss version='2.0'>\n\t<channel>"
    outputString+= "\n\t\t<title>Almightyolive Twitter</title>\n\t\t"

    outputString+= "<link>https://twitter.com/#!/almightyolive</link>\n"
    outputString+= "\t\t<description>The twitter feed for the Almighty "
    outputString+= "Olive</description>"

   
    # Cycled through the states
    for status in statuses:
        #Gets the statuses
        text = status.getElementsByTagName("text")[0].firstChild.data
        date = status.getElementsByTagName("created_at")[0].firstChild.data
        tweet = status.getElementsByTagName("id")[0].firstChild.data
       
        # Insert links into the text
        words = text.split()
       
        for i in range (len(words)):
            words[i] = linkify(words[i])
       
        # Recompile words
        text = " ".join(words)
       
        # Creates our output
        string = "\n\t\t<item>\n\t\t\t<title>" + str(date) + "</title>\n"
        string+= "\t\t\t<link>https://twitter.com/AlmightyOlive/status/" + tweet
        string+= "</link>\n\t\t\t<description>" + str(text) + "</description>\n"
        string+= "\t\t</item>"

        outputString+=string
       
    # Output string
    outputString += "\n\t</channel>\n</rss>"
    return outputString   

# Detects if it is a URL link and adds the HTML tags
def linkify(text):
    # If http is present in, add the link tag
    if "http" in text:
        text = "&lt;a href='" + text + "'&gt;" + text + "&lt;/a&gt;"
    elif "@" in text:
        text = "&lt;a href='http://twitter.com/#!/" + text.split("@")[1]
        text+= "'&gt;" + text + "&lt;/a&gt;"
  
    return text

# Create our application instance that maps the root to our
# MainPage handler
app = webapp2.WSGIApplication([('/*', MainPage)], debug=True)

And there we have it; the linkify() function will add links into our tweets to make them more usable and accessible!

References

Jun 9, 2012

Parsing HTML with lxml and Google App Engine: Python

This is a continuation of my Python 2.7 and Google App Engine series. If you are just starting out I suggest you start reading Getting Started and First App. If you are after parsing XML files please see my post 'Parsing XML with Google App Engine: Python'.

We are going to assume you will be using Eclipse and a fresh project. In this example we are going to use Triple J Unearthed's Top 100 charts HTML page to parse.

Adding lxml to Google App Engine

The first thing we need to do is add the lxml library to our app.yaml configuration file. In your Eclipse project add a new file called app.yaml and add the following:

application: almightynassar
version: 1
runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: triplej.app

libraries:
- name: lxml
  version: latest

Most of these fields were covered in Getting Started, but we now have a new field: libraries. This is where we declare all third party libraries not included in GAE default python environment.

Using lxml

Create a new file called triplej.py and add the following code:

# The webapp2 framework
import webapp2

# lxml parser for XML and HTML
from lxml import etree

# The URL Fetch library
from google.appengine.api import urlfetch

# Fetches an XML document and parses it
class MainPage(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
            # Grabs the HTML
            url = urlfetch.fetch('http://www.triplejunearthed.com/Charts/')
           
            # Parses the HTML
            tree   = etree.HTML(url.content)

            # Converts the DOM into a string       
            result = etree.tostring(tree, pretty_print=True, method="html")

           
           # Output the results onto the screen
           self.response.out.write(str(result))
   
       
# Create our application instance that maps the root to our
# MainPage handler
app = webapp2.WSGIApplication([('/*', MainPage)], debug=True)

If you run this code you will notice that all it does is simply download the HTML page, parses it, and then outputs the page exactly as it was downloaded (minus all the images and CSS styling). Nothing impressive, but we proved the concept works. Now on to something a little more beefy....

Parsing, Extracting and Cleaning the HTML

In this example we will perform multiple functions that will only extract the chart from the Triple J Unearthed website. Replace the triplej.py code with the following:

# The webapp2 framework
import webapp2

# lxml parser for XML and HTML
from lxml import html

# HTML cleaner
from lxml.html.clean import Cleaner

# The URL Fetch library
from google.appengine.api import urlfetch

# Fetches an XML document and parses it
class MainPage(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
        # Grabs the HTML
        url = 'http://www.triplejunearthed.com/Charts/'
        website = urlfetch.fetch(url)
       
        # Saves our content as a string
        page = str(website.content)

        # Parses the HTML
        tree = html.fromstring(page)

        # The ID string of the table element we want          # NOTE: This is bound to change!!! Double check the HTML source first!!!
        elementID = "ctl00_ctl00_ctl00_ctl00_MainBody_ContentPlaceHolder1_ContentPlaceHolder1_ContentPlaceHolder1_GridView1"
       
        # Grab the chart element
        #
        # style: removes styling
        # links: removes links
        # add_nofollow: adds rel="nofollow" to anchor tags
        # page_structure: removes <html>, <head>, and <title> tages
        # safe_attrs_only: only allows safe element attributes
        # javascript: removes embedded javascript
        # scripts: remove script tags
        # kill_tags: remove the element and content
        # remove_tags: remove only the element, but not the content
        #
        # There are more available. See the API reference for lxml
        cleaner = Cleaner(style=True, links=True, add_nofollow=True,
                          page_structure=True, safe_attrs_only=True,

                          javascript=True, scripts=True, kill_tags = set(['img','th']),
                          remove_tags = (['div']))

       
        # Grab only our chart (but scrub it clean first!)
        chart = cleaner.clean_html(tree.get_element_by_id(elementID))
       
        # Change all relative links into absolute links based on the url
        chart.make_links_absolute(url)
       
        # Converts the DOM element into a string
        result = html.tostring(chart)
       
        # Output the results onto the screen
        self.response.out.write(result)        
       
# Create our application instance that maps the root to our
# MainPage handler
app = webapp2.WSGIApplication([('/*', MainPage)], debug=True)

Running this code should result in a sanitized version of the Triple J Top 100 chart!

References