Jun 6, 2012

First App with Google App Engine: Python

This is a follow-on post from 'Getting Started with Google App Engine: Python'. Here we will explore using the default webapp framework that comes with the Google App Engine SDK (Note: You can replace this with another framework such as Django, but that is outside the scope of this post).

Note that we are using Python 2.7 and webapp2 for this tutorial

Quick overview

A basic application in the webapp2 framework will consist of three parts:
  • One or more handlers as defined by the RequestHandler class
  • A WSGIApplication instance that maps URLs to specific handlers
  • A configuration YAML file that tells Google App Engine to use Python 2.7
It's nothing fancy; it's supposed to be a very simple webapp framework to get you started. Now let's dive into some code....

The YAML configuration file

We will start with the configuration file because a) it's simple, and b) you shouldn't have to edit it ever again. Create a file called app.yaml and add the following:

application: almightyolive
version: 1
runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: main.app
An explanation of all the fields:
  • application is where you will store the appid you were given by the GAE (Google App Engine) dashboard
  • version is your code's version number. GAE offers basic version control, so updating this number will force GAE to back-up your old code an allow you revert at a later date
  • runtime the specific Python version you are using (at the time of writing Python 2.5 is also supported)
  • api_version tells GAE which API you are using. If Google ever updates their code, your application will still run on the API it was coded for
  • threadsafe is a Python 2.7 specific parameter that allows your app to handle concurrent requests
  • handlers is a list that maps specific URLs to your modules. In this case, everything will be routed to our main.app, which is what we are going to code next....

Hello World!

Create a new file called main.py and add in the following code:

# The webapp2 framework
import webapp2

#Our main handler class that just outputs "Hello Webapp World!"
class MainPage(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
        # Output to the browser that it will receive plan text
        self.response.headers['Content-Type'] = 'text/plain'
        # Output the Hello World message
        self.response.out.write('Hello, webapp World!')

# Create our application instance that maps the root to our
# MainPage handler
app = webapp2.WSGIApplication([('/', MainPage)], debug=True)

Run the python web server (right click on the project and select 'Run As..' - > 'PyDev: Google App Run') and see the results on your browser (default should be http://localhost:8080/)!

Something a bit more advanced....

Edit the main.py code as follows:

# The webapp2 framework
import webapp2

#Adds two numbers together that are passed through URL
class MainPage(webapp2.RequestHandler):
    # Respond to a HTTP GET request
    def get(self):
        # A try-catch statement
        try:
            # Grab the numbers from using GET
            first = int(self.request.get('first'))
            second = int(self.request.get('second'))

            # Outputs the addition of the numbers

            self.response.out.write("<html><body><p>%d + %d = %d</p></body></html>" % (first, second, first + second))

        # Our exception code
        except (TypeError, ValueError):
            self.response.out.write("<html><body><p>Invalid inputs</p></body></html>")

# Create our application instance that maps the root to our
# MainPage handler
app = webapp2.WSGIApplication([('/', MainPage)], debug=True)

Test it out by navigating to http://localhost:8080/?first=1&second=2

References

Jun 5, 2012

Getting starting with Google App Engine: Python

I have heard of the Google App Engine (a 'cloud' computing service) for a while now but never really had the opportunity to really use it. Luckily, I have a job on the horizon that seems like a perfect fit; a 'glue' app that connects two separate web apps together. I can simply host all the code onto Google's cloud, run a few CRON scripts to grab the data from one app and upload it to another.

But first, I need to learn how to tame this beast and get it to do my bidding....

What you need

First off, I am going to assume you have created a Google account and registered to use App Engine (you should be able to access the dashboard). You should also have Eclipse installed with the Google Plugin. I won't get into the details here since the reference links should provide you with adequate tutorials on how to set up each of the components.

Next, you should realize that this guide will only deals with Python for the App Engine environment. You can use Java and GO, but that is outside the scope of this guide. So if you don't know it yet, go ahead and brush up on some of your Python skills (and make sure it is installed on your machine AND Eclipse!).

Finally, go ahead and download the Google App Engine SDK for Python. Without this SDK you WILL NOT be able to create any projects!

OK, I'm ready. What's next?

  1. Fire up Eclipse and from the menu select 'File -> New-> Project'
  2. Expand the PyDev tree and select PyDev Google App Engine Project
  3. Give your project a name and check to make sure your settings are correct (for instance, are you using Python 2.5 or 2.7?)
  4. Did I mention that you need to download the Google App Engine SDK for Python? Download it, extract it, and locate the directory so that Eclipse can use it.
  5. Select the Python libraries you want to use (Defaults are good for our purposes)
  6. Now we need our details from Google such as our appid. Log onto the App Engine Dashboard, create an app and enter the appid into Eclipse. Note that for the rest of this tutorial I am going to assume you are using the Hello World template.
  7.  Right click on the project, select 'Run As' and choose the 'PyDev: Google App Run' option
  8. Depending on how you configured your Google SDK, it will start up a server on your machine on port 8080. Open up a new tab and navigate to http://localhost:8080/
  9. We should now see 'Hello, World!' on our screen. To upload the app to Google just right click the project and select the 'PyDev: Google App Engine' -> 'Upload' option
  10. A dialogue will open up and you may have to enter in some parameters such as you email address and password (note: for people who use 2-step verification, you WILL need to create an application specific password). 
  11. Wait until the dialogue displays FINISHED. Once it pops up you can now go to the URL displayed on the Google Apps Dashboard and see your app in action!

References

May 28, 2012

Setting up a CentOS 6.2 web server: Securing the file system

This is a follow on post from my guide to installing CentOS 6.2. We will go through some of the steps required to secure your server and get it ready for production use.

This section will outline how to lock down your partitions and file system. We will assume that you went for a file system structure similar to the one outlined in the above blog post.

Since the filesystem flags will be the most foriegn concept in this guide, I will give a quick outline about file system flags. However, I strongly suggest you follow the links provided in the references for more in-depth detail on any topic provided in this guide.
  • nosuid disallows the filesystem from granting a user the temporary elevated privileges of a file's owner or group.
  • noexec disallows the filesystem from running an executable.
  • nodev disallows the filesystem from running files as block devices (i.e. treat the file as an I/O source or sink).
The process for securing your file system is as follows:
  1. Secure your partitions by editing /etc/fstab as follows:
    • Add nosuid, noexec, and nodev to partitions like  /dev/shm, /var/log, /tmp, and /var/log/audit. Basically any partition where you only expect to read and write files.
    • Add nodev to all non-root file systems like /home and /var/www. You can add the noexec flag if you want, but note that cgi scripts stored in /var/www will break (as well as any scripts stored in the user's home directory).
    • Add nosuid only to file systems like /var.
    • DO NOT ADD ANY OF THESE FLAGS TO /!!!!!

     
  2.  Add the following line to /etc/fstab to hardlink /var/tmp to /tmp

    /tmp /var/tmp none rw,noexec,nosuid,nodev,bind 0 0
     
  3. Disable the autofs service if you do not need NFS (unless you have already un-installed the service):

    chkconfig autofs off 

References

May 25, 2012

Setting up a CentOS 6.2 web server: Software and package integrity and installation

This is a follow on post from my guide to installing CentOS 6.2. We will go through some of the steps required to secure your server and get it ready for production use.

These steps will outline how to check what is installed on your system and whether your system has been compromised.
  1.  Log in as root and grab a current software list (check out my previous blog post on this topic).

    #Using RPM
    rpm -qa
    #Using yum
    yum list installed

     
  2. Check to ensure that yum is forced to check the gpg signature when installing packages. This is default behaviour in CentOS 6.2, but for the sake of completeness I have included this step. Check /etc/yum.conf and all the files in /etc/yum.repos.d/ for the following line:

    gpgcheck=1

     
  3. AIDE is an intrusion dectection environment that checks the integrity of installed packages and files. It can report on the changes to your system. Install it using yum:

    yum install aide

     
  4. It should be your priority to read and understand /etc/aide.conf and tailor it to your system. While the defaults should be adequate for most installations, you should nevertheless know what AIDE does.
     
  5. Generate the initial AIDE database (by default it will be stored as /var/lib/aide/aide.db.new.gz):

    /usr/sbin/aide --init
     
  6. Back up the database (in this example we are copying it to root's home directory):

    cp /var/lib/aide/aide.db.new.gz ~/
     
  7. Install the AIDE database:

    mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
     
  8. Run a manual check:

    /usr/sbin/aide --check
     
  9. Stop the yum update daemon (we are going to write our own). If you followed the above guide the service won't even be installed, but again I am including it here for completeness.

    chkconfig yum-updatesd off
     
  10. Edit /etc/crontab by adding the following line (this will do a daily check of the system with AIDE):

    50 4 * * * root /usr/sbin/aide --check
     
  11. Add a file called update.cron in /etc/cron.weekly/ and add in the following (NOTE!: cron.weekly is run by the anacron service. Keep this in mind when you are disabling services later on in this guide).

    #!/bin/sh
    #
    # Update yum, then the rest of the system
    /usr/bin/yum -R 120 -e 0 -d 0 -y update yum
    /usr/bin/yum -R 10 -e -0 -d 0 -y update
    #
    # Save a list of software currently installed on the system
    /bin/rpm -qa > /root/`/bin/hostname -s`_software_`/bin/date +%Y%m%d`.txt
    #
    # OPTIONAL: Fix all the prelinks (otherwise you may get alot of prelink messages)
    /usr/sbin/prelink --all
    #
    # Update the AIDE database
    /usr/sbin/aide --update
    #
    # Make a back-up of the new database
    /bin/cp /var/lib/aide/aide.db.gz /root/aide.db.`/bin/date +%Y%m%d`.gz

     
  12. Make /etc/cron.weekly/update.cron executable:

    chmod 755 /etc/cron.weekly/update.cron
     

References

May 21, 2012

Setting up a CentOS 6.2 web server: Installation

CentOS is the free release version of Red Hat Linux with all the branding removed. It also does not have the support options and some of the fancy trimmings the Enterprise version offers, but it is still a solid server OS. This guide is a brief step-by-step guide in how to install CentOS 6.2 and configure it as a web-server.

Install from DVD:

  1. Boot up from your DVD (you will need to enter into the Boot menu of your computer OR edit your BIOS to do so)

  2. Select Install or upgrade an existing system from the menu

  3. If you are worried about your DVD you can choose to to test it, but this is not a necessary step so you can skip it.

  4. On the Welcome screen select 'Next'

  5. Select your language (in our case we are going for the default of 'English (English)')

  6. Select your keyboard type (in Australia we use 'U.S. English')

  7. If you are just going for a standard local hard-drive set-up then just choose the 'Basic Storage Devices' option. If you are going for something fancy (such as network storage or special drives), or you just want to disable some devices for that extra level of paranoid security then choose 'Specialized Storage Devices'.

    If you have no idea which one you should choose then just select the Basic option.

  8. Enter in the host-name of your new server (for best results you should append your domain name to the end so it works seamlessly with SSL certificates) i.e. testserver.example.com

    If you want to configure a static IP address click on the 'Configure Network' button, select the your network card (probably eth0) and enter away.

    If you are going to use DHCP, or just don't know, just hit 'Next'

  9. Select the correct timezone for you (just click a location on the map and it should select the closest one to you).

  10. Enter in an appropriate root password. Make as long and complex as possible (long sentences with mixed character types are easier to remember than jibberish strings; for instance 'My office is situated in 1234 fake street, Fakeville!')

  11. In this example we are going to go for a custom partition layout, so select 'Create Custom Layout'. If you are fine with defaults, just skip to part .

  12. Delete all existing partitions and do the following:
    • A /boot partition of about 100MB. Use the ext4 format

    • Create a LVM Physical Volume that fills up the rest of the hard-drive

    • Create a LVM Volume Group with a Physical Extent of 4MB.

    • Create LVM Logical Volumes on the Volume group as follows:

      • Swap space that is at least equal to how much RAM is in your server
      • /tmp/ should be as big as the largest file you will be manipulating (for instance, if you are copying a DVD you will need at least 4GB)
      •  /var/log and /var/log/audit are separated so that if your log system goes haywire it does not kill the space for other applications. Dedicate a couple of gigabytes to each.
      • /home/ and /usr/ should be a few gigabytes each. /usr/ just holds your applications and should remain pretty static, while /home/ is where you will store your personal files.
      • /var/ and /var/www/ will contain the majority of space on your system. MySQL stores your database files in /var/lib/mysql/, while Apache runs from /var/www/. Dedicate adequate space to each folder.
      • Your root folder (/) will only need a few GB of space. It will mainly hold configuration files.

  13. The system will take some time to format your hard-drive. Once it is complete it will ask you to install the boot-loader. While the defaults are suitable, for extra security you should consider password protecting your boot-loader.

  14. We can now select our packages. You can customize the system to suit your needs, but for the basics just select 'Basic Server' from the menu and the 'Customize now' from the radio buttons. Hit 'Next'.

  15. Do the following edits:
    • Base system - Remove 'Java Platform' and 'Directory Client'
    • Web Server - Add 'Web Server' and 'PHP support'

  16. Reboot your system!

References

May 19, 2012

Subject notes for 48270: Engineering Enterprise

These are my subject notes for the UTS subject 'Engineering Enterprise', which used to be a core subject for the Bachelor Engineering degree. Hopefully someone else can get use out of them.

The Business Plan

  • A written business plan shows stakeholders where the business is going and how to get there. It also analyses the internal and external environments to determine the competitive edge.
  • Opportunity based entrepreneurship are driven by the founder's personal interests. They are usually found in the high technological and business services sectors.
  • Necessity based entrepreneurship are driven because no other avenues for work can be found. They are usually found in the consumer driven (restaurants) and social services (security) sectors.
  • There are three avenues of product development:
    1. Creator develops a new transformational science or technology that is brought to market.
    2. Integrator acquires core products and combine them to develop value-added products and services.
    3. Adopter acquires products and tailors them to local needs.

Strategic Analysis

  • The competitive analysis of a business involves:
    1. Suppliers: Are there enough suppliers to meet current and future demand? Are all the products roughly the same? Are they easy to substitute?
    2. Substitute: Is it easy to switch to a substitute product? Is the substitute market growing? Does the substitute have an advantage over our product?
    3. Entrants: Is it easy to enter in our market? Are there economies of scale or huge start-up capital required? Are there knowledge or regulatory requirements?
    4. Customers: Are the customers close by? Do they need the product? Have they already got a similar product, and if so will they be willing to switch?
    5. Competitors:How much market share do the competitors already have? Can the market support another business? How competitive is the pricing? 
  • Pareto's 80/20 rule states that 20% of your customers will provide 80% of the revenue. It is vital that the business identifies these ideal customers and concentrate on meeting their needs.
  • Demographics are customer characteristics that are measurable and objective. They include demographic population size, composition (age, sex, race) and geographic location.
  • Pyschographics are more important as it aims to determine the personal values and attitudes of the market. It classifies customers according to social class, personality type and lifestyle.
  • The environmental analysis involves:
    1. Political influences includes government and industry regulations, restrictions and standards that must be followed.
    2. Economic influences involve determining whether economic indicators such as inflation, interest rates, unemployment, GDP growth, and market seasonality are favourable. 
    3. Societal influences includes population, cultural values, geographical density, and concerns of the populace.
    4. Technological influences involve determining whether the pace of innovation and technological convergence is favourable.
  • The internal value chain is:
    Purchase -> Inbound logistics -> Operations -> Distribution logistics -> Sales -> Services
  • The external value chain is:
    Suppliers -> Business -> Customers
  • Vertical integration is where one business acquires another in the same market. It increases the risk because the business is now heavily invested in the same industry.
  • Horizontal integration is where the business purchases a supplier or customer business.
  • The SWOT analysis takes into account:
    1. Strengths: What are the major competencies of the business compared to competitors?
    2. Weaknesses: What are the weaknesses of the business compared to competitors?
    3. Opportunities: What existing products can be extended to new markets? What new products do the existing market need? Is there an opportunity to bring a new product to a new market?
    4. Threats: Are the prices for supplier products increasing? Are there new regulations or technologies that threaten the existing business? Is the customer base shrinking?
  • It is important that a business excels in one or more major competencies. They can be:
    • Features: Range and distinctiveness
    • Innovation: Development of new products
    • Quality: Regularly and reliably meets customer needs
    • Intellectual capital: Patents and IP
    • Financial resources: Lots of money to spend
    • Distribution: Reach customers where ever they are
    • Service: Customer issues are dealt with efficiently and effectively
    • Efficiency: Operation is efficient compared to competitors
    • Brand: High brand recognition
    • Price: Competitive price compared to competitors
  • Types of market opportunities are:
    • Market penetration (Existing Market, Existing Product). Easiest to get into.
    • Market development (New Market, Existing Product)
    • Product development (Existing Market, New Product)
    • Diversification (New Market, New Product). The riskiest opportunity.

Direction

  • The mission statement is the summary of the business as it stands now. It is generally derive from information in the business plan. The mission statement answers the following questions:
    • Who are our customers?
    • What are the customers' needs?
    • How do we solve the need?
  • The vision statement aims to predict the future position of the business. It is where the business wants to be. It answers the following questions:
    • Who will the customers be?
    • What will their needs be?
    • How will we solve those needs?
  • Objectives are developed by converting the mission and vision statements into specific performance targets and results. They are the destination. Objectives should be SMART (Specific, Measurable, Assignable, Realistic and Time-related).
  • Strategy determines how the organisation will achieve the objectives. The strategy should be outlined in the business plan.
  • The Strategic management process can be defined as follows:
    • Analyse the internal and external environments
    • Develop the mission and vision statements
    • Set the objectives
    • Formulate the strategy and plans
    • Implement the plan through projects and business operations

Pricing

  • The pricing structure of a product can be:
    • Controlled: External entity (such as government) will offer regulated pricing or guidelines.
    • Demand: Prices reflect the value of the product to the customer. If is solves an expensive problem then the product can demand a high price.
    • Competitive: Similar products are used as a price benchmark.
    • Cost-plus: Products are priced according to how much it costs to develop plus a profit margin. Ensures that money is made but not necessarily that they will sell.
    • Premium: Products are priced higher than competitors because of perceived benefits.
  • Direct or Variable costs are those which are directly related to the amount of product produced. These are typically material and labour.
  • Indirect or fixed costs are those which are incurred regardless of how much product is produced. These include administration, utilities, R&D, and Factory costs.

Marketing and Sales

  • Key Performance Indicators (KPI) are important to measure marketing and sales. Small increases in sales conversion and repeat purchases can lead to large increases in profit.
  • A sales pipeline is used to identify individual sale opportunities for a specific product. It helps the seller sort and track the progress of 'sales opportunities', not 'accounts'.
  • A sales opportunity should be measurable ($ amount of the sale) and dated (expected date of closure). It will then undergo the following stages:
    • Suspect evaluation: A potential customer has been found and they have contacted us or we need to contact them.
    • Prospect evaluation: The prospect is evaluated in terms of demographics and psychographics.
    • Needs evaluation: Determine collaboratively what the customer needs
    • Opportunity evaluation: Determine what opportunities exist with the customer. Can we sell our product to them to solve their need?
    • Commitment: Get a solid commitment from the customer.
  • To elicit the needs of the customer, we use a combination of the following question types:
    • Background questions uncover facts about the organisation and their environment
    • Problem questions identifies problems, difficulties or dissatisfaction about the current environment
    • Implication questions help build the seriousness of the problem so that it becomes large enough to take action
    • Solution questions focus the buyer on the solution and help them develop part of it. It also reduces objections as the buyer is explaining how the solution can help them, convincing them of the value of the solution.
    • Attitude questions uncover the individual needs and serves the buyers own self-interest. These questions uncover motivations.
    • Commitment questions determine where the seller is in regards to the sale. Good questions are answerable with a yes or a no. You can determine whether the customer is ready to commit when they go from 'whether' to implement the solution to 'how' to implement it.
  • The objective of negotiation is to develop a Win-Win situation. A solution should ideally be reached through objective standards and reasoning, not threats or pressure.
  • A list of tradables should be written up, prioritised and understood in terms of flexibility. These are effectively the bargaining chips of the negotiation. Alternative solutions should be established to ensure there is no pressure to commit to an agreement.

Operations

  • The five C's of credit are:
    • Character (credit history)
    • Capacity (ability to meet credit obligations through operating cash flow)
    • Collateral (pledged assets in case of default)
    • Capital (financial reserves)
    • Conditions (general economic conditions of industry)
  • The three types of business entities are Service, Retail/Wholesale and Manufacture.
  • Types of organisational structures:
    • Simple: The founder is no longer the sole employee. Founder does managerial and critical tasks while employees perform other duties.
    • Functional: Employees do specialised roles and are partitioned into functional groups. Delegation of authority is given to functional managers.
    • Matrix: The organisation is split into line managers (for projects, products or demographic) and functional managers. Each employee has two managers to report to, with line managers handling most of the daily management tasks.
    • Divisional: Each division has it's own self-contained functional units. Each division competes for funds. Economies of scope can be created by sharing resources between divisions.

Calculations

Industry margins
Gross income = Sell Price - Cost of Goods Sold (COGS)
Gross Margin = (Sell Price - COGS) / Sell Price

Break even analysis
Revenue = Sell Price * Sell Volume
Gross Margin = (Revenue - COGS) / Revenue
Contribution Margin = (Revenue - Variable Costs) / Revenue
Break even point = 0 = Revenue - (Variable Costs + Fixed Costs)

Liquidity ratios
Working Capital = Current Assets - Current Liabilities
Current Ratio = Current Assets / Current Liabilities

May 17, 2012

Using RPM

RPM is a based software management system for Red Hat based systems such as Red Hat Enterprise Linux, CentOS, Scientific Linux and Fedora. It can refer to both the command-line utility and the file format used to contain the compiled software source for distribution. If you wish to learn more about the RPM system, I suggest you read Edward C. Bailey's excellent Maximum RPM.

  • Installing a downloaded .rpm file:

    rpm -ihv filename.rpm
  • Updating to the latest .rpm file when an application is already installed (to ensure no files are overwritten):

    rpm -Uhv filename.rpm
  • Querying for an installed package with a certain filename (\* is a wildcard character):

    rpm -qa filename\*
  • Remove an installed .rpm file:

    rpm -e filename
  • Remove multiple installed applications that have similar names:

    rpm -qa filename\* | xargs rpm -e
  • To get even more detailed information about an .rpm file we use:

    rpm -qpi filename.rpm