Development Frustrations with Google App Engine

July 3rd, 2009

app_engine_logo_smFor the past two weeks I have been working on a project that has great potential of really taking off in a big way. I’m developing the site using Python and the Django framework running on Google App Engine. I have a lot of good things to say about working with this development stack. Some of the big wins are:

  • Python is an awesome language – easy to read, write, and maintain.  There’s lots of libraries available which makes development go faster.
  • Django is a great framework that makes developing webapps very clean.  There’s a great separation between templates, views, and urls.  Once I got the hang of how things are supposed to be done in django it’s very easy  to get things up quickly.
  • Google App Engine has a really amazing admin interface that gives access to the logging information, database tables, and website statistics.  The free quotas are generous, it scales well, and takes almost no time to set up.  The GUI development app for OS X works really well and does development debugging better than the stock django manage.py script.

But there have been some really frustrating points during the development of my first real web service running on GAE.

  • There are too many choices/variations of Django – none have great documentation
    • The built in Django that comes with GAE is stripped down to the bare essentials – no admin interface, different forms, different Models, different User/authentication.  Big portions of the documentation at djangoproject.org are useless if you use this version of Django.
    • app-engine-helper – provides a way to get more of the standard Django installed.  I haven’t tried this one.
    • app-engine-patch – similar to helper, but development seems more active.  app-engine-patch also includes a bunch of app-engine ready Django applications such as jQuery, blueprintCSS, and registration.  It supports using standard Django user accounts and the admin interface.

The biggest problem I’ve had is with user registration and authentication. Between the app-engine-patch and Google App Engine, there seems to be at least 4 different authentication and session schemes, and multiple User Models to choose from. Some require additional middleware – others don’t. I want to use the registration application and standard Django Users but it doesn’t seem to want to work with a Model’s UserProperty. To top it off there’s very little documentation and I haven’t found an example application to see how it should be done. Argh.

The exciting news is that I expect to have my first web service up and running in about a week. The second one is in development and I expect to launch it in early August.

Technorati Tags: , , , , , , , , ,

RSS Twitter Bot in Python

July 1st, 2009

I was a little bored today and decided to write up a simple script that pushes RSS feed information out to Twitter and manages to keep track of the history so that tweets are not sent out more than once.

It was actually a very trivial little script to write but it could actually be useful for something that I’m working on in the future.

The script makes use of an Sqlite database to store history and bit.ly for shortening URLs. I’ve made heavy use of some really nice open source libraries to make for a very short and sweet little script.

Grab the necessary python libraries:
python-twitter
python-bitly
feedparser

You’ll need to sign up for free accounts at Twitter and Bit.ly to use this script.

Hopefully someone out there can take this code example to do something really cool with Twitter and Python.

Update: I’ve added some bit.ly link tracking output to this script. After it twitters the RSS feed it will print out the click count information for every bit.ly link.

import twitter     #http://code.google.com/p/python-twitter/
import bitly       #http://code.google.com/p/python-bitly/
import feedparser  #available at feedparser.org
import sqlite3
from time import strftime

DATABASE = "tweets.sqlite"

BITLY_LOGIN = "bitlyUsername"
BITLY_API_KEY = "api key"

TWITTER_USER = "username"
TWITTER_PASSWORD = "password"

def printStats():
	conn = sqlite3.connect(DATABASE)
	conn.row_factory = sqlite3.Row
	c = conn.cursor()

	b = bitly.Api(login=BITLY_LOGIN,apikey=BITLY_API_KEY)

	c.execute('SELECT title, url, short_url from RSSContent')
	all_links = c.fetchall()

	for row in all_links:

		short_url = row['short_url']

		if short_url == None:
			short_url = b.shorten(row['url'])
			c.execute('UPDATE RSSContent SET `short_url`=? WHERE `url`=?',(short_url,row['url']))

		stats = b.stats(short_url)
		print "%s - User clicks %s, total clicks: %s" % (row['title'], stats.user_clicks,stats.total_clicks)

	conn.commit()

def tweetRSS(url):

	conn = sqlite3.connect(DATABASE)
	conn.row_factory = sqlite3.Row
	c = conn.cursor()

	#create the table if it doesn't exist
	c.execute('CREATE TABLE IF NOT EXISTS RSSContent (`url`, `title`, `dateAdded`, `content`, `short_url`)')

	api = twitter.Api(username=TWITTER_USER, password=TWITTER_PASSWORD)
	b = bitly.Api(login=BITLY_LOGIN,apikey=BITLY_API_KEY)

	d = feedparser.parse(url)

	for entry in d.entries:

		#check for duplicates
		c.execute('select * from RSSContent where url=?', (entry.link,))
		if len(c.fetchall()) == 0:

			tweet_text = "%s - %s" % (entry.title, entry.summary)

			shortened_link = b.shorten(entry.link)

			t = (entry.link, entry.title, strftime("%Y-%m-%d %H:%M:%S", entry.updated_parsed), entry.summary, shortened_link)
			c.execute('insert into RSSContent (`url`, `title`,`dateAdded`, `content`, `short_url`) values (?,?,?,?,?)', t)
			print "%s.. %s" % (tweet_text[:115], shortened_link)

			api.PostUpdate("%s.. %s" % (tweet_text[:115], shortened_link))

	conn.commit()

if __name__ == '__main__':
  tweetRSS('http://www.halotis.com/feed/')
  printStats()

Technorati Tags: , , , , , , , , , ,

YouTube Search Results into Database

June 29th, 2009

I discovered this very handy trick for getting relevant YouTube videos in an RSS feed and I have used it to build up some very powerful blog posting scripts to grab relevant content for some blogs that I have.  It could also be helpful to pull these into an RSS Reader to quickly skim the newest videos relevant to a specific search.  I thought I would share some of this with you and hopefully you’ll be able to use these scripts to get an idea of your own.

To start with you need to create the URL of the RSS feed for the search.  To do that you can do the search on YouTube and click the RSS icon in the address bar.  The structure of the URL should be something like this:

http://gdata.youtube.com/feeds/base/videos?q=halotis%20marketing&client=ytapi-youtube-search&alt=rss&v=2

The problem with the RSS feed is that they don’t include the HTML required to embed the video. You have to parse the RSS content and find the URL for the video which can be used to create the embed code so you can post the videos somewhere else.

In my example code I have categorized each url to a target keyword phrase.  The code below is not a comprehensive program, but just an idea of how to go about repurposing YouTube RSS content.

import feedparser  # available at: feedparser.org
from time import strftime
import sqlite3

DATABASE = "YouTubeMatches.sqlite"

conn = sqlite3.connect(DATABASE)
conn.row_factory = sqlite3.Row
c = conn.cursor()

def LoadIntoDatabase(phrase, url):

	d = feedparser.parse(url)
	for entry in d.entries:

		#check for duplicates with the url
		Constants.c.execute('select * from YoutubeResources where url=?', (entry.link,))
		if len(Constants.c.fetchall()) == 0:
			#only adding the resources that are not already in the table.
			t = (phrase,entry.link, 0, entry.title, strftime("%Y-%m-%d %H:%M:%S", entry.updated_parsed), entry.summary)
			Constants.c.execute('insert into YoutubeResources (`phrase`, `url`, `used`, `title`,`dateAdded`, `content`) values (?,?,?,?,?,?)', t)

	Constants.conn.commit()

def getYouTubeEmbedCode(phrase):

	c.execute("select * from YoutubeResources where phrase=? and used=0", (phrase,))
	result = Constants.c.fetchall()
	random.shuffle(result)

	content = result[0]

	contentString = content[3]
	url=content[1].replace('?', '').replace('=' , '/')
	embedCode='<div class="youtube-video"><object width="425" height="344"><param name="movie" value="%s&hl=en&fs=1"> </param><param name="allowFullScreen" value="true"> </param><param name="allowscriptaccess" value="always"> </param><embed src="%s&hl=en&fs=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="344"> </embed></object></div>' % (url, url)

	t=time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
	c.execute("UPDATE YoutubeResources SET used = '1', dateUsed = ? WHERE url = ? ", (t, url) )

	return embedCode

Technorati Tags: , , , , , , ,

Meetup with John Chow

June 27th, 2009

johnchow-photoThe other night I attended a Meetup where John Chow was speaking to a room of about 35 people.  If you’re not familiar with John Chow, he runs a blog – johnchow.com that makes $40,000+/month. This year he’s expecting to make $500,000 just from his blog.  His talk was amazingly inspirational in that it became painfully obvious that the work involved is pretty trivial.  John only works 2 hours a day on the blog.

He walked us through the back-end of his business model which is where he makes 2/3rds of his profit.  It is a system that I’ve had in place for a few of my websites but never really paid much attention to really tweaking it out.  I think that with just a bit of consistent work I could duplicate some of John’s success with my blogs.  Basically John has none of his own products so everything is affiliate and joint venture deals and, surprisingly, 70% of the income is not coming from the blog but from his email list.

His number one goal for his site is to capture the email address of the visitor so that he can keep in contact with them and build a relationship.  Then he just gives out a couple of his recommendations and he makes lots of money.  His number one recommendation for us was to focus on building the list and then then push people through a sales funnel to get some conversions.

John uses and recommended everyone that wants to make money online get an account with the best email auto responder service out there – AWeber.  The truth is that sending lots of email is a very difficult technical problem it’s much better to start with the correct service than try to move over your email list once you have it.  Both John and I agree that AWeber is by far the best email service out there.

Trying to save a couple of bucks early on by collecting emails and sending them yourself is a big mistake.  John used a different email service while he grew his list to 25000 then he discovered that only 70% were getting through the spam filters.  That’s a lot of money he’s missing out on!  So he decided to move the list over to aweber and he ended up losing 80% of his subscribers in the process…  By CANSPAM law you can’t move your email list without opting everyone into the list again.  That was a huge setback for John so he recommends that you start with the best.  It is probably the most cost effective tool you can buy to make money online.

John Chow’s business model is such a simple system but it works because John is consistently posting to his blog and is always getting fresh traffic to grow his list.  It can actually be a lot harder in practice since it’s so easy to try blogging for a week and then give up.  Consistency is key.

In my experience the inconsistency of posting to this site has almost killed it.  And if it wasn’t for a few good posts that got ranked well in Google then almost nobody would be here.  It is actually surprising that you never really know what posts will get good traction so it’s important to keep writing good stuff and keep things keyword targeted as much as possible.  With enough content out there eventually something will stick and you’ll continuously get traffic without much additional work.

John gave us lots of little tips – some of which I have already implemented on my sites.

His site is proof that the concept does work and it really got me stirred up again with ambitions to start writing more content and get my business model ironed out a bit more.

Technorati Tags: , , , ,

DropBox for sharing & syncing Files

June 25th, 2009

dropbox

Dropbox is one of the best services there is for moving files from computer to computer, sharing videos or pictures work or even applications. If you’ve ever emailed yourself a file, or wondered how send a friend a video without publishing it on youtube then you should at least check out what Dropbox will do for you.

In a nutshell, Dropbox is a web service where you select a particular folder on your computer to be the “dropbox” and any file you put in there gets immediately synced up to the web and pulled down to any other computers that you have the software installed on. The files are also available through their website if you want to get at them from a computer without the software.

So the way that I have it set up is that I have it on my laptop, and also on my computer at work. When I get some MP3s that I want to send to myself at the office I’ll just download them at home and copy them into my dropbox. Then the next day at work the files will already be on my computer and I can listen to them right away. Or when I find something online that I want to have that I want to look at on my laptop I can download it at work and it will be on my laptop when I get home.

One of the other neat tips I’ve figured out is that I can use it to keep my personal project code in sync. I use Mercurial to manage the history of my code and I can keep a repository of that code in my dropbox. Then anytime I work on the code at home those changes are immediately available to me at the office, or anywhere with internet access.

It also acts as a small backup for my secret files. I have an encrypted file that contains some sensitive documents, and all my saved internet passwords. If I lose my laptop or the hard drive dies I will still be able to get at that information and restore it.

There’s also a special folder within the dropbox folder for shared files. Any files placed within the Public folder is visible to anyone with the URL. I’ve seen people use it as a place to host video files which are posted to blog sites which is a cool way to host the files and bandwidth for free. Here’s my secret text file that I made public.

The free account gives you 2GB of space which is not too shabby. I’ve found it to be an awesome tool for moving bigger files between computers and I would recommend it to just about anyone that wants to share or move files over the internet.

Check out dropbox at getdropbox.com

Technorati Tags: , , , , ,

Download My Free Internet Marketing Software

June 23rd, 2009

I am a programmer and over the years that I’ve been working to make money online I have spent a considerable amount of time writing small scripts and tools to help me manage my marketing efforts.  Already on this site I have published a couple of things such as a url redirection script that allows you to create branded urls very easily, and a twitter client for Excel.

But I have a secret stash of software that I’ve written which I’m going to be making public very soon on this site.

Stay tuned for my program that re-purposes content from youtube and some other sources and pushes it out to your network of blogs.  This software will allow you to scale up your network of websites massively to drive some serious traffic.

Up to now I’ve been holding back on publishing my software on this site, but look forward to seeing more code examples and software ready for you to download and use in the near future.

Technorati Tags: , , , , ,

Twitter in Excel

June 18th, 2009

Have you ever wanted to send and receive tweets right from Microsoft Excel?  Well now you can!  This Excel based Twitter client allows you to read your timeline and post tweets right from within Microsoft Excel.

Here are some possible uses:

  • twitter from work without installing any client software or keeping a web browser open
  • pull your twitter timeline into a pivot table for analysis
  • write a macro to post tweets for you
  • graph your tweets
  • archive your timeline as a .csv file
  • convert your timeline into a .csv file so that it can be easily read by another script
  • keep up with the trending topics around the world in Excel
  • use it as an example of how to write complex applications in Excel

I found this file somewhere online a while ago, translated it into English and made a few modifications.  If you’re interested in trying it out, you can download the file.


Download Twitexceler.xls

Technorati Tags: , , , , , ,

NLP Copywriting

January 29th, 2009

Just a quick link to post.

Dr. Harlan Kilstein is an NLP practitioner that specializes in using NLP techniques to make your copywriting sing.  He’s currently in the middle of a 30 day video series that is packed with useful tips about how to write better copy.

If you haven’t heard of NLP you are missing out. It’s a fascinating subject and the implications are simply amazing.

This is just one of the videos:

I’ve been devouring all of the videos on his site for the last few days… it’s a real goldmine of information. His site is NLPCopywriting.com

Technorati Tags: , , ,

Blog Auto-Posting Robot… Interested?

January 27th, 2009

Over the past year I’ve hacked together a program to help me maintain a lot of blogs without actually having to write posts for them.  The idea was to just refresh them with relevant, good quality information with zero input from me.

As a result I have built up a piece of software that reliably downloads content from a number of supported sources including youtube, and prweb, and queues it up in a database for later automatic posting.

The software is then scheduled on my computer to post one item from the database for each blog I have set up.

In addtion to simply posting the content such as a youtube video, it also has the ability of inserting additional html before the post.  That way I can add in an affiliate link or an adSense ad to each post.  The way I have it currently set up is to pick one at random from a list of potential html snippits for each site.

The software supports auto-posting to self-hosted wordpress, wordpress.com, and tumblr blogs.  With the wordpress sites it supports putting the posts into separate categories depending on the keyword your targetting.

The question I have is:  would people be interesting in purchasing such a piece of software?  What would you do if you could be hands off and maintain hundreds of blogs automatically with fresh content and videos?

The reason I ask is because even though the software works awesome for me, it would take some time and effort to clean it up and make it easy to use.

Leave a comment below and let me know if you think it’s worth pursuing further.

Technorati Tags: , , , , ,

Inevitability Thinking

December 2nd, 2008

I was just introduced to the concept of Inevitability Thinking and I felt so excited that I just had to write a post about it.  inevitability thinking is a real game changer and perhaps the most important thing that I have ever come across to guarantee success.

You are probably familiar with the concept of goal setting.  Goal setting has been recognized as a key part of achieving the success you want in life.  It basically involves thinking about where you see yourself in the future.  It’s a powerful and important aspect of developing your plan for success because without a destination you’re really just meandering through life.

Earl Nightingale recorded the very first gold spoken word record.  This record was really the beginning of the modern self-help market and in it he impressed the importance of proper goal setting.

An evolution to that idea was made by Robert Biel who found that a majority of people prefer to think in terms of problem solving.  Make a list of the things standing in the way of you reaching your goal and then systematically solve the problems that are in the way of reaching that ultimate goal.

Inevitability thinking operates at the root of both of these.  The real problem that most people have it is the follow through.  All the best intentions in the world won’t ensure that you actually get it done.  You have to ask yourself these two questions:

  1. What condition would force me to make a necessary part of my goal?
  2. What can I do to get that condition in place as soon as possible?

If you’ve taken the time to really establish your long and short term goals and developed those goals fully to understand what your life will be like when you have reached them, and then enumerated all the necessary things that will have to happen. Then the final step is to make sure that they do happen.

For example, imagine you’ve set a personal to get fit and run a marathon.  Then you would have to visualize what specifically you would have to do to reach that goal. How much exercise would you need to do every day, when and where would you do it?  If you’ve developed a plan and know what you can do today to take a step towards your goal then you need to create a condition that would force it to happen.  This might be to find someone to be accountable to for your workouts, higher a coach, form a mastermind group to talk with regularly or put something in the way.  Anything you can do to make avoiding the things you have to do impossible.

It’s perhaps the most important question that somehow was never impressed upon me until now.

What do you think?

Technorati Tags: , , , ,