Comments on: Scrape Digg Search Results Python Script

By: Jan Paricka

Jan Paricka — Mon, 07 Dec 2009 14:54:43 +0000

Matt, we use mostly bitnami (rightscale) server images running on GoGrid. I can bring one server instance up if you want to have a look? I tried everything … :-( Thank you! I very much appreciate your help with this.

By: Matt Warren

Matt Warren — Mon, 07 Dec 2009 14:48:54 +0000

I have tried it on a number of my computers – Linux, Windows and Mac and it works fine. (all with python 2.6)

Are you stuck behind a firewall or proxy server ?

By: Jan Paricka

Jan Paricka — Mon, 07 Dec 2009 14:22:34 +0000

[ jparicka dev ~ ] cat delete.me import urllib, json address="http://services.digg.com/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json" data = urllib.urlopen(address) print data.readlines() [ jparicka dev ~ ] python delete.me ['{"error":{"timestamp":1260195682,"message":"HTTP User-Agent header required","code":1029}}'] I get the same problem on all our boxes. :-(

By: Matt Warren

Matt Warren — Sun, 06 Dec 2009 19:41:33 +0000

I wasn’t able to duplicate the 403 error. but a couple of things to note.

since you’re using the developer api, you don’t need to specify the user agent stuff and you can actually make it much simplier:

>>> import urllib, json
>>> address=”http://services.digg.com/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json”
>>> data = json.load(urllib.urlopen(address))

By: Jan Paricka

Jan Paricka — Sat, 05 Dec 2009 02:23:30 +0000

OK, so I tried to simplify your code to

import urllib, urllib2

USER_AGENT = ‘Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)’

address = “http://services.digg.com/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json”

request = urllib2.Request(address, None, {‘User-Agent’:USER_AGENT} )

urlfile = urllib2.urlopen(request)

page = urlfile.read(200000)

urlfile.close()

………..but receiving 403.

Digg failed to help me out – and I tried from like 10 different servers (all hosted on goGrid)

I am desperate. Any ideas?

Thank you!!

[jparicka@25358_2_42578_205369:~] python abcd.py
Traceback (most recent call last):
File “abcd.py”, line 9, in ?
urlfile = urllib2.urlopen(request)
File “/usr/lib/python2.4/urllib2.py”, line 130, in urlopen
return _opener.open(url, data)
File “/usr/lib/python2.4/urllib2.py”, line 364, in open
response = meth(req, response)
File “/usr/lib/python2.4/urllib2.py”, line 471, in http_response
response = self.parent.error(
File “/usr/lib/python2.4/urllib2.py”, line 402, in error
return self._call_chain(*args)
File “/usr/lib/python2.4/urllib2.py”, line 337, in _call_chain
result = func(*args)
File “/usr/lib/python2.4/urllib2.py”, line 480, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden