[...] How To Get RSS Content Into An Sqlite Database With Python (tags: python rss web programming) Leave a Comment [...]

Hi,

Sorry to bug you with another question, but I really like this script and am wondering if you have any general tips on how to modify it such that it would read from a text file containing a number of different RSS feeds, then put the information from all the feeds into a single pysqlite database?

Thanks in advance for your time.

Greg

Sorry about that–I should have seen that that was already built into your script.

It works great by the way, so thanks!

Greg

OK, I do have one question that I can’t seem to figure out. I am interested in using a list of Twitter RSS feeds, and I’d like to put the http://bit.ly URLs as well as the hashtags in separate columns in the table. However, there does not seem to be an element or item in the Twitter RSS feed separating out these things, so that the “content” column in the pysqlite database contains the content, tiny url, and hashtags all together. So, here’s my question: would you separate these out with a regular expression, or some other method?

Thanks for any advice.

Thanks. That’s great!

Hi Matt,

I’m trying to convert this script to use sqlalchemy and have a couple of questions. I am able to successfully get everything put into the RSSEntries table using sqlalchemy, but there are a few “leftover pieces” in the code that don’t seem to translate:

1) To replace your “conn=sqlite3.connect(DATABASE) line I have written “conn=engine.commit” (and that works). However, I don’t know what to do with the “conn.row_factory = sqlite3.Row” line. What does that do? I’ve left it out and the script still seems to work.

2) If I try to use the “c=conn.cursor()” line I keep getting an error:

c = conn.cursor()
AttributeError: ‘Connection’ object has no attribute ‘cursor’

I guess sqlalchemy doesn’t use “cursor” in this fashion?

3) Because of the “cursor” error, I replaced your “feeds = …” line with “feeds = conn.execute(‘SELECT id, url FROM feeds’).fetchall()”, which works well. However, I can’t get your “if len(c.fetchall()) == 0″ line to work, even when I replace “c” with “conn” or “engine.” (In my initial test, I just commented out that line then the script works). I am guessing this line helps avert duplicates? Is that right? And if so, any idea how to do this using sqlalchemy? Here’s the error I get:

if len(conn.fetchall()) == 0:
AttributeError: ‘Connection’ object has no attribute ‘fetchall’

I hope this makes sense–I can attach the whole code if it doesn’t–and thanks in advance for any help.

-Greg

Assuming you have the models defined as RSSEntries and RSSFeeds classes then converting to sqlalchemy would be something like :

CONNSTRING='sqlite:///rss.sqlite'
 
Base = declarative_base()
class RSSFeeds(Base):
    __tablename__ = 'RSSFeeds'
 
    id = Column(Integer, primary_key=True)
    url = Column(String)
 
    def __init__(self, url):
        self.url = url
 
    def __repr__(self):
        return "" % (self.url, ))
 
 
class RSSEntries(Base):
    __tablename__ = 'RSSEntries'
 
    id = Column(Integer, primary_key=True)
    url = Column(String)
    title = Column(String)
    content = Column(String)
    date = Column(Date)
 
    def __init__(self, url, title, content, date):
        self.url = url
        self.title = title
        self.content = content
        self.date = date
 
    def __repr__(self):
        return "" % (self.url, self.title, ))
 
 
engine = create_engine(CONNSTRING)
 
metadata = Base.metadata
metadata.create_all(engine) 
 
Session = sessionmaker(bind=engine)
session = Session()
feeds = session.query(RSSFeeds).all()
 
def store_feed_items(id, items):
    """ Takes a feed_id and a list of items and stored them in the DB """
    for entry in items:
        checkitem = session.query(RSSEntries).filter_by(url=entry.link).all()
            if not checkitem:
            item = RSSEntries(entry.link, entry.title, entry.summary, entry.updated_parsed)
            session.add(item)
    session.commit()

Note: I haven’t actually tested this code to make sure it works.

Hi, I am new to Perl but I do need this functionality for the site. How would I import this into a MySQL database using Python? And also, where can I find some information on how to set up Python to run under Apache?

Oops, meant to say – new to Python :)

for mySQL – if you take the code in the comments that uses sqlalchemy, then you can just change the connection string to mysql://user:password@sqlserver:port and it should just work.

getting python working on apache- look into mod-python

[...] the blog post from Halotis that started my looking into [...]

[...] How To Get RSS Content Into An Sqlite Database With Python – Fast [...]