Using Ruby to parse HTML and create RSS

[elfs]

I can't remember if it was jaylake or Charlie Stross who first turned me on to the blog SF Novelists, a group blog which quickly entered my daily feed of things to read. There was one problem with this blog: it had a secondary feed that was completely worth putting into an RSS Feed, but which the purveyors of the website had somehow neglected.

If you look at the second column, "What our members are saying," it's a list of things the website's contributors are doing elsewhere. That kind of niftiness should have its own feed but doesn't. Naturally, your Elf has a solution.

Many RSS programs can take a script as a source, rather than a URL. The script just has to spew out RSS like any other RSS feed. Either you can use that facility, or you can put this script somewhere in a CGI-aware portion of a website and run it like any other script, and use that URL as the source for your feed. (Note how blithely I assume everyone has access to these sorts of things.)

Hpricot uses XPath, so it was easy to find the second column, and then disassembling it was little more than regular expressions. I used RSS/Maker not because it was the best thing I found, but because it was the first that did the job.

 #!/usr/bin/ruby
 
 require 'hpricot'
 require 'open-uri'
 require 'rss/maker'
 
 # load the RedHanded home page
 version = "2.0" # ["0.9", "1.0", "2.0"]
 content = RSS::Maker.make(version) do |m|
   m.channel.title = "SF Novelists secondary feed"
   m.channel.link = "http://www.sfnovelists.com"
   m.channel.description = "An RSS feed of the second column at SFNovelists.com"
   m.items.do_sort = true # sort items by date
 
   doc = Hpricot(open("http://www.sfnovelists.com/"))
   (doc/'div#mem-saying/ul/li').reverse.each do |en| 
 
     i = m.items.new_item
     en = en.to_s
     en =~ /<a.*href="(.*?)"/; i.link = $1
     en =~ /<a.*title="(.*?)"/; i.description = $1
     i.title = en.gsub(/<[^>]+>/, '')
     i.date = Time.now
   end
 end
 
 print content

Comments

[kyriani.livejournal.com]

Kickass! Definitely goes in my memories list (I've been working on a version of my personal site in Ruby on Rails lately). Thanks for sharing. ^___^

[elfs.livejournal.com]

If you like Ruby on Rails but don't have much use for the database component, consider looking at MERB (http://merbivore.com/). It's younger, but it claims to be faster and database-agnostic. I'm trying to find time to play with it this hectic holiday season.