Rogers Cadenhead recently wrote a post “The Reason RSS Cloud Failed to Catch On“. In this he argues that RSS Cloud is not sustainable when it begins to scale to thousands of users needing to receive pings every time a post is made. While he has a point, I believe his argument is much more accurate when put into the context of 1999 as opposed to 2009, and it is even more irrelevant a few years from now. This points to one thing… RSS Cloud was ahead of its time.
Today, supporting RSS Cloud is cheap. Bandwidth is a fraction of a percent now compared to what is was 10 years ago along with cloud based virtual servers available for ~$10 / month. I’d suspect a properly tweaked VPS at the Rackspace Cloud or EC2 could handle millions of pings / day without issue. Unlike PubSubHubBub, the content of the post is not delivered with RSS Cloud, it is just a body-less ping, so bandwidth isn’t even an issue. A server supporting these types of capabilities back in 1999 would have cost thousands of dollars per month. Now? Just a few bucks. 5 years from now I suspect it will be free (example: Google App Engine is currently free and could be configured for this service).
Let’s go ahead and take into consideration the dream scenario with RSS Cloud, and that is that we have completely replaced Twitter. In this scenario Ashton Kutcher will have 5 million followers and will need to notify each of those followers when he tweets out 100 mundane posts / day. Let’s not kid ourselves and not think there is a middle man in there somewhere. These services are the “Cloud” portion of whole term. Just like Feedburner and Google Reader are proxies between publisher and reader now (by storing local caches of RSS feeds), there will be many services that would be happy to host the feeds, providing a web client to the reader and analytics to the publisher in turn receiving all the statistics they chew on. Proof? Twitter purchased Summize (search.twitter.com) and Google bought FeedBurner because they saw the value of this type of service. Afterall, the analytics portion of that is the whole business model that Twitter is rumored to be building.
But ok… let’s assume those “Cloud” services never pop up because, hell… I don’t see any reason why they wouldn’t, but let’s just imagine they don’t exist. Solution? TURN OFF THE NOTIFICATIONS! That’s the beauty of RSS Cloud feeds, it’s completely backwards compatible with non-Cloud enabled feeds. If the Cloud server fails to send out notifications, then their subscribers will simply revert back to the old method of routine polls looking for new content. But, failing to notify your subscribers of new content in a sea of real-time has drastic consequences. Even if you are the first to report information, you will always be “scooped” by others because the lag caused by your inability to notify your subscribers. That alone will be enough to even the playing field. As a compromise, you could get away with just notifying a few dozen RSS aggregators (Google, Yahoo, Feedburner) to take care of the majority of your subscribers, and those using readers you don’t notify will revert to polls.
So as you can see, RSS Cloud is currently gaining traction because it wasn’t until now that it was a viable service for the masses. Even now there is very little support within the client portion of RSS Cloud, but I suspect that will be a battleground many will fight for in the next couple years.
9 Comments for The Reason RSS Cloud Can Work Now
Rogers Cadenhead | September 9, 2009 at 1:29 AM
Author comment by Derek | September 9, 2009 at 1:41 AM
“When all each client wants is the newest update.”
The publishing server tells the client what URL to go to to grab the latest, so, serve up the last couple messages instead of the whole feed if that is necessary.
Besides, that example was given to show that the RSS Cloud system -isn’t- designed for 5 million pings to be sent out. You are missing the “Cloud” element here (the aggregators, Google Reader, Feedburner, Twitter, etc…) and that the publishing server can decide who to syndicate content to in real-time if he or she wishes. In 2009, real-time syndication to millions could be an issue, but in 2015, it may be a piece of cake.
mcdtracy | September 9, 2009 at 9:13 AM
Derek:
If the “ping” also carried the 140 character “twit” that could be cached in the cloud then RSS Cloud would address the issues if real-time and scalability.
In it’s current form it’s distributes the ping service and assumes a massive growth of RSS aggregators that are all coordinating well with the distributed/cloud ping service.
Adding the twit to the ping would be a better optimization. The delta would go out into the cloud at the same time and elimiate duplication.
Rogers Cadenhead | September 9, 2009 at 11:29 AM
“The publishing server tells the client what URL to go to to grab the latest, so, serve up the last couple messages instead of the whole feed if that is necessary.”
That’s a good idea, but not one that’s part of RSSCloud at this time. The more I look at PSHB, the more I like how it pushes the actual item to interested clients, as opposed to pushing a request to poll a feed.
Author comment by Derek | September 9, 2009 at 12:02 PM
Rogers: Am I not reading it right? Implementor’s guide to rssCloud: The Aggregator.
Your server will receive a single parameter, the URL of the feed that updated. On receipt of the message you should read the feed, and do what you normally do to locate and present new items to the user. The return value is ignored by the cloud.
You must tell the aggregator something in your POST, it can’t just be a empty. That “something” is the URL that was updated, and you can determine exactly how to present new info whether it is a 200 item feed or a 1 item feed.
“The more I look at PSHB, the more I like how it pushes the actual item to interested clients, as opposed to pushing a request to poll a feed.”
Now we’re back to the greatest debate in syndication formats…. Atom vs. RSS (yes I know PSHB can hook into RSS). Both are good protocols and I’m cool with whichever wins out. To me though, the beauty of RSS Cloud is that it’s so simple, and when it comes to decentralized systems, simplicity usually wins out. While PSHB could potentially be more efficient, the only thing RSS Cloud changes in the RSS picture is that it decreases stale polls in return for teeny tiny POST notifications, which are cheap to begin with.
When you talk about scaling a system like this to 10 million followers, it becomes an issue that has nothing to do with RSS Cloud and is a fundamental issue with notifying that many people, regardless of protocol.
Nome | September 10, 2009 at 4:11 AM
Appengine can’t support rsscloud because of the stupid restriction that the subscription request must come from the same ip as the subscrption endpoint. That’s not the case on appengine, nor in a lot of distributed applications
pubsubhubub and rss-cloud : changing the way you read the web « Thoughts on technology and social web | September 10, 2009 at 12:29 PM
[...] Vs. PubSubHubbub: Why The Fat Pings Win There’s a Reason RSSCloud Failed to Catch On The Reason RSS Cloud Can Work Now Possibly related posts: (automatically generated)Real time roundup – Part 3 : Instant [...]
Wes Felter | September 16, 2009 at 10:18 PM
It’s interesting that you mention bandwidth, since PubSubHubbub uses less bandwidth than RSSCloud (especially if posts are little 140-character tweets where the overhead will be magnified).
feeds, realtime, and stuff. a link dump. « turnings | October 21, 2009 at 12:36 PM
[...] the-reason-rss-cloud-will-catch-on [...]
Under your scenario, Ashton Kutcher will require five million cloud notification updates sent from cloud servers to RSS clients over XML-RPC, SOAP or REST. Each notification will prompt the client to request Kutcher’s RSS feed again, so he’ll serve it five million times. When all each client wants is the newest update.
As a general rule, it’s a bad plan to create a technology that assumes a massive infrastructure will spring up to make its astonishing levels of resource consumption possible.