Comments on: RSS Scaling Problems http://joseph.randomnetworks.com/archives/2004/07/26/rss-scaling-problems/ cat /dev/random Sat, 04 Jul 2009 08:07:38 +0000 http://wordpress.org/?v=2.9-rare hourly 1 By: Joseph Scott http://joseph.randomnetworks.com/archives/2004/07/26/rss-scaling-problems/comment-page-1/#comment-28 Joseph Scott Wed, 28 Jul 2004 20:37:06 +0000 http://joseph.randomnetworks.com/archives/2004/07/26/rss-scaling-problems/#comment-28 From what I read the biggest problem is not bandwidth, it is the sudden load spike on the systems hosting the RSS feeds when many clients try to get updates at the top of each hour. To reduce the load the obvious thing to do is spread it out, not to reduce bandwidth but to reduce system load. Using something like FreeCache would reduce both load and bandwidth. Another really important factor that no one seems to have mentioned is that this should be transparent to the aggregators. Throwing in things like BitTorrent and new HTTP extensions are great ideas, but getting aggregators to support them may never happen. Again, using something like FreeCache requires no changes to any of the RSS/ATOM clients out there. It's a solution that will work right now, which is generally better that proposing solutions that will only work when everyone rewrites their software. From what I read the biggest problem is not bandwidth, it is the sudden load spike on the systems hosting the RSS feeds when many clients try to get updates at the top of each hour. To reduce the load the obvious thing to do is spread it out, not to reduce bandwidth but to reduce system load. Using something like FreeCache would reduce both load and bandwidth.

Another really important factor that no one seems to have mentioned is that this should be transparent to the aggregators. Throwing in things like BitTorrent and new HTTP extensions are great ideas, but getting aggregators to support them may never happen. Again, using something like FreeCache requires no changes to any of the RSS/ATOM clients out there. It’s a solution that will work right now, which is generally better that proposing solutions that will only work when everyone rewrites their software.

]]> By: th0m http://joseph.randomnetworks.com/archives/2004/07/26/rss-scaling-problems/comment-page-1/#comment-27 th0m Wed, 28 Jul 2004 20:02:05 +0000 http://joseph.randomnetworks.com/archives/2004/07/26/rss-scaling-problems/#comment-27 you read the slashdot article? NNTP is designed for this sole purpose of a topology. Also, if aggregators were designed better, HTTP has the Modified header and they would cause significantly less traffic. Algorythms for randomizing subscription requests could also help. Have fun! Also, check out http://writtorrent.sf.net/ you read the slashdot article? NNTP is designed for this sole purpose of a topology. Also, if aggregators were designed better, HTTP has the Modified header and they would cause significantly less traffic. Algorythms for randomizing subscription requests could also help.

Have fun! Also, check out http://writtorrent.sf.net/

]]>