Best Practice For Retiring RSS Feeds? 71
GBJ writes "I work for an organisation that runs seasonal online competition events. Each event has its own news feed which becomes obsolete shortly after the event finishes. We're still getting RSS requests for some events as far back as 2004. I'd like to close a few thousand old feeds and remove the resource hit they cause, but I'm not sure what is the best approach. Currently I'm considering just returning a 404, but I have no idea if there is a better way to handle this. Uncle Google hasn't turned anything up yet, but sometimes it's hard to find something when you don't know what it's called ..."
Retiring feeds (Score:5, Funny)
A phase-out needs.
As facial bristles,
Or torn skin bleeds.
Burma Shave
Re: (Score:3, Insightful)
Forward them to a page listings current feeds, possibly with custom tailored advertisement-style links to events or other websites in your network.
Never waste traffic. At the end of the day, traffic is KING.
Retirement (Score:1)
301 (Score:5, Insightful)
You should be able to do a 301 redirect to maybe some generic feed that just has one entry that says "This feed is out of date, please use try these feeds instead." Or whatever you want to let them know.
410 (Score:5, Informative)
Gone
Indicates that the resource requested is no longer available and will not be available again. This should be used when a resource has been intentionally removed; however, it is not necessary to return this code and a 404 Not Found can be issued instead. Upon receiving a 410 status code, the client should not request the resource again in the future. Clients such as search engines should remove the resource from their indexes.
but then you loose the traffic (Score:4, Informative)
No way man. You can't just dump all those people like that!
Either redirect them to a feed that says "here are some other events you might be interested in subscribing to" or create a last post on the feed that says the same thing. Dumping them with a 410 is a great way to loose your traffic.
It sounds like in this case, a redirect is the proper thing. Just be forewarned that some readers (cough Google Reader) will redirect, but ignore the "permanent" in "permanent redirect" and will continue to hit the old feed.
Re: (Score:3, Insightful)
Dumping them with a 410 is a great way to loose your traffic.
I'd like to close a few thousand old feeds and remove the resource hit they cause...
So? (Score:3, Informative)
Are RSS feeds really causing an insane amount of traffic? Does that traffic cost more than it does to acquire new subscribers? I doubt it.
Let me qualify that though. Before I did anything drastic like 410'in their ass, I'd come up with the total number of subscribers to the entire pool of feeds. Most of the big-boy newsreaders will usually include a subscriber count in their User-Agent. Tally up the numbers and make sure you aren't dumping 100,000 subscribers in an attempt to save a piddly 1,000 feed-h
Re: (Score:3, Insightful)
Anyone who runs a website would be extremely short-sided to discourage traffic in any way. What I'd do is create a new, more generic, news feed and redirect all the other feeds to it. The few people who simply forgot they were subscribed to the 2004 feed will either realize their mistake and drop off, or be still interested in your new events and stay on. If you just make the feeds disappear, those users disappear as well.
Despite what the summary says, I severely doubt that hits on old feeds are any kind of
Re: (Score:1)
I've had slashdot complain to me, which was a surprise. (By the way, it's short-sighted, not short-sided, unless you're playing hockey or something.)
Re: (Score:2)
Re: (Score:1)
My guess is it was a NAT issue, as the cheeseball ISP here NATs pretty much the entire country.
Re: (Score:2)
... yet. If there's anyone actively reading the feeds, then put up a 310 redirect for a while. Determine a good interval that will catch most of the readers-- a week, a month, six months? Look at your logs. After that, anyone who will switch has. Change over to a 410. Anyone who hasn't switched probably will not, or is a bot who needs to be told to stop.
Re: (Score:2)
There are plenty of people like me, who aggregate news feeds on our sites. I'm pulling a few hundred good news sources, and making them available to my readers. I don't know how the software handles 310 or 404's. I started using it, because it does a good job of reading the feeds, parsing them, and dumping them into a databse.
On occasion, I go through and houseclean. Some sites disappear. A few mainstream papers have gone out of business lately. Some change their RSS URL,
Re:301 (Score:4, Funny)
Re: (Score:1)
Something simple (Score:5, Insightful)
The most appreciative thing you could do for the preservation of history is to place static simple RSS files at those addresses that include a link to your archives for the event.
IMarv
Re: (Score:1)
Re: (Score:1)
I heard that the decline of the brain which starts at 27 [slashdot.org] makes people occasionally comment on the wrong articles.
Re: (Score:2)
I'm right there with you. The RV is half way through renovation. I work because it pays. I only do what I love in my spare time. Right now, I'm cloning virtual drives, so I can't do much until they're done, so it makes for good Slashdot time. When I get home, I'm going to work on my own web site, and talk to my friends on the phone. This weekend, I hope to have some time to work on my RV. :)
When the time is right, I hope to go on a road trip, decide a spot is right, buy a
Just cut it off (Score:2)
RTFRFC (Score:4, Informative)
RFC 2616 is the one to read. It specifies a "410 Gone" for resources that are gone for good.
Re: (Score:1)
I doubt it (Score:4, Informative)
Newsreaders are amazingly stupid. Google Reader, for example, will ignore both your "301 Permanent Redirect" and ignore your Atom link="self" [atomenabled.org] as well. Something tells me it will ignore your "410 gone" too. You should test what major newsreaders (MyYahoo, Bloglines, Google Reader) do when you deep-six a feed with a 410.
Even still, 410 is stupid. A 301 redirect to something generic would be better, even if the readers are too brain dead to get the message. 410="lost traffic".
Text message pointer? (Score:1)
I know that it will still be a hit, but a lightweight one, and more polite than a 404.
If I may.... (Score:2, Informative)
If so, you can either archive the feeds on a web page (of only web page versions) and charge for "back issues" to cover the costs.
Still getting requests? Is it folks who are emailing yo explicitly for the feed or you're seeing traffic on those feeds in you logs?
If it's just traffic, maybe it's just some default setting on someone's phone program or something? In other words, those requests aren't "real".
Either way, I say archive them and if they really want to see
Dear ask slashdot: (Score:1, Funny)
Re: (Score:1)
See things positive! "Coward" is a beautiful last name.
410 gone (Score:5, Informative)
use a 410 Gone more informative than 404
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11 [w3.org]
HTTP 410, not 404 (Score:4, Informative)
The resource isn't "Not Found", it's "Gone". HTTP 404 is inappropriate in this instance.
Likely the best solution will be to ensure that people are notified first. If you're receiving a large number of hits, replace the content with a single RSS item that has a guaranteed-unique guid for every single request (say, based on the request time). This way, with each request, people will receive a "new" item that will display as unread, reminding them to unsubscribe from that particular feed.
After some amount of time, start returning HTTP 410 for all requests.
HTTP 410 Gone, possibly with archives elsewhere (Score:3, Insightful)
HTTP 410 is better than HTTP 404 in this case; Uncle Google and the like have a better understanding of what it means.
If you have a version of the feed that covers the whole event from start to finish, you might also want to offer a static version of that for download as an archive (but if you do this, put it on a different URL from where the feed used to be). This isn't strictly necessary, but I can see scenarios where people might appreciate being able to get at the feed's contents again.
Aggregate? (Score:3, Insightful)
Be evil (Score:4, Informative)
Just return an unending stream of crap from /dev/random on your server until it crashes the RSS reader at the other end. After a few days of this I'm sure they'll sort it out from their end :-)
Re: (Score:2)
Make sure you use /dev/urandom, /dev/random blocks when it doesn't have enough entropy.
Re: (Score:2)
Better is to craft a php file that returns a valid mp3 header and then a stream from /dev/random. I dump the contents of /dev/video to a mp3 header its' more fun and has cyclic components.
But then I think it's fun to SSH tunnel to the VoIP phone system and cat files at the /dev/paging interface.
reuse? (Score:2)
I may have missed something. But if its seasonal, why create a new feed for each event?
Why not reuse the same feed over and over again each year? If you are getting alot of old requests, its possible that one of your old feeds is featured someplace or getting good search results. (not to mention all the old subscribers that already had it). Everyone that was involved in the old event would automaticly get the new content.
Think about it
Re: (Score:3, Insightful)
He mentioned a couple of thousand old feeds. Either he's been serving feeds since the days when RSS was chiseled by hand into stone tablets, or he has distinct feeds which run concurrently - in which case reusing isn't going to work.
Re: (Score:1)
But what's the point of creating one-off RSS feeds by time/date? Doesn't that sort of defeat the whole purpose of RSS? Temporal updating is built in!
Instead of having one feed for "Online Competition Foo:2008" and another for "Online Competition Foo:2009", why not simply "Online Competition Foo" and update it with new info/articles as they arrive for each annual instance?
I mean that's supposed
Imagine the Olympics (Score:4, Interesting)
You've got one feed for updates in each sport or event.
Imagine horse racing. You are making updates to racing scores throughout the day. Gamblers are monitoring each race. How would you structure this king of thing? One feed per day? One per season?
What about comments? You've got a feed per thread/story. When the story is closed for comments, what do you do with the feed?
What about auctions? You've got an auction and every bid triggers a new entry on your RSS feed. When the auction is over, what do you do with the feed?
Re: (Score:1)
Imagine horse racing. You are making updates to racing scores throughout the day. Gamblers are monitoring each race. How would you structure this king of thing? One feed per day? One per season?
One feed per track/site. As new results come in (e.g as races finish) new entries added to the feed.
What about comments? You've got a feed per thread/story. When the story is closed for comments, what do you do with the feed?
Either nothing (as a story gets stale it's going to get further and further down the list, most RSS readers won't present old content multiple times, or only present the latest X items. Or alternatively, you could have the story removed from the feed once it is closed. This could be quite easily automated.
What about auctions? You've got an auction and every bid triggers a new entry on your RSS feed. When the auction is over, what do you do with the feed?
Well, auctions might be a problem to handle in RSS, but RSS feeds seem ill-suited to this use. I'm s
Do it the Eskimaux Way (Score:2, Funny)
Put the feeds on an ice floe.
Disposable DNS record (Score:1, Interesting)
Use a "disposable" DNS record e.g. eventname.rss.domain.com
When time comes to bin the feed you can either alter the record to a server that just serves 404's or remove the DNS entry completely. Useful for feeds getting lots of hits/second as removing the record drops all the useless traffic from your site.
Discontinued Notice Increased Traffic (Score:5, Informative)
My company has had the same issue. We just wrote out a single item feed that explained that the feed had been discontinued and provided a link to the homepage. We wrote this feed over all the feeds to be taken down.
Interestingly, in the weeks following this action, our homepage views spiked followed shortly after by increased hits on more active feeds.
Re: (Score:3, Insightful)
Think about it before hand (Score:2)
Your best bet is to think about it beforehand. Assign each feed a unique DNS hostname. When it's time to retire the feed, change the DNS hostname to point to 127.0.0.1 with a TTL of 30 days.
That won't help you with the feeds you've already created but it'll help you with the ones you're creating now.
Why? (Score:2)
It would be bad SEO and hitting up 127.0.0.1 will probably confuse some newsreaders.
Worse, now you've got to either maintain a buttload of random noise in your zone file, or you have to have a wildcard entry pointing to a wildcarded virtual host and maintain a buttload of feeds in a database. Either way, you've added a bit of complexity.
Re: (Score:2)
It would be bad SEO
I'll have to take your word for that.
hitting up 127.0.0.1 will probably confuse some newsreaders.
You're hitting my server years after you should reasonably have stopped requesting the feed AND after I've done reasonable things like sending a message through the feed alerting to its termination and sending a 404 for a while. I'm supposed to care that with the final turndown your newsreader flakes out?
Worse, you've added a bit of complexity.
That's a cop-out. If the DNS hostname is the same
Yes (Score:5, Informative)
Yes. Because if the newsreader flakes out, the *person behind the computer* might blame you and never subscribe to your feed again. We are dealing with people here, not machines. Were are not here to seek spite on stupid newsreaders or "lusers who dont unsubscribe". This is business. Petty bullshit only wastes time and loses money.
Even if the newsreader was well behaved and did the right thing, you've just stupidly removed your brand from being thrown in their face every time they opened their reader. If you've got like 10,000 subscribers sitting on a bunch of year-old feeds and you cruelly dump them with a 410/404, that is lost mindshare and lost traffic.
The best solutions are the ones that encourage feed users to keep using your services by nudging them to your new stuff.
Bottom line is, it is easier to keep existing customers than it is to get new ones. Think about it.
Re: (Score:2)
that is lost mindshare and lost traffic.
I'll buy that.
Now, what do you do about the aggregators and the folks who are receiving your feed but not viewing it, burning your bandwidth at no benefit to you? The ones who after 3 months of hourly advertisements are still receiving that feed?
Re: (Score:2)
.
Why are you assuming he wants to keep these people?
What if he wants them to just fuck off?
Well (Score:3, Insightful)
This is Slashdot, and I guess some here view the world in some kind of mechanical way free from the constraints of the real world. So yeah, if he wants to tell them to fuck off and feed their newsreader streaming crap from /dev/urandom, by all means do so. But that is a child's thing to do. Adults view the world different.
A mature adult would see that the cost of bandwidth is minimal and the untapped potential in all those people sitting on ancient feeds outweigh the potential costs. A mature adult who
Re: (Score:3, Insightful)
Interesting that you associate blind commercialism with maturity and bandwidth that has no evidence of human attention with "untapped potential". Sometimes the bandwidth use is just the result of automated feedgrabbers who, like you, believe bandwidth is so cheap as to be free at any magnitude and so continue to download anything that is available in vague hope of hitting paydirt instead of using analytics to guide them to efficient use of capital.
The article made it clear the feeds have been dead, any pos
Re: (Score:2, Informative)
Your best bet for solving the problem you currently have is to have done something different in the past?
That's probably the least helpful advice ever :)
Redirect to a 'new events' feed (Score:3, Insightful)
MacCentral and MacSlash feeds on /. (Score:2)
How about the ones on /. home page? They still show old stories for MacCentral and MacSlash. :(
Don't put a target on yourself (Score:2)
It seems to me that the best practice would be to have a feed for each event that's re-used annually, rather than a feed for each event each year. Presumably folks interested in a given event in 2008 remain so in 2009.
change the feed (Score:2)
Just pump 100 posts an hour with viagra/watches/penny-stock spam links into those feeds, they'll either get unsubscribed immediately or pull you a huge profit .. it could be fun finding out which ..
[no, really don't do this]
Don't call fictional traffic traffic (Score:2, Informative)
Sounds like you would benefit from a site level "current events" feed, that always has the most recent events. Then a per-event feed that, after the event, changes to "thank you for coming, see more at "current events". And that feed expires 3 months after the event.
If people are collecting your old feeds, but not getting value from them, is that actually valuable traffic? Are you interested in getting the word out, or counting fictional traffic?
You can also rename the feed to archive old information. R