Is There Demand For A Better Usenet Search Engine? 106
Anonymous Employee writes: "I was asked for a feasibility analysis to provide high-quality searching in a large Usenet archive (all expect binary/porn groups and several years worth of archives). This is similar to what Dejanews wanted to provide before they re-branded to Deja last year. Do you think there is a need for this or is high-quality Web searching + Usenet browsing meeting your everyday needs in terms of information retrieval? If not, do the existing Usenet search interfaces suffice (Deja, one year worth of archives, not-so-good search interface - Remarq, three months worth of archives, okay search interface)? ...and also, is real-time indexing (i.e., you can search for an article 'very soon' after it has been posted) important?" In light of Deja's recent faux pas, I think this question is rather timely, and I have to admit, I wouldn't mind the ability to search Usenet posts older than one year.
There is definitely a need for searching (Score:1)
dogpile dude. (Score:4)
it's pretty much the whole freakin thing :)
-Jon
Yes (Score:1)
Oh yeh! (Score:2)
Re:Napster and Google sued (Score:1)
Deja has dropped the ball (Score:5)
In the last few months, the quality of the results that I'm turning up has decreased markedly. Deja has decided to shelve all their 1995-1999 Usenet archives and concentrate on just the newer stuff, apparently because that older traffic only accounts for 10% or so of their bandwidth.
WHAT? Of course it does! There are enough people using Deja as their Usenet client for this to be obvious. The 10% or so of their traffic that was a result of the 1995-1999 archives was th result of hundreds of thousands of other people like me searching and finding answers.
Deja has made a mistake in alienating the audience that made them one of the most visited sites on the web. For this, I predict that Deja will either fold of massively re-organize within the next year.
They screwed us over and broke a trust. You can't regain THAT in an IPO.
Re:Deja has dropped the ball (Score:2)
S/N too low. (Score:1)
Current state of Deja's archives... (Score:5)
----
Greetings, Recently we moved the Deja.com servers to a new facility in order to provide greater reliability and performance. The move is now complete and we thank you for your patience.
Please note that currently our Usenet Discussion Service only retrieves messages from the past year (back through June 1999). As announced, we are reconfiguring the service that provides messages posted more than 1 year ago in order to provide greater reliability and performance. This will take some time though, possibly a few months. Have no fear: We're committed to bringing these messages back online as soon as possible.
-----
So I would wait for a few more weeks, and see if the situation improves.
Be explicit (Score:1)
As regards 70% disk / 10% bandwidth, they do have to draw the line somewhere, but you can't go with just what makes the most money. I'd be curious to see if the cost of bandwidth and storage outweighs revenue from advertising, and other intangible revenue such as name recognition.
You betcha there is (Score:1)
Also, fast indexing would be a real bonus - so that if you look for a comment on recently-released software, for example - you won't get two monthes old data. But at least decent search and archival is necessary. Deja desperately needs strong competitor.
I'd pay for a subscription (Score:3)
I have several clients who have almost completely abandoned deja because the quality has disappeared. They've asked me how hard (i.e. how much $$$) it would be to set up a similar service for them internally. I give them the cost estimates for a full time usenet+searchengine system admin and a pair of good machines. Then they ask if there is a company out there who would do the same thing for less money than the US$100k/year it would cost to do it themselves.
It would be especially nice to see corporate accounts set up as well, so any employee in a company could do high quality searches.
My own opinions [slashdot.org] on deja are pretty vituperous right now. If you could buy a copy of their old archives and provide a better service than those losers, you'd have a fairly large audience. Try doing what dejanews did when they started, going around to usenet admins and asking for copies of backup tapes. Be prepared to get old DC-150 carts and 9 track reel to reel and many other esoteric formats. Could be a fun project
the AC
Something is needed out there (Score:1)
The old Dejanews site was a great way to find information, and was displayed in an easy to navigate format. The new layout is awful.(Not to mention their stupid policy of advertising placement -- I post my messages to Usenet, NOT to Deja.)
I don't know what the overall reaction to a site like this would be, but definitely count me in. jh
Usenet is nearly dead, if it is not already (Score:1)
Now, sadly, the internet is a corperate money making buzzword. Companys try to reach any audience with advertising they can, and the internet is a cheep way of reaching millions of potential customers. However in the need and greed of modern sociaty. People wanting to make a quick buck outnumber the people wanting a nice place for discussion, and it is easy for the money-grubbing people to write one message to hundereds or thousands of groups. Thus many many groups are significantly more spam than relavant posts. This drives off the people who would otherwise have been frequent posters, makes good posts hard to find and generaly makes the experience to un-enjoyable for a large part of the public to continue to have any intrest in. True, fringe groups will continue for a decade or more, but sooner or later the nntp protocal will become to much a bandwidth hog (thousands of spam email messages can do that) and most servers will close down.
Alas poor usenet, I new it well
Re:S/N too low. (Score:1)
Usenet's strength is providing a single UI (your choice of newsreader) to thousands of discussion topics. Plus, given its academic roots, I guess that it has a higher level of discourse than some similar multi-topic setups. (Though I know some people who cut their teeth on AOL with its user moderation find Usenet a little more noisy than their liking. Personally I think that a greater "S" could make up for the greater "N" but I'm not going to swap communities or start paying AOL to find out.
Here is how to win Mac friends (Score:2)
What
A USENET search engine isn't profitable (Score:2)
To much of the public, the net is becoming something like TV. Not many people on the street know what a newsgroup is (if you don't believe this, you don't know many people on the street - try asking random people, you'll be surprised.)
Given this scenario, it's not likely that a usenet search engine will last for very long. People who want to use USENET will actually use it. Now that's a surprise.
w/m
Privacy concerns & tech NGs (Score:3)
Just as food for thought, there are also some privacy issues here. You have to ask yourself: do you really want a decade or two of your scribblings to be instantly available and indexable and searchable by anyone on the planet? Think about it - every immature flame, every embarrasing post, every moment you'd love to live down, now showcased and painfully easy to find by someone with a couple of minutes and a computer... I'm kind of glad that Slashdot "forgets" or de-indexes my comments after a few weeks. There are a lot that I'd just love to bury and in effect have as soon as they exit my user info page and leave the search index. Now imagine them staying with you for years, even decades.
And it can get worse than simple embarassment. I know for a definite fact of one case where two guys were engaged in a long-standing flamefest in a NG. Guy 1 went on dejanews.com to look at what else the other guy (Guy 2) was posting... and found some two-year old backposts to a cancer support group because guy 2 was battling some form of cancer. Guy 1 brought that up in his next flame and really just humiliated guy 2 in front of hundreds of people. Until deja killed their backlog, you could still find both those posts, and hundreds more just like it. Imagine trying to live that down.
It's incidents like that that really cause me to agree with privacy advocates about the danger the Internet poses. Never in human history has it been as easy to delve into a person's past as it is now even without a superorganized listing of their thoughts and opinions of everything they felt compelled to write about for years into the past. Such a complete archive really would pose a lot of problems for many people (imagine just a 10 year log of alt.support.cancer
With that in mind, like I say stick to the tech newsgroups, and you'll run into far fewer problems.
--
What is on Usenet that is worth searching? (Score:1)
nt (Score:2)
yes.
--
Can the Usenet archive be saved? (Score:2)
The Dejanews Usenet archive was one of the best resources on the Internet. I'd always check there before doing searches on company sites. The recent decrease in the archive database has reduced the usefulness of the service dramatically.
I suppose the important question should be: Is the old Usenet archive worth preserving (for general use, or even as an historical record (it might appear that most of it is useless waffle, but who know's what people will think in 100 years))?
If the answer to the above is "yes", then how can the archive be saved? Leaving it in the hands of a single company (Deja) means it's vulnerable to any silly decisions that the comapny makes.
Perhaps a better solution would be a huge distributed database, where sites archive particular groups for a particular time (eg. some of the big Linux companies could "sponsor" the comp.os.linux.* newsgroups for the dates between 1995 and 1998). These could then be mirrored by other sites with the same interests.
The two negative points I can think of (aside from the nightmare administrative aspects) is 1) what would the sponsor get out of this, and 2) just how big would the archive be?
It is possible that a Gnutella-like system could evolve where people could search for archives, with a set of "root servers" providing searching facilities. With 100GB disks becoming available, the possibility of smaller newsgroups being archived becomes a possibility.
All we need then is to persuade Deja to reimplement the full database (which I believe the eventually intend to do), and then get a tool to archive interesting articles. Anyone out there think they have the skills to write a "deja extractor"?
When did Ask Slashdot become... (Score:1)
Re:Current state of Deja's archives... (Score:1)
Re:Deja has dropped the ball (Score:3)
It doesn't mean that only 10% of their audience does old searches and 90% does recent searches. It means that 90% of their audience occasionally need to do older searches.
I would say that 10%-20% of my searches are for older information. Especially for hints that someone else has already solved a problem I am up against. Mostly I search comp.dcom.* and related alt groups. Quite often I am working on systems installed 10 or 20 years ago, and many people have documented what they have done in similar situations. A solution posted 5 years ago is still relevant to a problem today.
So when they cut out 10% of their traffic, why did their total hits decrease by 60%? Because they pissed off 60% of their user base enough to abandon the system. Thats a pretty bad move for any company. Its why they have had to lay off 10%-20% of their workforce, and abandon their IPO.
With any luck, a new company will buy a copy of deja's archives in the bankruptcy sale and do something useful with it.
the AC
Distributed message-based file system anyone? (Score:2)
The biggest risk with such an automatic scheme is that some data would eventually timeout because no one requested it anymore. I guess these messages would start to be treated like endangered species. Maybe we could just send them out into a deep-space, time-delay file system to save them.
What this butthead thinks. (Score:2)
However, the Usenet junkie in me is kicking and screaming over this. My Usenet service from my ISP drops posts off after a couple weeks. And even the group specific server that my group's moderation pool uses drops stuff after a couple months because of the sheer quantitiy (and because it's only purpose is support of the moderation bot).
Personally I'd pay money for and/or put up with a reasonable amount of banner ads to be able to search back through all the content.
Chas - The one, the only.
THANK GOD!!!
Dealing with the current Deja situation (Score:2)
A few months ago, Deja made an announcement [deja.com] about the site move. According to the accouncement, which has not been updated since its original release, the old messages would temporarily be taken down, but we should "have no fear: [Deja is] committed to bringing these messages back online as soon as possible.
In the meantime, Deja has been transformed into a mere free Web-based Usenet server that happens to have unusually long retention, but no binaries access.
It has been a couple of months since then. Last month, Deja announced that the move was "complete"; however, most of the old posts are still nowhere to be found. There was an interesting Usenet discussion [deja.com] on the state of things, which included at least one thoughtful post [deja.com] as well as possibly a little light at the end of the tunnel [deja.com].
Perhaps not all is lost. When (if) the Deja archive ever comes back in its entirety, it will still be the best Usenet archive around, hands down.
I disagree with the Slashdot article's claim that Deja has a merely "okay search interface". As long as one uses the Deja Classic Power Search [deja.com], Deja has one of the cleanest interfaces around, with extremely flexible and powerful query options.
One would be hard-pressed to come up with something better at this point. Even if one were able to cook up a better interface with even better query features, where would the content come from? Who has been archiving Usenet all these years other than Deja, Remarq, and perhaps a few other little-known entities? [remarq.com]
I daresay that none of the current archive holders would be willing to grant archive access without considerable compensation. Unfortunately, one would have no choice; it's a little too late to start archiving the old stuff now!
All in all, I would probably be in favor of just trying to get Deja back up in its full glory; this would be so much easier than starting from scratch. Perhaps all Deja needs is to hear (from thousands of concerned Slashdot readers) that their "old" archive is their most valuable resource, and should thus be given the attention that it deserves. I personally consider the "old" archive so valuable that I would be willing to pay a subscription fee to access it; I'm sure I'm not alone in this.
So shall we all write Deja now, and let them know what we think?
--
Message boards/forums are taking over. (Score:1)
Hits pays the bills, so people are moving to message forums.
What about Ultimate Bulletin Board exporting messages to newservers?
Then its the best of both worlds.
Only newsgroups I use anymore are hardware vendors.
I'm on a dozen webbased message forums, all those banner ads.
-Brook Harty
[I have the Enemy Flag heading back to our base. Clear the mines from our flag...]
Just get rid of alt.* -- no problem (Score:3)
My news server has just the big 8 and only the alt groups that users request. With a 15 gig news spool, I only have to expire articles after two months.
Doesn't take a math wiz to extrapolate that to see how mucn disk space a years worth of REAL usenet newsgroups would hold.
They should have never trashed 1995-99 without notice. 95 was when the net started to explode and removing that removed history that can never be recreated.
(Then again, I'm glad some of my old posts finally went away. x-no-archive works, but since everyone these days just quotes entire articles when replying with one line at the top, x-no-archive was a bit useless anyway...)
what other alternatives? (Score:1)
I have been using DEjA [deja.com] for a while and in the past year or so, the quality exponentially went down.
So now I am looking for alternatives? A search [google.com] on google [google.com] doesn't reveal much.
Can people name any similar services that exists? A poster mentioned dogpile [dogpile.com]. ANy others?
I would really like to see GOOGLE getting into this.
Re:I'd pay for a subscription (Score:1)
What's a pitty for maybe the most usefull tool on the net along with (Google and Slashdot
USENET Searching... (Score:1)
Deja probably can't keep everything that is on USENET (anyone have any idea the total number of newsgroups there are?) and for them to make any money, the ability to search an increasingly large archive becomes unfeasable.
I used to love USENET, but now it's nothing more than a tool for the would be marketers to send unsolicited e-mail and to shameless promote crap. It basically has lost a lot of its appeal lately.
One good way to get by the search problem is to distribute the content of USENET to separate archives based upon their hiearchy, so that comp.os.linux gets archived by VALinux (or similar) and that all you need is a client tool that would search the indexes of these separate archives.
Unfortunately, it is somewhat late to actually implement this. If only this was a requirement far earlier in the process of creating newsgroups...
old goodies (Score:1)
During that period, in the sci. and comp. groups, there was a lot of good information as it was mainly university people exchanging info. I remember back then reading about people archiving the net to glass disks and such. Where did all the old net go?
Re: Dogpile search limits? (Score:2)
It's an NNTP problem (Score:2)
It's too easy to spam, even semi-private NNTP servers, like Netscape/Mozilla server are hit with spam enough to piss you off. All the serious development takes place on mailing lists and that can be a lot of email. (and it's not spam proof either, the kernel list is spammed at least once a week)
We need a public key infrastrucutre added to NNTP such that to post you need to use a key that has been submitted to a server, at least for technical forums. Call it automatic moderation, once you're trusted then you can do whatever, spam causes your trust to be yanked. This has the added benefit of building up the key web.
Services like Deja provide a useful service, once you weed the crap out. Most of what they archive is junk. And there is so much of it that they can't keep it all online. There have been some critical usenet threads that need to be archived for easy access. There are still important threads and messages posted.
Re:Privacy concerns & tech NGs (Score:1)
and indexable and searchable by anyone on the planet?
Check. Go search on dvdeug or dstarner98 on Google, and you will find all my scribblings instantly available. Including some stuff I'd just as well forget. It's life - think before you post. Use x-no-archive wisely and there won't be much of a problem.
It's nothing new - lots of people are embarressed by things they said decades ago. If you plan to go into politics, then start censoring yourself now. Otherwise, you should still think before you speak, but it's probably not going to matter.
If the contents of the web are archived... (Score:2)
Yes; do a better interface. (Score:1)
I have found that at my current place of employment, where there is no news feed, it is considerably harder to get work done without that resource. I used to ssh into my cable modem box to browse and post from there, but now I've moved to an apartment without cable. So I'm negotiating with our sysadmins to try to get a newserver set up.
The dogpile interface is definitely better than the old deja. In particular, each "hit" gives you the entire thread, instead of spreading a thread out amoung individual article links.
I think that many of the complaints about usenet being swamped with spam and useless are from people who are not familiar with better news readers. You can filter a lot of that stuff.
One thing I would definitely like is a usenet interface to slashdot. If it was read-only and you had to log in and go through the web interface that would be fine.
Re:Dealing with the current Deja situation (Score:2)
What I think would be ideal right now is if deja split into two - they could have a dedicated usenet searching facility and archive, and another company for all the crap they've been doing now. They could work together very nicely, but DON'T let their web hosting or whatever get in the way the most important thing - Usenet archives.
And why oh why oh why did they change their name to deja!?!?!? Dejanews was a cute pun... I liked it! Perhaps if they DO split up a bit, the news division can inherit the name. Then we could have dejamail, dejaweb, dejashop, and dejanews.
Does anyone even USE the non-usenet services they provide?
Re:What we need is spam-free Usenet (Score:1)
I dunno -- it would be a mess, but would sure solve some problems. Of course, kiss privacy and anonymity goodbye...
Search in the works.. (Score:3)
What I want in a USENET search engine... (Score:2)
You can connect to this server using any USENET newsreader program, read any article you like that way (but not post, of course). That would let people use their favored newsreading environment--which already has functions for threading and searching individual messages--to go in and read whatever they want without having to screw around with USENET search engines' moronic interfaces (of which I have never yet found one that worked decently).
Granted, this would make it harder to do global searches across multiple years...but I'd gladly sacrifice that in exchange for an interface more useful to me in searches of smaller scope.
--
Yes - I want it! (Score:2)
Type in a search phrase and pull up all matching web pages and usenet articles which matched.
This was EXTREMELY valuable, especially if you were hunting down the answer to a question or problem. One search did it all, and the breadth of knowledge in the web + usenet could not be beat.
AltaVista modified their service so that they no longer do usenet searches, about a year ago, I think it was. I couldn't believe that they would take out that excellent feature, and I wrote them and complained, but of course they didn't care (for the record, you can still do some kind of usenet search, but its not on articles it just returns pointers to "relevent newsgroups").
I never use AltaVista anymore. I use Google, which doesn't do usenet either (although I suggested it to them and they said they would be considering it in the future), but I'll be damned if I use AltaVista again.
Re:What is on Usenet that is worth searching? (Score:2)
I am a small-time computer programmer (for my own enjoyment) and local technical expert (called upon for practically all Windows installations). I fiddle with Linux a lot - my computer has a 17.2G drive for bloated software like Windows 2000, and a 4.3G hard drive devoted entirely to Linux distros (PC PLUS in the UK is great with this [Mandrake 7.1 this month.]) I also do a bit of amateur spamhunting (you'll see me a lot on news.admin.net-abuse.email.)
One of my most used tools is Deja.com/usenet. I have it on my slashboxes. With deja.com, I can immediately search out whether this pernicious "Find Out About Any Poor Shmuck Fast Now" spam (I recieve it every week or so) has been posted to N.A.N-A.S [admin.net-....sightings] yet. Also, if I have technical problems I need to solve, I call up Deja first. I don't search AltaVista, partly because I don't like AltaVista (I prefer Excite [excite.com] or Lycos [lycos.com]) and because all Web search engines, especially when faced with computer support queries, don't successfully find what I want. Also, several big pr0n and other dubious sites will definitely use support queries in META tags to drive you off the route - I've had this happen to me before.
Therefore, services like Deja help me to find out about new drivers, or how to make my Sony tape drive work - they help solve problems. This is the entire purpose of Deja - not to make up a huge Britney Spears fan collection, but for technical information. And yes, there is a demand for a better engine (for a start, one that you don't have to click three different time-consuming links to move into a thread) - but at the moment, Deja and remarQ are all we've got. And thankfully, we have them.
Re:Deja has dropped the ball (Score:1)
(different users are not equally active)
Saying that they pissed off a lot of users (who together accounted for 60% of the hits) is probably more accurate...
I just couldn't stop myself from saying this.. sorry!
Re:dogpile dude. (Score:1)
might be a demand but who's going to pay (Score:1)
Instead of asking "is there a demand" ask
is there a demand if each search costs the user $0.01
or
is there a demand if each results page has three banner ads (and is there a demand from advertisers for this)
So, what's your projected cost per search and how do you indend to cover it?
Re:Privacy concerns & tech NGs (Score:2)
Remember...engage brain before posting to USENET. If you don't, it's your own fault and you can't expect the rest of the world to look out for you.
--
Re:I'd pay for a subscription (Score:2)
But then again, a case could be made that Deja(news) didn't promote it at all, hence it was doomed to failure from the start. But what do I know, I've only worked at Deja.com for less than a year.
Indispensible (Score:1)
I pretty much gave up reading Usenet on a regular basis probably about 3-4 years ago due to the volume and noise. Dejanews at least provided me with a way of looking for specific info when I needed it. Granted, for that kind of use a shorter, year length history is usually fine. However, the much longer history has proved indispensible on a number of occasions when dealing with older systems. One particular project I worked on was upgrading the version of Tcl/Tk from one of 5 years ago (when the project's software & hardware were frozen) to the current version. There were a number of major changes in the language & libraries in that time, but instead of having to find all the difficult migration issues (some of which the docs didn't mention) myself, I did a bit of searching on Dejanews and saved weeks of time and considerable fustration. One is rarely the first to have a problem -- and interactive knowledgebase of Usenet often provides the solution. I'm really disappointed by deja cutting back this resource and would love to see a more complete & better archive out there!
Also, aside from the practical aspects, the contents of the archive are a valuable bit of .net history & group memory -- it'd be a shame to lose it...
solution: distributed search? (Score:1)
All we need is an open protocol for communication between newsreaders, for passing around searches and results, and enough readers to support this. Searches would be independent of the moods and financial motives of the owners of a single archive and consistency could be assured by the redundandcy of hundreds of people having the same article in their archives.
You're Not Kidding: Re:Deja has dropped the ball (Score:2)
Not only does the Internet need a better Usenet search engine, it needs an entire Usenet frontend. More and more ISPs are either not offering news service, or simply pointing people to supernews.com. To me, that's a waste of time, as supernews is often overloaded, and the traditional Usenet interface isn't exactly as user-friendly as it once was, what with it being mostly spam and porn these days
I'd like to see a site that I can not only use to search, but also to post and reply to Usenet articles. I could give out a lot of help and free advice if only I didn't have to fire up a news reader. Don't get me wrong, because I love command-line interfaces, but just being able to bring up the relevant information is much more helpful than having to look through a buch of posts about how I should buy these printers or use these domain registration services.
Brad Johnson
--We are the Music Makers, and we
are the Dreamers of Dreams
A few months of archive is a dangerous thing (Score:2)
I doubt you'll see ANY service carry more than a year's worth of articles and for many corporate lawyers a year may be too long. One day someone IS going to sue these services for copyright violation and the Usenet services live in fear of that day.
The best way to search Dejanews (Score:3)
Unfortunately Jeremy doesn't have his own back archives
----
Re:A USENET search engine isn't profitable (Score:1)
Brewster Kahle's archive project (Score:1)
Deja was useful and still is, but they seemed to decide supporting USENET was not where the money is. No surprise, USENET is largely abandoned by the software development community and money community now. The top newsreader is from microsoft!
Re:You're Not Kidding: Re:Deja has dropped the bal (Score:1)
Need new usenet search? Altavista does. (Score:1)
But even for web searches, I'm coming to the conclusion that Altavista is just sucky. Firstly, it's way out of date, half the links are broken and it's still indexing my homepage as having content that was changed over two months ago. But mostly I get annoyed that the search page refreshes itself every 5 minutes. Presumably this is done in an effort to fake ad impressions but it is annoying to have a search page disappear while you're trying to read it (especially since at work, I frequently have to disable the proxy for development purposes so it refreshes to an unavailable page) but it doubly sucks at home where I have a dialup connection and leaving Altavista on screen means my phone line is always busy.
Sorry for the rant, I just had to say it. Suffice it to say that I'm looking around for a decent alternative (I'm starting to use Google from Monday I think)
Rich
Open Source Project (Score:1)
We definitely need a good Usenet site for searching and timely browsing/retrieval. I have used deja religiously. When my company blocked it as explicit content I freaked and sent an e-mail to everyone in the company I knew. A slew of people backed me up and those are only the people who cc'd me. They wrote long descriptions of why the site was so valuable, ordering our firewall group to return it's access. Took about a week but we got it back.
We must start an open source project to provide this functionality. It is our duty to expose this wonderful free forum to the general internet community. The problem is the technical limitations. Such a program(probably distributed) must have these characteristics:
1) Fast Searching - this is rather tricky. Dejanews was, or still is, capable of searching the full text of 100 million articles VERY quickly. I don't know how they did it but I would imagine this would require custom database like functionality that could preprocess all articles indexing each word such that all the work was done in advance. This would certainly be the most difficult part of the project.
2) Quicker Posting/Retrieval Response Time - As it is deja does not respond too quickly to posts. It takes several hours I believe before a message is searchable on the site. I don't know about browsing particular groups but it would have to be on the order of minutes. This should be possible as MS Outlook News Reader is quite fast at this and we all know it has nothing to do with MS Software :~)
3) Ability to populate database with mailing lists. It would be a very cool feature if one could add an arbitrary mailing list such as that of an open source project to the indexable archive.
Also I believe someone could make a viable business out of it. A Usenet search site could be very profitable. I'm surprised no one has caught on to this. Deja dropped the ball and hid the "discussions" pages behind it's new facade. Odd. I can only imagine how many hits a site like that would get. If the USENET search capability was presented as the premiere offering of a site it might not seem so obscure to average users. They would perhaps discover Usenet as the great alternative source of information that it is. Usenet would be the latest and greatest thing! With all the "instant messaging" going on it would not seem so foreign to people. Also I would think some people would be willing to pay for this service. I would be willing to pay a small fee. Say $20 a year?
KidSock
Re:Need new usenet search? Altavista does. (Score:1)
Massive regexp search engine needed (Score:1)
Exceptations (Score:2)
Of course we all expect binary/porn groups and several years worth of archives...
What? Oh, you meant except? Damn...
Another Usenet archive (Score:1)
Eggs in baskets (Score:1)
In a world where technology companies companies can sink overnight, are we to leave this to Deja and others? Is there no public repository where these Usenet articles are archived for all time? Is this publicly generated mass of information not available directly to the public?
No, the question is not whether we need access to a better search engine for Usenet but that we should have access to Usenet itself. I should be able to order a 100(or whatever) DVD set of Usenet posts to have in my own home. This is too important to leave to the whim of the market, this needs to be preserved for humanity for all time.
Maybe some of the people who have copyright on Usenet posts could bring some pressure to bear on the archive companies to make something like this available.
Rich
Re:Need new usenet search? Altavista does. (Score:1)
Rich
Re:A few months of archive is a dangerous thing (Score:2)
That's what X-NO-ARCHIVE: YES is for. If you don't want your posting archived, add that to your header. That can be both good and bad, by the way. I've been noticing that more and more people automatically decline to have their Usenet postings archived, which can mean that genuinely useful advice or commentary disappears forever. I wish that this option would be used more selectively.
The Internet Archive Project has usenet 1996-1998 (Score:2)
The Internet Archive Project is the project attempting to archive the entire web and related internet contents as a matter of public record. They currently have around 15 terabytes in the archive.
Push them to resume archiving of Usenet, and to get their old stuff online from the tapes. This is HISTORY, people! Historians 50-100 years from now will be DIEING to look at this stuff, and won't be able to belive that we threw it all away, even though the cost of storing it was dropping exponentially.
I would kinda hope that my great-great-grandchildren could get to know me by reading some of my better usenet posts.
--Braddock Gaskill
Why the Web all but killed Usenet (and Dejanews) (Score:3)
That creates an effect not everyone sees. Usenet was the birthplace of hundreds, maybe thousands, of electronic communities, long before people were using "e-" as a prefix. Those communities, the people and personalities and cultures, are what made Netnews so attractive, so involving. (The current buzzword is "sticky.") Of course you're going to come back to see if your favorite netscum posted something outrageous, or if someone answered your question or replied to your answer.
Web-based discussions didn't kill Usenet, but they darned sure hurt it. Instead of one "'Net," there are tens of thousands, maybe more. I can't count the number of Web-based discussion forums I've seen. This conversation we're having right now is off in some tiny little corner instead of in a news group. There are lots of advantages to having it here
Instead of a grand city, with some wonderful neighborhoods and some seedy ones, we've got surburban sprawl.
Netnews could have survived spam. It could have survived the astonishing growth of online participants in the past five years. (It survived AOL, in many senses.) It's having a hard time suriving its current competition. Part of me is very sad to see it wither.
Ironically, the Web is both the medium in which Dejanews tried to grow, and the medium that choked off some of its best source material.
I'm saddened by Deja's dwindling support for Netnews archives. (Did they used to go back as far as 1990?) I understand why they failed to turn a profit on the business, why they've got a terabyte and a half (literally) of archived material they consider too expensive to keep online. I appreciate what they've done, and I'm glad to have what they still offer. I wish the Dejanews business had thrived; I still wish it well. --PSRC
"I'm not speaking for the company, I'm just speaking my mind." (my old Netnews
Re:A USENET search engine isn't profitable (Score:2)
The new user types their question into the search engine and out pops an answer. Without ClassicDejanews I likley would never have gotten my, printer, soundcard or joystick working under Linux. It was there and it did its job.
If people recognized the potential a USENET search engine has to help them they would flock to it.
deja's archive goes back further. (Score:3)
As of May 15, all messages posted approximately a year ago or more have become temporarily inaccessible via Deja.com. We will be taking this opportunity to reconfigure the service that provides messages posted prior to September, 1999. Therefore, these messages will not be accessible on the site for some time, possibly a few months. Have no fear: We're committed to bringing these messages back online as soon as possible. We request your patience as moving our server bed to a new facility will greatly increase our reliability and performance.
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
Oh, also, use ija: (Score:2)
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
Well... (Score:1)
Archival Monopoly (Score:1)
What? That's "Socialism?"
Fine, let Microsoft compete against AOL/Time Warner/CNN/Netscape on an equal footing -- no break up.
Nice Idea, But Not a Money Maker (Score:3)
I was really excited when the new Deja went into Beta testing of their expanded capabilities. Unfortunately, the potential was never completely developed, and now Deja has changed directions: Usenet is almost an after-thought, now.
Another example of Deja's Usenet scale-back: some of the slick graphical Usenet navigation tools have been removed. Remember the four-way arrows introduced early last year? I believe the up and down arrow would jump to the next thread. The left and right arrow allowed movement within a thread. Very handy tool. Now it's back to the old style, still effective, but not as user-freindly as the arrows.
It's a shame that Deja has moved away from Usenet, but I suppose it was inevitable. As a 5 year veteran of Usenet, a self admitted newsgroup junkie, and an unapologetic devotee to Agent, a piece of software that's seen little modification in two years, I have to admit that Usenet is not a tool that is easily mastered. Well, at least not by the majority of moderate-use Internet visitors, that is. I'm still explaining the concept to my co-workers but, for some odd reason, they seem to be intimidated by Usenet. Guess if it gets beyond point and click, homepage and favorites, most people lose interest.
To sum up, although I'd like to see a service similar to the one that you mention, I don't think it's a money maker. If it were, then Deja would be promoting, expanding, and improving their Usenet capability, rather than scaling back and minimizing it.
There is, of course, at least one alternative possibility: Deja mismanaged their upgrade, and squandered it's potential.
I don't know enough about the inner workings of the company to say one way or the other. However, I tend to think that the problem lies not with Deja, but with the nature of Usenet. Usenet is intimidating to many Internet users. For some, the concept can be difficult to grasp. Obviously, it's not as simple as the Web, and of course, the simplicity of the Web spoils many Internet users. My point is this: I don't believe Usenet, outside of the binary groups, particularly MP3 and porno, will ever attract the level of usage that the Web generates, even with tools such as you propose. And, of course, you specify that binary groups will not be implemented in the proposed service (and rightly so). So, although I'd like to see you give a favorable report, I doubt that you will. Please let us know one way or the other.
One good thing that will come of this: Deja's "Power Search" has had some of it's fangs pulled: all of those embarrasing posts I made to Usenet years ago, before I realized they could all be traced back to me, as the years go by and Deja loses Interest in archiving, they'll be that much harder to access
Message to Anonymous Employee: Yes (Score:1)
But be sure to make your stuff better than Deja and to better respect people's rights and feelings. I.e. NO editing of the messages, please, and certainly not to insert ads. As a matter of principle, I'm busy nuking all my posts from Deja because of the ad issue. The home ones are gone already, the work ones will be as soon as I get back to the office.
--
Re:Why the Web all but killed Usenet (and Dejanews (Score:2)
My ISP gets newsfeeds from several sources, and still I get maybe 30% of articles missing. Loss of articles is simply not acceptable in a modern system, when you have to compete with message boards which have no article loss at all.
Usenet needs a rebirth to position it as a viable alternative to message boards. It needs to concentrate on high traffic groups which actually benefit from worldwide mirroring.
Meanwhile, private message boards can take over the low end of the market, while providing NNTP interfaces and all the features of a real news server. The big difficulty here, in fact, is that most web hosts would not allow their users to set up a 'server', restricting the idea to the big boys, which is the opposite of the proposed low-end niche message boards should have. Perhaps something could be jury-rigged to send text via HTTP, with a client-side translator program acting as a proxy news server. (Too techy, though)
And news readers, the final point in the triangle, need to have solid and seamless support for getting groups from many servers.
Even the best web-based message board is nowhere near a good newsreader for ease of use. Sadly, the Web is seen as the only interface needed for internet applications these days.
Whew! I'm off the hook! What a relief! (Score:1)
Soon, with the help of Web discussion sites like Slashdot, maybe Usenet will just cease to exist.
Where ya gonna go? (Score:1)
Re:Deja has dropped the ball (Score:2)
Now, they are actually reducing the size of the archive! The one good thing about this, I guess, is that certain crap I posted in younger days is no longer publically available. The bad thing, is that there is also a bunch of Usenet stuff from the 1991-3 era that I'd really like to get my hands on. (Old alt.tasteless!)
The sad thing is that DejaNews was always the big guy, and because of their presence, nobody else got into the biz, or bothered put those old archives up on-line. (If I'm wrong, please tell me!) Now they are getting out of the Usenet business, we've really lost a valuble resource.
--
replication (Score:1)
Multiple copies via diverse methods is the only nearly secure approach. That way when one of the versions fails, there will probably be time for one of the copies to replicate.
The problem comes in selecting which information to preserve, since one can't save everything. Probably the best approach is for those who would back up the internet to specialize in certain areas. Remember that nobody can learn everything anymore. Well, nobody can store everything either. Other groups need to take the place of public libraries, and index the sources of information in various ways. It won't be as easy as it used to be, but consider the rate at which new information is being created.
How is archiving done and who has the archives? (Score:1)
Is there a DVD-R set that contains the entire usenet archives? usenet is distributed - is there any node that has been archiving since the start?
Thanks!
Loss of DejaNews a major blow (Score:1)
Re:Current state of Deja's archives... (Score:2)
About a year ago, they moved all of the older posts to an inobvious location; in other words, first you had to do a search in the current archive, then you could do a search in the older archive.
Then a few months ago came the announced re-organization. So far, all they have told us is the same promise that the older archives should be back online.
Deja has been stupid about this. Right now, they are -- or were -- the only people who offered this service. They could have offered the older posts as a premium service -- maybe charged to the user by charging ISPs. Instead they apparently decided this unique service was not worth offering, & have discontinued it.
That would be typical yuppie dot-com thinking. I'd like to think that Deja didn't subscribe to that kind of thinking.
Geoff
Re:****FREE PORN PICS*** (Score:1)
Re:Message to Anonymous Employee: Yes (Score:1)
How do you do this, please?
Fragmentation of usenet (Score:1)
If a new search engine would allow me to search these "private usenets" as well, I would definitely use it over deja! Especially since I can't access NNTP from work because of our firewall and well, I fully agree that deja's new focus on reviews is stupid. They should've spawned a new site for reviews.
Yup. (Score:1)
Re:Deja has dropped the ball (Score:2)
It isn't that dejanews somehow owns the exclusive rights to usenet archives. Anyone has as much rights to do what dejanews did. (Whether they actually have the right to do what they do is questioned by some groups of people). I think dejanews provided a great service, and I am utterly disappointed by their recent policy changes. However, dejanews doesn't own me anything. It's easy to critize dejanews, and suggest what they should do. But before dejanews, there weren't any public Usenet archives at all, and dejanews has been one of the few successful and original ideas on the web that hasn't been copied a thousand times over.
It's a pity that dejanews doesn't provide as much service as they used to. But it isn't dejanews fault noone else does either.
-- Abigail
Where Deja Went Wrong (Score:2)
The biggest advantage Dejanews has in dealing with all this Usenet data is its ability to sort by subject, newsgroup, date, author, and so on. One can't do this with web pages, either. With Deja, you can be really imaginative, trying to recreate in your mind how someone might have phrased a particular question or who might have been interested in a certain topic which you want information about. This is why Dogpile (which another poster mentioned), which does not offer such options, is inferior.
Deja hosts chats about obscure technical questions, breakfasts in Pittsburgh, debates about graduate schools--thousands of real communities, and opinions, categorized to the tiniest niches, so nicely sorted and searchable, and which can be captured through time. What a sociological resource, if nothing else!
Usenet is a database, and Deja provides a proper search feature for it. That's value. Great value. What Deja refuses to realize is that it could charge for the resource. I would pay $20 a month for it, easily. Especially if Deja promised to maintain it well (and of course put the old archives back online).
Deja could add even more value to it. If it had been a little more ambitious, it would have added to its Usenet database discussion forums akin to ForumOne's [forumone.com]. These are Web discussion forums: Salon, the Utne Reader, etc. Searching each one alone for a topic of interest is arduous, since each forum's population can be comparatively small and topics broad. Usenet archives are useful because they provide enough of a range for niche subjects to be covered.
ForumOne, though in concept magnificent, is practically useless because you cannot use boolean and you cannot sort by author, date, etc. But can you imagine being able to search the entire range of discussion on the web PLUS Usenet with one search engine, and being able to sort via useful database fields? It would be a treasure trove easily equal to the Web in value.
Deja could charge even more for that.
Re:A few months of archive is a dangerous thing (Score:2)
That doesn't cut it. You can't copy books and distribute the copies and get away with it because you won't copy books that have "X-NO-ARCHIVE: YES" written in crayon on the cover.
Whether Usenet archives are violating copyright or not, I don't know. (I hope they aren't, but I can see the reservations people have), but the respecting an opt-out header isn't going to cut it. One does not have to defend a copyright. If you want to legalize it with headers, only opt-in headers will do.
-- Abigail
Re:Deja has dropped the ball (Score:2)
Where's the money ? (Score:1)
that older traffic only accounts for 10% or so of their bandwidth.
The more important question is, "How much revenue does that older traffic generate ?". Deja has turned far more commercial of late, the shopping review emphasis, the embedded advert links. We, the geeks who used it for Usenet searches, just aren't a useful revenue stream to them, so they've dumped us.
Face it guys, Deja is no longer your handy geek-friendly Usenet archive. It's now a "What HomeVideoTheatre Pork-Rind-O-Rama" review site, selling dumbed-down content and adverts to the stupids.
Time for pastures new. Maybe Dogpile.
Re:Where's the money ? (Score:1)
old warez and porn. (Score:1)
Re:Deja has dropped the ball (Score:1)
And where did you hear that they dropped 60% of their traffic? This is news to me and sounds like pure conjecture (did you konw that 92% of all statistics are made up?)
And I hope someone does buy a copy of the archives in the fire sale, but they also will have to buy a couple of key people in order to get the info out of the format that it's stored in. That's part of the reason that they took those machines offline. Their current (proprietary) solution doesn't scale well.
Re:Be explicit (Score:1)
So did the cost of bandwidth and storage outweigh
revenue? I'd say definately yes. The intangibles? Don't know. measure them and I'll tell you.
Re:I'd pay for a subscription (Score:1)
It lasted less than a year, and then the company that provided it (or maybe just the department and the service) got bought by usenetservers.com.
Re:binaries archive (Score:1)
Just curious
Re:Need new usenet search? Altavista does. (Score:2)
But even for web searches, I'm coming to the conclusion that Altavista is just sucky. Firstly, it's way out of date, half the links are broken and it's still indexing my homepage as having content that was changed over two months ago. But mostly I get annoyed that the search page refreshes itself every 5 minutes.
I don't think Altavista will autorefresh if you use it in text mode [altavista.com]. It doesn't seem to send any meta-refresh tags that way, plus it's a much faster loading, easier to read interface. Text mode used to not even have banner ads, but Junkbuster takes care of those easily enough.
Re:Why the Web all but killed Usenet (and Dejanews (Score:2)
I've only been on the internet since 1989, but I, too, used to read Usenet. I haven't for several years. Why? The sheer volume of it makes it too difficult to keep up with.
As you point out, Usenet used to be the default place for discussion. But that in itself--not the rise of other discussion forums--led to the current state, where a much smaller fraction of people on the internet use Usenet for discussion. As the volume of Usenet users grew, newsgroups simply became too big for the average person to follow. Face it, if you want to discuss the latest episode of your favorite TV show, would you rather do it in a group of twenty people, or in a group of a thousand? Sure, the larger group will be more diverse and the best comments will be more insightful than in the smaller group, but that's a secondary consideration compared to the time it takes to follow the group.
It's almost paradoxical--fewer people use Usenet now because so many people use it that it's inefficient. I imagine a much smaller percentage of people use it than did 10 years ago, while the absolute numbers who use it are still growing.
The rise of web-based and email-based discussion forums are a result of the death of usenet, not its cause. It's a far from ideal solution. The implicit goal is to cut the time it takes to follow a discussion to a reasonable amount. The solution is to create smaller groups of people simply on the basis that not everybody knows about the forum. (Not that this was intentional, mind you--I'm not saying anyone ever sat down and said, "I'll create a web-based forum so that only a handful of people will know about it and thus discussion will occur at a manageable level, despite the fact that I'm not going to actively exclude anyone." It's more of a Natural Selection sort of pressure.) But for the average user, it's better than Usenet.