Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
News

Where's the Open Data? 56

Posted by Cliff from the diamonds-in-the-rough dept.
blamanj asks: "There's a lot of open-source code around, and generally, it's quite easy to find. Finding open source data, on the other hand, can be quite a pain. Why isn't there a common reponsitory for public domain data sets? I'm thinking of things like lists of world cities, dictionaries of stemmed words, population data, etc., etc."
This discussion has been archived. No new comments can be posted.

Where's the Open Data? More

Where's the Open Data?

Comments Filter:

  • Your Tax Dollars At Work (Score:5, Informative)

    by Dr. Bent ( 533421 ) <<ben> <at> <int.com>> on Monday November 11, 2002 @11:18PM (#4648055) Homepage
    Right Here [usgs.gov].

  • Soon you'll be able to try... (Score:4, Informative)

    by damien_kane ( 519267 ) on Monday November 11, 2002 @11:30PM (#4648112)
    MIT's SuperArchive [technologyreview.com]

    Grabbe the link off of rootprompt [rootprompt.org] in case any of you care

  • NIMA and NOAA too (Score:5, Informative)

    by stanwirth ( 621074 ) on Monday November 11, 2002 @11:33PM (#4648138)

    NOAA [http] provides Bathymetry data [noaa.gov] and electronic navigation charts (vectorized) [noaa.gov] and NIMA (that's right, .mil, -- NIMA used to be the Defense Mapping Agency [nima.mil] provides city lists and populations for all the countries in the world, as well as DEMs (digital elevation models--i.e. gridded topography). The National Atlas project [nationalatlas.gov] provides boundaries of federal lands, outlines of states, locations of major cities, stuff like that.

    ENJOY!

    • Lots of great links there, but you left out The CIA's world fact book. They publish as much as they can so that anyone (including their own agents) can access the needed information, from anywhere. World Fact Book [cia.gov] http://www.cia.gov/cia/publications/factbook/index .html

  • How 'common' do you need it? (Score:5, Funny)

    by tswinzig ( 210999 ) on Monday November 11, 2002 @11:37PM (#4648173) Journal
    Why isn't there a common reponsitory for public domain data sets?

    There is, it's right here. [google.com]

    (aka The Internet)

    • Re:How 'common' do you need it? (Score:1, Funny)

      by Anonymous Coward
      Slashdot ought to implement a new filter for its comments section in preferences: Score penalty for reference to Google in Ask Slashdot question.
      • Slashdot ought to implement a new filter for its comments section in preferences: Score penalty for reference to Google in Ask Slashdot question.

        Unfortunately, most of the Ask Slashdot's are so lame they can be answered with a simple google search.

        The editor that posts the Ask Slashdot should first see if he can easily answer the question with a google search before posting the article.
  • the world almanac? Oh no, a book? made of paper? what's that? Just because it isn't in digital form doesn't mean it's not out there. Try the public library sometime.
    • on Monday November 11, Apreche said:
      > Try the public library sometime.

      For many people that is usually not a valid option. Public libraries are, too often, one of the first "luxuries" a government cuts back on, and are rendered not useful by sheer neglect.

  • Missing the point (Score:1, Interesting)

    by Lavos ( 52351 )
    I understand the importance of Google and the like, but let me give an example from the MS side of things, and why we love it.

    SQL Server and Access both come with the Northwind database. If I have some new query that I'm trying to write, for instance randomly returning different numbers of products for each product category, it is pretty darn handy to have a standardized data set to pull from for my example code.

    Otherwise, I have to include DDL and DML just to create the example data. Instead, I can just say "Run this against Northwind."

    The same applies for training and learning. Northwind is a pretty well known database, and most established developers won't have to learn a new schema in order to demonstrate a new concept.

    So rephrase the question from "Where can I find some data?" to "Where can I find a data set that other developers are using so we can more intelligently exchange information?"
    • Oh yeah, and I realize that the article's poster/asker really is just wanting data, but I would like to see this turned into another topic entirely.

  • here ya go (Score:5, Informative)

    by zogger ( 617870 ) on Tuesday November 12, 2002 @12:17AM (#4648406) Homepage Journal
    --here's a great site. refdesk.com [refdesk.com]. Matt Drudge's father runs this site,AFAIK, a boatload of data links.

  • Amen (Score:5, Informative)

    by pizza_milkshake ( 580452 ) on Tuesday November 12, 2002 @01:00AM (#4648601)
    That kind of data is out there if you want it, but it isn't always in the format you want it and sometimes it's hard to find (less hard with google). I started a site for this kind of data at www.tonsoflists.com, which contains data in MySQL tables that you can format into different kinds of sets,orderings, formats, etc. Just did it for fun, but haven't gotten any feedback.
    • Hey, nice resource. I've actually had need for some of that information in the last year.
    • Just did it for fun, but haven't gotten any feedback.
      I just checked the site out. Interesting, but I have a couple issues:
      1. There isn't any contact info on the site.
      2. The capital of MN is St Paul, not St Louis. I hope that's not representative of the rest of your data.

  • A little surprised . . . (Score:3, Informative)

    by Discoflamingo13 ( 90009 ) on Tuesday November 12, 2002 @01:15AM (#4648687) Homepage Journal
    nobody posted this - standardized data sets for training AI. [nju.edu.cn] It's a start, anyways - useful for comparing one machine learning system to another. Maybe you could use it for something else?
  • What's the deal with NASA data? Especially Hubble data? Sure would be nice to make some screensavers without those unsightly logos on them.

    Oh, yeah... just remembered a nice bookmark! :-)

    http://earthobservatory.nasa.gov/Newsroom/NewImage s/images_index.php3 [nasa.gov]

    The NASA Earth Observatory. Don't know how open, though.

  • Where's the financial data?? (Score:4, Interesting)

    by grammar nazi ( 197303 ) on Tuesday November 12, 2002 @02:00AM (#4648922) Journal
    Screw all of the other data mentioned above! I want to run pricing models on historic financial data... e.g. intraday option prices and vols, dividend schedules 30 years back, intraday stock prices for way back.

    This is stuff you can't download for free from Yahoo, CBOE, or other places.

    If I can just get access to this data, then I will make enough money to purchase the other data.

    • Re:Where's the financial data?? (Score:1, Interesting)

      by Anonymous Coward
      Some sources:

      US Company Security Filings:
      http://www.sec.gov/edgar.shtml

      Historical SEC findings in XML format:
      http://bulk.resource.org/edgar/

      Limited stock prices (15-30 years) are available from yahoo.

      What I'm looking for is historical stockbuyback lists.

  • Boardgame/Parlor game data? (Score:3, Interesting)

    by ClioCJS ( 264898 ) <cliocjs+slashdot AT gmail DOT com> on Tuesday November 12, 2002 @02:38AM (#4649067) Homepage Journal
    For a long time I've wanted a website to suppy open data for the purpose of playing board games / parlor games...

    For example, ever run out of trivia questions in your version of Trivial Pursuite? Or used up all the word cards in Taboo... etc etc.

    I think in the event of running out of data for your board game, it would be nice to download more. (And this would make a cool website.)

    Especially if they came with PalmPilot/Windows versions that would administer the game for you. For example Taboo consists of a word that you must get the other people to say, but there are 7 words that you CANNOT say as a clue. For example the word may be "George Bush" and you can't say "Texan", "President", etc. This game is fun but we calculate we'll use up all the data that comes with it in about 30 hours. The "electronic" version is $40. That's hardly worth it. If we could just download data, we could play forever. So .. um .. yes.. I want open data.

    • For example, ever run out of trivia questions in your version of Trivial Pursuite? Or used up all the word cards in Taboo... etc etc.

      I think in the event of running out of data for your board game, it would be nice to download more. (And this would make a cool website.)

      You know, I came up with this idea just two days ago, after discovering how lame the new questions are in the Trivial Pursuit 20th Aniversary Edition. It's really time these questions got open sourced.

      What I had in mind was a system whereby people could go to a website and contribute trivia questions. After some human screening, the questions would go into a database, which would then be used to generate PDF files of Trivial Pursuit Cards that could be freeley downloaded and printed at the users home computer. Alternately, users could pay 10 bucks to have a deck of pre-printed Trivial Pursuit cards sent their house.

      I'm interested in starting this project over at Sourceforge. Anyone else like the idea?
      • --I play trivial pursuit once a week on irc with some friends. We do it ad hoc round robin style, and the good part is, you come up with your own questions. It's your "turn" as long as you stump people, whenever someone gets the answer, it's their turn, and so on. Seems to work great. You start a session by locking in the theme of the questions. Everyone gets a chance to both share knowledge they have, and also to learn from the others.

        I guess if you wanted to get it in some sort of doc form you could use a session log and tweak it.
        • Where would I find such logs? What are the licensing restrictions on such things?
          • --you do it yourself when you are in the channel, save the session-the conversation- to a logfile of your choice on your machine, or the OP does and you can get it from them. Try it out, go to an irc channel someplace on some subject that interests you then search your client IRC program you have running for whichever menu button you mash or command to type to save the session to file. The few different ones I've used all seem to have that function. I'm not an irc guru, but they are there. Perhaps one of the olden tymes unix guys here who've used irc for like forever can be more specific on the saving and tweaking part. The log files are a shade clunky but readable, you might want to cut and paste a lot to turn it into a normal easier to read html page, not sure if there's an automatic way to do that, but it's doable that way at least and not that hard unless it's a humongous log file, then it's just tedious..

            Here's an example of where the logging function is found in x-chat, the client I use under linux, the path is > Settings > Setup > Options, you'll see the logging function then.

            Now licensing ya got me, ask permission if it's not yours is the best bet. I can't see folks getting real anal over it, either, but ya never know. Probably just depends what you intend to do with it, share it around, they'll probably say have at it, try to develop a commercial product and sell it, you need more serious advice and a contract I guess. Like "yo, zeke, mind if I take this logfile and tweak it and come up with a nice page of trivial pursuit questions?" "Sure man goferit, send me the url when you finished I want a copy" "thanks" "swell". Beyond that your gonna need a contract of some sort. Cash changes reality. Asking permission is always the safest bet, IMO.
      • Well hopefully the questions would undergo some sort of evaluation system so they could be grouped by difficulty & subject, then a database would manage them such that certain questions could be used for games OTHER than trivial pursuit.

        The idea would be to support as many different games as possible...

        • Trouble is, different games have different requirements for their questions; Trivial Pursuit is a simple question and answer, Jeopardy is an answer then question combination, some would prefer a multiple choice format, and games like Taboo are a separate animal entirely. Writing for all these different games would be like cross platform software; it would make the task more difficult, but the advantages are clear as well. If questions added for one game could be easily ported to other game formats, considerable effort could be saved in having to go out and find new trivia questions. How best to implement that is an interesting connundrum, though.

      • I like the idea a lot. And I have an idea for a starting point: rec.games.trivia [google.com] they run quizes in the group which should provide a substantial base.

        I'd recommend converting the whole thing into XML with fields for:
        • Major genre
        • minor genre
        • the question
        • a hint
        • some numerically scaled level of difficulty
        • multiple choice options A, B, C, & D, and
        • the correct answer
        that should allow anyone who want to build an interface a lot of leeway as to how they want to structure the quiz.

        A long time ago, there used to be an IRC based game that was a run like Jeopardy [google.com]. Don't know if the games were archived though.

        • While looking for available domain names, I came across OpenTrivia.com [opentrivia.com]. I like the spirit of what he is doing, though the license is a little more restrictive than what I would prefer.

          Much as you suggested, it uses XML to allow people leeway on structuring a quiz. It doesn't offer multiple choices, though. Thats not really a concern for the Trivial Pursuit application, but still, I'd like to have it as an option.

  • It should be clearly labeled (Score:4, Insightful)

    by jki ( 624756 ) on Tuesday November 12, 2002 @03:49AM (#4649296) Homepage
    There's a lot of open-source code around, and generally, it's quite easy to find. Finding open source data, on the other hand, can be quite a pain

    When you go to Google to find software to fill some specific need, you already know quite clearly how to search. The problem with finding "open data" is that there currently is not any commonly used clear label on such texts, research and articles. I tend to mention that the content is released under the GNU Free Documentation License [gnu.org] or FDL when I want to release something to be freely utlized by anyone. One such case is for example the Amazon Discoveries [cyberian.org] series. Not that it would be any useful for anyone :) This problem is a bit related to the problem of releasing your idea or concept under such license - there does not seem any clear practise how to go on about this :: what to do if your idea might be unique but you do not want to patent it. We have that exact problem with for example the Openchallenge concept submissions [openchallenge.org]. Any ideas on what practises to use in that case would help us out.

  • Timelines and the 'Necessary Web' (Score:4, Interesting)

    by RobotWisdom ( 25776 ) on Tuesday November 12, 2002 @03:55AM (#4649313) Homepage
    I think you'd lose more than you'd gain if you tried to centralise this process-- it's hard enough to keep a local webpage up-to-date.

    I agree in theory that we need a Semantic Web where content is easier to find, but I don't think XML-etc can really help. [rant] [robotwisdom.com]

    My current theory is that individuals need to build the 'Necessary Web' which consists, like an encyclopedia, of a page for each topic (or many pages by different authors, on their own websites). Four special traits make a page qualify as 'Necessary':

    -- an attempt to be FAQ-like, and briefly cover all the important subtopics on a single page.

    -- an attempt to sort thru and link all the best web-resources on the topic. (By reducing the linktext to one- or two-word [text buttons] [robotwisdom.com] you can fit hundreds of links into a useful page.)

    -- a timeline, to present the most possible data in the neatest possible way. [theory] [robotwisdom.com]

    -- The Open Web Content License [robotwisdom.com] to encourage others to recycle-and-update your content, requiring only that they clearly link your page as one of the original sources.

    Most recent example of this format: Linux/Unix [robotwisdom.com] (timeline w/100s of links)

    I believe that once a critical mass of authors adopt this format, taking on the most useful topics, there will be a rapid shift from the current search-frustrations to something very much like the Semantic-Web ideal, without even requiring any fancier technology than simple HTML.

  • not an easily distributed task (Score:3, Insightful)

    by joe094287523459087 ( 564414 ) <joe@jo e . to> on Tuesday November 12, 2002 @06:00AM (#4649639) Homepage
    it costs more money to get data than to make code.

  • Surprise. No one as yet seems to have mentioned ibiblio? [ibiblio.org]
  • When it comes to electronic music (house/techno/idm/electro etc) there are some excellent user-contributed databases out there.

    Check out and add to:
  • I realize this is slashdot, but maybe there's other stats geeks out there that like baseball. Baseball-Reference [baseball-reference.com] is hands down the best stats site out there. And it's based on the Lahman database [baseball1.com] which is freely available (newest version coming soon).

    -Gabe
  • There are plenty of government sources of data that is free and open to anyone. The Census Bureau [census.gov], Energy Information Administration [doe.gov], Commerce Department [commerce.gov], a good starting place is FirstGov [firstgov.com].

    In addition, most state governments and even county level governments publish large amounts of data.
  • Someone should mention the UCI Machine Learning Repository [uci.edu] and the UCI Knowledge Discovery in Databases Archive [uci.edu].
  • A good starting URL - http://www.biotech.ubc.ca/bioinform.html#public

Slashdot Top Deals

Scientists will study your brain to learn more about your distant cousin, Man.

Close