Geographical Mapping of Website Traffic?

inputsprocket asks: "Having gotten into a project that involves doing some geo-statistic data mining analysis on Apache log files, I couldn't believe how there seems to be such a large chasm, in open source's vast library, for such a powerful analytical tool. Unless I'm wrong, then there is no such open source project available for *nix variants. There appears to be a great deal of brand software (heh — only for Windows off-site analysis though), yet there is are great open source tools for generic mapping tools (GMTs). The available GMTs seem to be overkill for mapping of weblogs and require a lot of in-house programming to map IP addresses to cartographic data using free geo-location databases. The closest I found was Palentir. Does one really have to rely on the powerful needs of Map Server, MapTools, Carto Web, or Map Builder to draw simple maps with statistics on web usage, given the extra programming needs required to do this?

"To see what is possible, have a look at DeepMetrix's LiveStats.XSP — a commercial option, which isn't a problem. It's just a shame that it's only for Windows and for of-site browsing.

Somebody, please tell me there's more to open source geolocation of web traffic than Analog."

Try hostip.info

[Space cowboy]

I created the [gornall.net] hostip.info [hostip.info] website a couple of years ago which is an open-source geo-location system. I've since passed it to others [slashdot.org] to maintain.

On their front-page right now is a geographic map of website traffic, which is a "public-beta" of a mapping system (upload IP addresses, get a map).

They don't actually appear to give out the code (from only a quick look on the site, that may not be correct), but this ought to be pretty trivial to do yourself if you wanted... The public API of hostip.info returns the lat/long of any IP address it knows, and plotting lat/long on an image of the world ought not be too hard.

I designed the whole system so you could download and set up your own geolocation database, slaved from the master at hostip.info with regular updates. Then you can query IP addresses locally and generate charts however/whenever you wanted...

Just a thought,

Simon

NeXT

[2.7182]

I seem to recall that the NeXT machines that I used in the 90's could do stuff like this with some piece of software. Anyone recall the name ?

perl and cpan are the answer

[stry_cat]

Like most things, just use perl and a cpan module. Apache2::Geo::IP [cpan.org]

Google?

[CodeMonkey22]

Google Analytics [google.com] does a decent job of this. I wonder if they have an API you could hit against it...

Azureus

[slim]

Azureus can display the location of your peers on a world map.

That's Open Source -- so look at their source.

Azureus-RIAA.

[Anonymous Coward]

"Azureus can display the location of your peers on a world map."

Thanks

ip2location

[Utopia]

http://ip2location.com/ [ip2location.com] is the best database out there.

Re:ip2location: accurary != precision

[DaveCar]

I can't see much point in anything that is supposedly more "accurate" than by country. Using the demo site there I got either just "UK" or "UK, London" for a whole bunch of addresses dispersed around the UK. Seeing as most home users will be on some xDSL connection which is probably peered/registered in London/$MAJORCITY, but they are perhaps a few hundred miles away it doesn't seem very "accurate" just more precise (but wrong). Maybe for the US you'll at least get an idea of which time zone they might be i

Re:

[poot_rootbeer]

Agreed, attempts to map IP addresses to geographical locations are so imprecise as to be largely useless. I don't know if it's still the case, but the last time a company I worked for attempted to do geotargeting by IP discovered that $SOME_HUGE percent of our users were all apparently living in Virginia. Turns out those users were all coming through an AOL traffic relay located there. They could have been anywhere in the country in reality.

If you want to find out where your site's users live, ask them!

Re:

[bcrowell]

Hee hee ... I just went to http://ip2location.com/ [ip2location.com] , and they informed me that I was located at "(US) UNITED STATES, VIRGINIA, HERNDON." Mm...no, I'm in Los Angeles, and no, AOL isn't even my ISP :-)

I have at least two misgivings here:

1. The precision seems to be horrible.
2. It might not even be a good thing if this kind of capability was reliable and easy. I recently got my site DDoS'd by a subdomain that reverse DNS revealed to be a block of servers owned by the government of a certain province in Spai

OpenLayers.org, OSGeo Foundation and GeoRSS

[Lord Satri]

I don't know the answer, but I can direct you to OpenLayers.org [openlayers.org] and to the Open Source Geospatial foundation [osgeo.org] to look for answers. I'm interested to the answer because we want to geolocate visitors for slashcode itself, and I copied your question on Slashgeo.org [slashgeo.org], which may help attract more answers to this /. story.

While your at it, GeoRSS is also open and, in my opinion, the future of RSS. Learn more about it here [georss.org].

Invented On Slashdot

[Doc Ruby]

As seen on Slashdot [slashdot.org]: Map-O-Net [map-o-net.com] maps IP#s on a fractal-ruled IP#-space map, including geography and orgs.

Perl Module Geo::IP and MaxMind Database work ...

[xmas2003]

A while back, I wrote a quick little Perl script to do geolocation and other misc. info [komar.org] - see the page for more info, but it uses the free Perl Module Geo::IP and the MaxMind database. The free version of the later used to only provide country data, but they now provide lat/long and city info ... seems reasonably accurate, although with all of these things, the results can sometimes be way off.

So with that, it wasn't too hard to write another Perl script that parses the IP address from the Apache logs, acquires misc. geolocation info, and then dumps out the lat/long data in Javascript for the Google Maps API - see this graphically shown on my christmas maps page. [komar.org]

Re:Perl Module Geo::IP and MaxMind Database work .

[LanMan04]

Yes, I'm sure we can *really* control your xmas lights....weren't you around a year or two ago when all similar sites turned out to be BS?

Re:

[inputsprocket]

Xmax -
please get in touch if you wish, but I did some Apache benchmarks with Geo::IP and Pear's GeoIP, both using Maxmind's free database and while Pear was quering at 1/5ms on 100 concurrent connections, Geo::IP was answering 1:180ms but with apache balking after 2000 request...

This really disappointed me and as such we only run PHP on the maxmind dbase....

I never looked into why the delay was there, but care to comment?

phpmyvisites

[inputsprocket]

Replying to my own question, but I think I found the most complete OS project out there for geolocation log statistics....

I've been playing with phpMyVisites [phpmyvisites.us] and it looks a very nice and thorough implementation of what I was looking for.

Although it's method for country geolocation is a bit *cough* rough, I think a module to hook into Space Cowboy's hostip.info [hostip.info] shouldn't be too difficult...

The only problem with it is it uses mysql to store the logs *grrrrr*

OpenLayers [openlayers.org] looks slick!

Re:

[slowbad]

The statistics from Ritecounter [free invisible web counters under 3000 per day] ties itself into Google Maps, but so far it only does 10-20%

They don't distinguish between Time-Warner customers in Chicago versus New York, so you get a lot of DC-area longitude and lattitude URLs.

ClustrMaps

[Anonymous Coward]

Why don't you try http://clustrmaps.com/ [clustrmaps.com] you can tag your site to it. Easy really.