Crazy Firewall Log Activity — What Does It Mean? 344
arkowitz writes "I happened to have access to five days worth of firewall logs from a US state government agency. I wrote a parser to grab unique IPs out, and sent several million of them to a company called Quova, who gave me back full location info on every 40th one. I then used Green Phosphor's Glasshouse visualization tool to have a look at the count of inbound packets, grouped by country of origin and hour. And it's freaking crazy looking. So I made the video of it and I'm asking the Slashdot community: What the heck is going on?"
Comment removed (Score:5, Funny)
Re: (Score:3, Insightful)
Re:Skylab Shreds (Score:4, Insightful)
Yes, he knows the firewall and the traffic. The question is - why is there suddenly traffic suddenly appearing from every country in the world at the same time? and again a number of hours later? And again 5 or 6 times? Suddenly there is inbound packets from every country in the world, for an hour or two, then it dies off. For some countries, the first 'stripe' is also the start of consistently higher traffic from that country. Does this mean anything?
I think it might be more useful to know the actual dates, and see if this corresponds with any spikes in spam or virus activity. What would be most useful would be know the dest port number of the inbound traffic, that could give us much better clues as to the reasons behind the patterns.
Re:Skylab Shreds (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
"Video conference calls do not last for hours or days."
Maybe not in your world, but then again it's likely you've never been in a Camfrog room. Also, on Skype, my UK and AUS partners and I just leave the conversation going. If any of us are near the computer and hear the others, we'll speak up and start a conversation. It's much simpler. Our machines are all located in our in-home offices.
I usually leave Camfrog and Skype open and connected 24/7. It's just much simpler that way.
Re: (Score:3, Funny)
Re:Skylab Shreds (Score:5, Insightful)
It's an ad for a 3D graphing service.
Indeed, the guy from the graphing service is the same guy who made this.
Re:Skylab Shreds (Score:4, Insightful)
Piss-poor ad though. How many people saw the video and thought "I must get me some of this graphing tool!"? My first thought was "interesting way of presenting information, but his graphing tool is crap".
Re: (Score:3, Insightful)
My first thought was "why does everybody have to make everything a video?"
Re: (Score:2)
Conference calls, backups, and synchronization from damn near every country on earth? For an agency within a single US state? No.
Also, too: The packet rates are far too low for those activities. If you watch TFV, you'll see that the largest users are only up to around a couple hundred packets per hour, which is such small number that even if you multiply it by 40 (due to the scaling done by the geo-IP service[1]), it's still far too small for those activities that you listed.
Any other theories?
[1]: It'
Re:Skylab Shreds (Score:4, Interesting)
A botnet attack? But then the activity shouldn't be concentrated by country, but spread around the world about evenly.
Or it could be that someone's seeding a torrent from behind the firewall. That would explain the suddenly starting continuous activity. It might also explain the concentration by country (language or timezone). It would help if the graph could be organized by such factors.
Re:Skylab Shreds (Score:5, Insightful)
That would explain most of it.
Re:Skylab Shreds (Score:5, Insightful)
Also is he plotting this based on potentially spoofed IP addresses? I'm thinking not just a botnet, but a botnet that doesn't care if it's getting packets back or not. It may not be every country in the world, just a bunch of random IPs coming from zombies which may (or may not) be in far-flung places.
Mal-2
Re: (Score:3, Interesting)
Re:Skylab Shreds (Score:5, Interesting)
Yes, he knows the firewall and the traffic. The question is - why is there suddenly traffic suddenly appearing from every country in the world at the same time? and again a number of hours later? And again 5 or 6 times?
I get a lot of distributed dictionary attacks like that. Its pretty normal.
Re:Skylab Shreds (Score:5, Insightful)
Bingo. My thoughts exactly.
Unless his gives up some more data, hard to tell for sure.
But, I agree, it sounds like someone is using their employer's (government)bandwidth to torrent. Could be a machine that someone shuts off the monitor on but P2P downloads overnight with a scheduled P2P app.
The peaks/valleys might be explained by reset packets introduced by the ISP temporarily killing the outbound requests and it takes the inbound requests awhile to trickle off.
You can see this same type of log traffic by simply starting a torrent, waiting a little bit, then stopping the P2P client, waiting awhile again, then restarting it. Rinse, repeat and you will see something that looks awfully close to what you have.
Reset packets essentially create the same traffic pattern, but for a different reason (ISP- introduced traffic "shaping").
Re: (Score:2, Informative)
Computers are used by people. People who wake up, work, play, sleep, have weekends, business holidays, religious holidays, events and a pantheon of other reasons why they might act in seeming semi-concert.
You're suggesting that for the five day period in question, the majority of people work up at the same time GMT? Not 7am local time, but 9pm GMT everywhere in the world? Or did you just not actually look at the video (which shows spikes of data from every country in the world at the same time)? "Timezone effects" should eliminate these sorts of lines, not cause them, by spreading that kind of activity out over 24 hours.
Re: (Score:3, Funny)
Not sure what it means, but I'm tempted to plug-in Guitar Hero and jam along to your firewall logs.
Just let me finish my Klax game first.
Re:Skylab Shreds (Score:5, Informative)
2001 (Score:2, Funny)
Anyone else tempted to hum the theme tune to 2001 when they looked at that?
And also... "oh my god... it's full of stars"
Re:2001 (Score:5, Funny)
And also... "oh my god... it's full of bars"
Fixed that for you.
I'm confused (Score:2, Funny)
Re:I'm confused (Score:5, Insightful)
Re:I'm confused (Score:4, Informative)
How does /usr/share/GeoIP/GeoIP.dat ban my IP address?
Re: (Score:3)
I didn't even think about that being a possibility.
On a related note, thank you for that tidbit of information - I'm sure I'll find it useful in the future.
--- Mr. DOS
Re:I'm confused (Score:5, Informative)
Eh what? There's several GeoIP databases that you can install locally. In fact it seems like Quova is the only database you have to query remotely, which is somewhat crazy if you ask me. Or buy a server from them.
MaxMind [maxmind.com] is the best known one. Installing it on Linux server using yum merely takes "yum install GeoIP*"
vertical stripes (Score:2)
Re:vertical stripes (Score:5, Informative)
It looks like an active attack probably from one source with a number of controlled bots helping out.
The packets from every country at once are probably spoofs sender IP addresses from one or more sources (probably the spike countries).
The spiked country traffic are probably the controlled bots attacking the host actively.
Without seeing the actual packet data it's just a guess though.
Re: (Score:2)
This could just be a case where traffic is routed through different proxies at nearly the same time by a relatively small group of computers or Something coordinated many different machines to connect to their server(s) like a botnet.
Re: (Score:2)
I'm actually a lot more interested in the vertical stripes than the horizontal ones. It looks like at certain times, every country in the world sends a packet.
Yes, I noticed that. The edges on the stripes are so sharp that I suspect a bug in the analysis or graphing program. Either he's being attacked intermittently by an widespread, tightly synchronized botnet, or the breakdown by country is bogus. I'll bet he has some bug like getting the bytes of an IP address backwards, so when he gets a traffic s
Re: (Score:3, Insightful)
Yeah, I meant to say that it's also difficult to tell what's going on because you conflated all destination protocols and ports together.
Re: (Score:2)
That would make more sense if they were regular - but those lines appeared to show up at several irregular periods throughout the day. Though on the flip side, they may have several cron jobs that run and ping (most of) the outside world to make sure there wasn't a nuclear detonation during teatime or something.
Without knowing more about the environment and having more data, we can only speculate. But I doubt it's malicious - seems unlikely to follow that consistent of a pattern for the vertical stripes. S
Re: (Score:2, Interesting)
If we assume the video conference included people from all of those countries, who all endeavored to join at the same time GMT regardless of local time, and they keep conferencing for several days without sleeping, then yes, that would account for those horizontal lines that suddenly get thick at the first vertical stripe and continue until the end of the five-day period. That definitely makes sense... ~
No forreals... (Score:2)
RTFV: this is one of the more interesting problems ive seen posted in years.... Especially as a China resident... Odd... Thought /. community?
"Does this mean anything?"
RTFV?!?!??!?! (Score:2)
Don't you mean WTFV?
Finally (Score:2)
Somebody who doesn't forgets Poland.
(even if traffic from there wasn't unusual in any way)
Another Slashdot Ad? (Score:5, Insightful)
Wait, is this just an advertisement for Glasshouse? The voice in the video on Green Phosphor's website is exactly the same.
What gives?
Re:Another Slashdot Ad? (Score:5, Informative)
Wait, is this just an advertisement for Glasshouse? The voice in the video on Green Phosphor's website is exactly the same.
It is totally the same guy - the background noise sounds identical too - like he recorded it on the same microphone with the same environmental conditions.
Hell, he even starts each narration exactly the same with the pattern of, "Hi <name> here."
Re: (Score:2)
Heh. Well, if they need voice talent (and they *do* need voice talent, let me tell you), I'm available.
Re: (Score:2)
Re:Another Slashdot Ad? (Score:5, Informative)
Welcome to the future of advertising.
Re: (Score:2)
And to think that I was going to ask what kind of person has enough time to make data visualizations like that. Guess it's easy when that's your job.
Still, the video raises an interesting question, slashvertisement or not. (FWIW, I wouldn't have known what company was being slashvertised if it hasn't been pointed out a dozen times in the comments)
Re:Another Slashdot Ad? (Score:4, Funny)
Re:Another Slashdot Ad? (Score:4, Informative)
To be fair though, he didn't link to the companies in submission [slashdot.org], only the video and merely mentioned what he used. I guess kdawson added the links. While certainly promoting their own software, the bitching about it has been taken to quite irrelevant levels in this story. Instead of bitching about that, we could had have much more interesting discussion about what it actually is or did anyone else see such spikes on the same days. Personally I think it might be some botnet scanning either for exploits or to find each other (this might be extremely relevant if some botnet was taken down on the same day and P2P scanning to find other nodes kicked in). Port numbers and a little more info would had been helpful, though.
Interesting. (Score:3, Insightful)
It's pretty interesting. You can see the countries with the largest botnets in the log... which also seems to suggest that a large majority of the packets are coming from the one botnet... since a good number of them kick in at the same time.
It also looks cool. Which is critical.
Filter your data... (Score:4, Insightful)
Why am I worried? (Score:4, Insightful)
Comment removed (Score:5, Insightful)
My guess (Score:3, Interesting)
Re: (Score:3, Insightful)
Temporal Discontinuity in Data (Score:2, Insightful)
Looking at the pop-up labels that show up when you mouse-over the data, there seems to be a huge temporal discontinuity in your data set: right at the first vertical stripe, the displayed date/time labels jump from 2009-09-17 to 2009-09-27. Maybe I'm just misreading the display, but a 10-day discontinuity would seem to account for the anomaly you describe.
It couldn't be that easy, could it?
Re: (Score:2)
It might account for the first vertical stripe directly (ten days' worth of minimal packet data accumulated into one data point), but then you would expect the data from the busy countries to then be ten times as high for that one data point.
But what it does indicate is that there are ten days of missing data that most likely show the start of this behavior and could provide further insight.
I wonder whether this data was inadvertently left out by the submitter, inexplicably dropped by the third-party proces
Ad (Score:5, Insightful)
it means that this is an ad for Quova and Green Phosphor's Glasshouse
"And its freaking crazy looking" (Score:5, Insightful)
Am I the only one who found the five minutes of this video to be about as interesting as listening to a stoned person describe the cracks on the ceiling?
You designed the visualization, buddy. If it's "freaking crazy looking," rather than yielding any useful insight, then obviously you did not visualize it in a meaningful way. You failed, in other words.
But as an earlier poster noted, this is just a Slashvertisement for the visualization tool in question. No doubt it will be quite effective on the kind of people who talk as slowly as the guy in the video.
Re: (Score:2)
You designed the visualization, buddy. If it's "freaking crazy looking," rather than yielding any useful insight, then obviously you did not visualize it in a meaningful way. You failed, in other words.
I don't know this guy or how he obtained the data he used to build the visualization but based on his question asking what is happening, it would appear that he doesn't understand the data that he analyzed visually. So, to respond to your point that it's his fault because he couldn't properly frame the data v
Re:"And its freaking crazy looking" (Score:5, Insightful)
I wouldn't be so quick to support the author. The voice on the youtube video sounds a lot like the voice on the youtube video featured on the front of the webpage for http://www.greenphosphor.com/ [greenphosphor.com]. If not him, look at the related videos, notice a pattern? Maybe one of the other voices talking about features of the product will sound familiar.
That wasn't complaining. THIS is complaining. (Score:5, Insightful)
You want complaining? How about this: This visualization is terrible.
The video took five minutes to watch and most of it was him rolling over the bars in the 3-D chart so you can see what each of the lines means. If that's supposed to be a useful visual aid, I'll eat my hat. It's bad enough that you have to manually roll over every data element to figure out what it is; scrolling through the graph seemed dead slow. I hope that's not a limitation of the product itself.
Simple labels on the axes of the graph would have been nice. Far be it from anyone to try stick little flags next to the lines to represent different countries. Hell, just color-coding them in a totally arbitrary way would have made the graph easier to read.
BTW, a quick look at the Glasshouse site reveals all their output looks pretty much just like this demo. And there's no evidence that you can export one of their rudimentary 3-D graphs to "pretty it up" in a real 3-D app. Instead, their raison d'être appears to be allowing you to run around looking at these graphs... in Second Life.
I'm sorry, but if you're doing something like plotting fractals, for example, where visual similarity to patterns is the whole point, I can forgive you for coming to the conclusion that "it's crazy looking." If what you're doing is trying to provide a visual to aid in the interpretation of data, then the visual should -- y'know -- aid interpretation. A glance at this graph, on the other hand, reveals nothing; not even what it's supposed to represent.
In summary, Edward Tufte will be rolling in his grave when he dies from looking at this graphic.
Re: (Score:2)
You bring up a good point: the raw data would be more eye-friendly than this travesty. But also, if it's not backed up by the free and open raw data - that is, if there is such data, but it's being kept secret - then it cannot be good science.
Re:That wasn't complaining. THIS is complaining. (Score:5, Interesting)
Everyone always wants me to have labels on the graphs. I don't put them there unless you roll over the data, because I want you to see the patterns in the data without bias first.
Why? The only reason for that would be so you could go, "Whoaahh, it's crazy looking." You've proven that. Anonymous data with no points of reference has no meaning. If you honestly think your graph has more value to the viewer than this graph from 1880 [yorku.ca] showing the population of Sweden over time, I think you're kidding yourself.
It is actually pretty simple and makes it quite clear what is going on
That's debatable. I've argued that it could be much, much clearer.
Finally, I am not interested in producing graphs which show you everything "at a glance". Use a pie chart for that. I am making graphs which facilitate a deeper understanding of larger amounts of data than Tufte dreamed of showing using his 2D paradigms.
Careful. If you're trying to get into the data visualization business, it's a bad idea to make it known that you're completely ignorant of Edward Tufte.
For starters, anyone who knows the slightest thing about Edward Tufte knows that he hates pie charts. So he would never say "use a pie chart for that."
Second, contrary to your assertion, Tufte advocates for extremely data-rich graphics wherever possible. He does not advocate abridging large data sets out of laziness. He does, however, advocate data compression when it will reveal data, and he does not like "wasted ink." Your graphs appear to have miles and miles and miles of plotted data -- none of which is identifiable without mouse interaction -- but relatively few points of interest. As you scroll through the data set, half your movie seems to feature the text "empty" hovering in midair above the graph. In other words, your dataset may indeed be large, but your visualization of it is not particularly informationally dense.
Finally, until such a time as your product can reach out of my flat-screen monitor and tweak me in the nose, you're every bit as tied to a "2D paradigm" as Tufte is. All you're doing is making it possible to adjust what is plotted in real time. Tufte would probably argue that it's better to get the plot right the first time. Allowing viewers to take their time to absorb a lot of data points is fine, but they shouldn't have to waste their time fiddling around with the plot to reveal those data points.
Re: (Score:2, Interesting)
I do not want to produce a one-time "plot". I want to show data for what it is. If it doesn't look as nice as Tufte would have made it look, I don't care. The point is not to look nice... it's to provide the ability for people to see what is in databases, without bias. And I still don't think Tufte's paradigms work with as much data as these 3d ones d
Hey mods! Don't mod arkowitz "Troll" (Score:3, Insightful)
This is the guy whose product we're talking about. He wants to explain himself. If you think he tried to use Slashdot to advertise his product, you don't have to mod him up, but if you mod him down to -1 then he'll drop below a lot of people's thresholds and they won't even see that he tried to participate. That's not being fair.
Several factors contribute to this graphics ... (Score:2, Interesting)
First, we would need to know what kind of traffic we are seeing. TCP/UDP? Web? DNS?
On the other hand, I think you have only partial logs, that would explain many of the blanks on your data. Some blanks are too geometric to be correct, you are probably missing a shitload of data.
You have to take into account that, and timezones. Timezones are the key to this. This is probably some public service that gets hit at regular intervals (root DNS server, webserver holding news/stock/climate or similar information,
It looks like (Score:2)
Also take into account that China, Russia, et al are +12 from us So that might explain some of it. In other words, they might be caching your site.
Umm (Score:2)
So why is he using State property for personal gain? My guess is his logs for his website were way too boring.
Shouldn't there be some agency in Florida who does not want their logs posted, even in cartoon format, in an internet video. I'm guessing this is probably either the Florida Dept. of Revenue or the Florida Dept. of Financial Services.
It just means (Score:5, Interesting)
(this is a guess, obviously. Full netflow data would tell me more, but only way to be really sure would be a full packet trace)
This just shows that you're being scanned with random source IP adresses (that's why the vertical stripe lights up). It is essentially a check to see if part of the botnet has more firewall access than other parts, or if a loadbalancer directs stuff to different firewalls, or if you have additional BGP uplinks, some of which might not be quite as secure.
Then the real scan starts, which uses the information gained in the first phase to make sure it tests out all the firewalls the target network has. Especially in the case of backup bgp links, where traffic comes in on physically and administratively different lines (say 1 verizon, 1 at&t, if you've got money to burn, and most govt. idiots feel the need to burn money). If the company in addition to the multiple uplinks outsources firewalls to those ISPs (or "security", not knowing what they're buying and getting nothing more than a smug false sense of security), again this is done by too many govt. agencies, you are bound to find holes this way. This uses actual bandwidth, and cannot be done on some networks. So what you're seeing is a disproportionate amount of scanning traffic coming from countries with fast networks and few watchful netadmins (or netadmins that just don't care, in Turkey's case), and many unsecured computers (and dear God, Turks and Russians really do not see any need for virusscanners, but generally you'd see a few other countries in there too. Heh the Russians are probably worried that running a virusscanner will interfere with their development of new viruses)
The regular repeats of vertical lines are probably to rescan reachability information, in case something changed. BGP can be twitchy, especially with incompetent local admins (on the botnet side of the network I mean)
From the (low) speed of the attack you can further deduce that it was an advanced attack, meant to stay below rate limiters, and presumably meant to stay below the radar. And from the resources required to pull this off you can deduce that this was not a lone hacker. Perhaps an organization (these days, tracing source ip's for security attacks almost invariably yields an IP address in far inland China, which is not because the russians have stopped attacking networks, but the Chinese are putting quantity above quality it seems these days).
And frankly, if someone has this kind of patience, generally they will find at least something, even in a well maintained network. Best hope it was only some files left out in the "public" folder or ~username folders. It's a good bet they probed the network security in other ways too (esp. googling), with IP's that will tell you much more about where the attack is coming from (using many hops is possible, but results in very slow page loads. And we're all human)
Btw : looking up a net's country can be done quickly via dns, no need for external company, no need for any tax dollars :
[kimmy@t61 ~]$ host -t TXT 104.79.125.74.cc.iploc.org
104.79.125.74.cc.iploc.org descriptive text "US"
(don't forget to reverse the IP address : looking up 1.2.3.4 is done by host -t TXT 4.3.2.1.cc.iploc.org)
Re: (Score:2)
bot net (Score:2)
My guess is that it's a bot net becoming active.
The countries with higher traffic during that period are countries that are widely known to have high bot net activity they are also more likely to have server bot net activity, which is why they don't stripe like the over countries.
The stripes are likely day/night where infected PCs are turned off when not in use.
Great ways to start a conversation (Score:5, Funny)
"I happened to have access to five days worth of firewall logs from a US state government agency..."
"While skimming through my grandmother's cookbook, I stumbled upon a recipe for processing yellowcake uranium..."
"In passing, a close personal friend mentioned to me that he would deploy ~30k troops to a Mideastern country, but he's worried that the local restaurantuers won't serve fresh babaganoush ..."
"While I was talking to a famous adult film star about my successful experiment with cold fusion..."
"I was fighting against an alien invasion of the Soviet Union the other day. Natalie Portman and I prepared a platoon of sharks with frickin' hotgrits cannons on their heads, but the unwelcome overlords kept jumping the sharks..."
Re: (Score:2)
+1 Funny. (I'm in the thread, or I'd mod.)
Re: (Score:2, Funny)
Bot-Net attack (Score:2)
I'd guess you are seeing a bot-net attack. The bot-net army would have the greatest numbers in IT-heavy countries (US, India, China). The command structure would cause them all to attack at (roughly) the same time, regardless of time zone.
Or maybe you've been slashdotted.
Privacy concerns - how did you get the data? (Score:2)
Is no-one else bothered by the fact he has access to raw logs from a government system? Are there no privacy concerns from a private citizen being allowed to scan for users of government system? For instance, let's imagine it's the local IRS server - he now knows exactly what forms you were downloading, or perhaps visitors to a government site to help people find providers of mental health care. Really I don't care what the site was, it just seems like there's no valid reason for anyone to have raw data
It's the people avoiding patterns to fear. (Score:2)
This just doesn't seem like a big deal. The countries he points out are all in the same timezones so it's probably just their normal day starting. So this probably correlates to dns refresh or some other aspect (vertical) of general internet operations landing on the same hour.
He needs tcp port analysis and to compare days - the pattern is probably the same from day to day.
Re: (Score:2)
That would need some day/night bars on the graph per country of origin.
You get that kind of thinking with $1,000,0000 budgets from ex spooks selling their services back to a flay over state via power point. "Please note China and Brazil"
If a bot was written to target the US, why run your US bot during the US day, the gov admin might be just at their desk, awake and clicking.
Timezones? (Score:2)
Data jumps? (Score:2)
Maybe the fact that you put random chunks of data from days apart next to each other has something to do with it?
hey, i have access to this amazing tech (Score:2)
for a powerful client, but i need, you, random slashdork, to help me out here
no, i'm not a salesman
Re: (Score:2)
Looks like a sneaky ad to me. (Score:5, Insightful)
The fact that OP did neither, and is involved at a high level with one of the two companies, makes this whole post suspicious.
My best guess is that OP thought he had discovered a way to freely advertise via Slashdot, and victimized us as a result.
I get enough Spam. I don't need to see even more, on Slashdot. Can this user be blocked?
Re: (Score:3, Insightful)
I know it's trollish, but the real question is: can kdawson be blocked?
(yes I know you can block authors in your user prefs... I mean from Slashdot entirely.... save us the pain, please, for the love of god)
basic interpretations (Score:2)
The vertical stripes, indicating worldwide activity at the same time, are probably the result of botnets being ordered to target an area that includes your IP pool. (or possibly, specifically your organization - depending on where you got the logs this may be more or less likely) The horizontal stripes are of course showing continuous activity from specific regions, which can indicate activity of a regional botnet doing general penetration scans looking for more machines to infect. For example, botnets th
did you also have a coca-cola(TM) (Score:2)
while you were processing the numbers? did you use Microsoft(TM) Windows(R) Moviemaker(C) to make the Youtube(TM) video?
Distributed attack? (Score:2)
Maybe all the bots are part of the same botnet and were programmed to attack at the first spike.
The fact they are located in different countries doesn't mean anything, it's simply hiding whoever is really behind the attack.
timed zombies (Score:2)
Looks like BitTorrent. (Score:5, Interesting)
Nice visualization. Wonder if there is some way to do it in real time.
I've done networking and security for a university for the last 10 years. I can guess what this kind of activity would be if it was at my institution. Basically, there are several reasons why every country in the world will suddenly talk to us. They include P2P/Gnutella's, P2P/Swarmcasting, Bittorrent, Skype, P2P-poisoning, P2P-misdirection, and hacker/bot activity.
When we have pulses like you are observing, it is usually BitTorrent.
The Gnutella P2P variants don't usually have that many peers. And, they tend to last for several hours or days.
The various Swarmcasting P2P variants look very similiar to BitTorrent, but again, the users tend to leave them running for hours or days.
A popular Torrent makes connections to hundreds of locations at once, and usually the local user shuts down in minutes (or an hour) when they get their file.
Skype won't be narrow bands. It will be every country in the world talking to you all the time. We have had computers promote themselves up the Skype infrastructure until they are constantly talking to over 600K peers. Of course, it is more normal to see a Skype node talking to 10K to 20K peers, but still Skype won't be bands. Skype raises the floor for the entire graph.
P2P-poisoning would closely match your bands. For several years we observed pulses where every member of a large P2P cloud would attempt to talk to a non-existing IP at our institution. Eventually, we realized that somebody was attempting to render the P2P cloud non-functional by poisoning the P2P community with info on non-existing peers. Of course, since this is a Denial of Service (DoS) attack, this is technically illegal, but we saw it happening for years. But, it appeared to stop a couple years ago (about the time Obama replaced Bush) and we haven't seen any evidence of it lately.
P2P-misdirection is where a cloud will attempt to confuse traffic analysis by throwing out random connections/packets to random IPs. Typically, this misdirection happens all the time, and not in bursts/bands.
Bot attack activity doesn't match your patterns either. We observe several types. None would look like your bands:
- The spoofed attacks will look like every one of your IPs getting acks from a few remote IPs.
- The mapping activity will look like a representative sample of your IPs getting traffic from a few dozen IPs.
- An incoming DoS would have a few of your IPs get (spoofed) traffic from everywhere, but it would be sustained.
- Portscans will only involve a handful of remote IPs.
- The Tag-team SSH password guessing is close. During the last week, we observed about 3000 sources located all over. But, it happens all the time (in the aggregrate), not in bursts. And the sources this week are concentrated in Italy, Poland, Eastern Europe, Colombia, and Brazil. They aren't really all over the world.
So, I'm guessing it is BitTorrent. But, your situation may be way different from mine.
Miles
Re:Looks like BitTorrent. (Score:4, Insightful)
Then, I would say somebody with a large botnet is doing reconnissance on you.
I'm sure you have incoming port 137 blocked. So that traffic is outgoing. I expect that will be your Windows hosts responding to their probes.
They are probably attempting to find your end-hosts and your switching infrastructure.
Your clients shouldn't respond to the probes. If they are, make them stop. Your servers probably have to respond. If you have not already, you should make very sure that your switching infrastructure can't bleed packets to the outside world. Yah, I know, people tell you to send out 'fragmentation needed' but, you might have to chose between big packets and survival. Be nice if you only need to bleed 'Fragmentation needed' to a few specific external hosts and could discard it (and everything else from your switching infrastructure.)
One way you can you can mess with their heads (assuming they care about your switching infrastructure) is to modifying your border to discard any packet with a low hop-count. The apparent radius of the internet is currently a little over 16 hops. Nothing legit (except traceroute) generates packets with less than a 32 TTL. So, you can arbitrarily discard any packet at your border with a TTL of 8 to 12.
It messes up your ability to trouble-shoot your network from the outside using traceroute but if the choice is that or survival...
I've never been mapped by anything that big. We would see it in our darknet (non-allocated IP) sensors. Lucky you. Brace for impact..
I expect they will get to my institution eventually.
We've seen an explosion in hacker activity in the last week. All kinds of crap. The most unsettling is a series of compromises that carefully scan a locally attached /24 for 139, 445, 3389, 5900 8080, 40080. C&C appears to be innoculous accesses to local Akamai hosts. Almost impossible to spot.
Thanks for the heads-up.
Miles
Translation (Score:4, Interesting)
Vertical stripes may be from spoofed addresses -- nothing from real sources, even botnets, can be that uniform across the whole address space. It would make sense to check how much of traffic comes from unallocated address space, as packets from there are guaranteed to be spoofed. Why would anyone do such a thing? As a direct portscan it would be useless (he can't see the responses), however it might be used as a smokescreen to hide a real portscan or attack from some of those addresses. It may even be an attack that floods the DNS servers with fake responses in the attempt to poison DNS cache, thus redirecting some of the traffic to the attackers' addresses.
Then, after whatever kind of discovery was completed, you have seen some targeted host scans, [D]DoS attempts or actual exploits causing large amount of traffic (horizontal stripes).
Another possibility is that those packets are responses caused by something on your network being coerced into sending packets uniformly to the whole address space. It may be something as stupid as a web page with random redirects, however more likely it is a worm on some of your computers looking for other members of his botnet. After such discovery some hosts joined the botnet[s], producing horizontal stripes composed of traffic from other botnet members.
The reason is quite obvious: (Score:2)
and sent
several million
of them to a company called Quova, who gave me back full location info on every 40th one.
Well, there you have it. Unless you can prove, that that filtering that Quova does, does not influence your results, you can’t really draw any information from it. Could just be selectivity, applied by Quova. Or a otherwise bad filter.
Only if you are safe in that regard, would you first have to look at the actual outgoing traffic, in case there are correlations. (Which, considering the data, seems very likely.)
Statistically flwed? (Score:2)
It appears that the big countries, like china, and india shows up with more hits than the small countires like angola and cuba.
I wonder what that can mean? Is it similar to the statistical fact that most truck accidents happen in US made trucks?
In the latter, until you factor in that 95% of US trucks are made in the US, you have only meaningless statistics.
It seems that current incarnation of this analysis tool suffers the same flaw.
Re: (Score:2)
Not really. It makes green phosphor look like laggy shareware. Somthing with no effort spent on beautifying the interface and even less effort spent on cheating enough to make it visually smooth.
It made me think, "that's a really cool idea. If I had to do that kind of visualization (large dataset over two independent variables), I'd definitely be interested in something like that. But done well, instead."
Re: (Score:2)
Better colors, please. Perhaps labels for the axes, as well.
Re: (Score:2)
"Whaddayou think *this* is?"
"It's something like a man's penis, only smaller."
(Spider Robinson, "Fivesight", _Time Travellers Strictly Cash_, no commercial association)
Re: (Score:2)
I am not sure what is unusual about this, this is the type of thing you see when you watch a big firewall's logs, I used to parse through a big checkpoint firewall's logs with all kind of trending software all the time, and you always see strange trends like this. There could be all kinds of reason why a certain counties accesses your network or webpages at a certain time of day everyday, not to mention botnet activity or really just servers scanning for open ports etc... The vertical stripes would mean tha
Re: (Score:2)
Were you watching E! ? Office Space just ended, and I was thinking the same thing!
Re: (Score:2)
I used to date a girl who crewed on Office Space; do I get karma points for that?
Re: (Score:2)
That's pretty normal...
aptitude install denyhosts
should give you some relief by adding firewall rules against hosts that blatantly try to brute-force your machine for weak ssh passwords.
Re: (Score:2)
I'm prone to Samhain's SSH brute-force blocker script; I use the tcpwrappers approach myself.
Re: (Score:2)
Where can one get a list of IP addresses for countries like China and India so that server admins like myself can block these countries entirely?
Google can tell you within minutes what IPs ranges correspond to non-US locations. Here's one such list that's reasonably close. http://www.experts-exchange.com/Networking/Misc/Q_21787352.html [experts-exchange.com]. You should also be blocking bogons (address that you shouldn't see on the internet such as unassigned ranges) http://www.cymru.com/Documents/bogon-list.html [cymru.com].
Keep in mind that blocking all foreign IPs isn't foolproof as some US clients may still end up going through a foreign relay or some sort of proxy. Also syste
Re: (Score:2)
Why would you want to do that? You don't expect evil people to use botnet nodes in every country?
Simply this: If you don't expect any traffic from foreign countries, then it's safe and prudent to block traffic from foreign countries. It's the whole least-privileges approach applied at the firewall level. For example, you might have http/https accessible from anywhere, but VPN is only allowed from within the US where your sales staff is reasonably expected to travel.
You're right that it's not foolproof, given botnets and compromised computers within the US. Still it's a layer of security that can im