Finding the Truth Behind Cable Modem Traffic Bursts? 31
Techi asks: "I help to support a small cable modem network in Kansas, and we keep having recurring problems with all the modems in a node bursting extreme amounts of traffic for a period of about 30 seconds. At the end of this 30 second period, the upstream port that the node in question is connected to dies under the pressure. We have recently implemented a fix to keep people from uncapping modems in the form of a config file update from our DHCP system. I know we could have done it differently, but it wasn't my decision. Does anyone have any idea what could be causing 70 or so modems at a time to suddenly erupt with outgoing traffic nonstop until the upstream dies?"
ah... (Score:3, Informative)
fp?
Try monitoring (Score:4, Interesting)
I've seen something like this happen and suspect either dumb/misconfigured DHCP clients, an election process run amok, or some sort of ICMP flurry. No proof either way, since in this case I'm just a user & I just wait it out.
-- MarkusQ
P.S. I have noticed an interesting patterm to the timing though. You might try looking at the times / dates of past events to see if that suggests anything (and it can often suggest a lot).
Re:Try monitoring (Score:1)
Possible spyware and/or application updates? (Score:3, Insightful)
For example, all the Macs on my network are set to query Apple's network time server at midnight daily. And on my Windows machines, Windows Media Player is set to check for updates weekly. The amount of traffic involved in either example should be minimal, but you never know what's borked. There was a story here recently about some versions of Windows and MacOS causing too much DNS traffic, so it could even be something at the OS level.
Is this a recent phenomenon? Brilliant Digital said they were going to activate their leechware in May, and May is now more than half-over. Maybe they've flipped the switch and all your users with KaZaAaAaAaA are now sending uberpackets to BD at predetermined times.
Are there any specifics as to where the traffic is destined? Is the traffic burst from all of the nodes going to the same host, or to the same port on multiple hosts? Are ports 25 or 119 involved? There's been a fairly nasty Hipcrime attack (usenet sporgeries) over the past few days, and spam is always a problem; both of these abuse broadband relays as much as possible. Lots of possibilities, I guess - would help to get some more details, if they can be provided.
Shaun
Re:Possible spyware and/or application updates? (Score:2, Interesting)
Re:Possible spyware and/or application updates? (Score:2)
If yes: Can you find out the destination of the traffic? If it all points to a single or a handfull IP#s DDOS is very likely. (a tool like ntop is handy to find that out).
Is it TCP or UDP? Are the sender adresses OK or spoofed? What service are the packets targeted at (port numbers)?
Some FPS (Score:1)
Re:Some FPS (Score:2)
Re:Some FPS (Score:1)
But having 70 users all knowing the rate setting and all going to play at the same time seems a bit weird.
Re:Some FPS (Score:1)
Re:Some FPS (Score:2)
As others have suggested, you really want to run some kind of a network trace (e.g. tcpdump) and have a look. My guess is that it should show up what the hell is going on.
Re:Some FPS (Score:2)
You need to capture the data (Score:4, Interesting)
It seems to occur when a switch somewhere gets it's MAC table corrupted somehow and starts squirting rubbish onto the network.
I accidentally caused one of these at my uni, by changing the MAC address of my netcard, it brought down the whole network for hours, the switch was continuously broadcasting the last packet it saw.
They never found it was me though
Re:You need to capture the data (Score:2, Funny)
Well, that is, until now.
Expect to be called in first thing next week to explain.
Re:You need to capture the data (Score:2)
Re:You need to capture the data (Score:1)
Re:You need to capture the data (Score:2)
The spammed packets are probably udp, although it isn't impossible for some other broadcast-type packets to cause this (I'm thinking netbios/netbeui)
Best thing to do (if it's possible) is install some kind of packet sniffer (tcpdump/ethereal for unixoids, dunno about other OSes) on a laptop and plug it in at various locations. Sometimes just unplugging the patch from the offending port for 5sec clears this kind of thing up, sometimes you need to reboot the switch.
also cap locally (Score:1)
Sounds like a broadcast storm (Score:4, Informative)
One (misconfigured) machine broadcasts data (say, NMB update) with a source address of the broadcast address - everybody on the segment replies, (which causes everybody on the segment, including the misconfigured one to reply again, ad infinitum) - result: segment meltdown.
As someone else pointed out, a traffic monitor would be your best bet - you don't need to capture all of it, just the first part, to see what's starting it up - then you can decide what to do.
Forget trusting the modem (Score:1)
|---- Linux box (1 )--- [Node]----(internet/whatever)
|
|
|
--|-------|--------|
[modem] [modem] [modem]
Make the Linux box limit the bandwidth for each modem (TCP shaping/QoS) and then, if a user uncaps, you can even automatically cut them off and alert the node which modems need to be recaped (DHCP)
Meanwhile everything still works as usual, only that the linux box drops a lot of "extra" packets from uncapped modems.
I left WinMX running for like 3 days straight (Score:1)
Questions ? (Score:2)
Are you certain the modems have not been compromised ? Are they all the same type?
I'd suspect a DOS attack by one of you customers pissed at being capped.
check DHCP lease duration (Score:1)
For example, a 4 day lease duration would cause renewals at 2, 3, and 3.5 days into the lease.
You may want to search your traffic logs for one particular client and see if its traffic follows this pattern. Otherwise, I agree with everyone who said "sniff the traffic".