Handling User Grown Machines on a Large Network? 611
matth asks: "Recently with the outbreak of the MSBLASTER worm and the startup of the college semester here in the US we've been hit by a big problem here where I work. Many students are bringing in machines from home, often times infected. The infections are so bad that they bring the whole network to a crawl. Yes, you can install ACLs on edge routers and put a router between the dorms and the rest of your network, but it still brings the dorm to a crawl. You can make sure people install the patches, but what if someone re-installs Windows, or brings in another machine, and what about NEXT year? From the Slashdot community, how have sysadmins out there dealt with this? How can you manage each machine in a network such as a college, where people are bringing their own machines in from the outside? ACLs on routers... but what about for the segmented network?"
No more (Score:1, Interesting)
responsibility (Score:4, Interesting)
If you make them bear some financial responsibility for not checking their machines first this might help.
diversity and not allow attachments (Score:2, Interesting)
Email should be used for communication, not for transfering files.
CB
Deny them DNS services (Score:5, Interesting)
Chris
I'm actually wanting to know the same thing, but.. (Score:5, Interesting)
I'm a student at a university whose dorm network got nailed by blaster something fierce. Almost as bad as it was Klezed a couple years before. Anyways, because of all of this, the sys admins decided to completely eliminate the dorm network from the upper campus one - also cutting off 'net access - during school hours. This is a real big pain in the butt, and I'm actually hoping there are some great answers in this topic so I can give them to my sys admin.
Of course, compounding the situation are seemingly (dunno if they actually are or not considering I've never even SEEN one before) incompetant dorm techs taking an entire day to clear out just one dorm building of ~50 rooms (2 people per room, but often less than 2 PCs per room...). Considering Blaster only affects 2000/XP/2003 machines, that means that the roughly 50 computers running those took 8 hours to clean? Something seems wrong here.
I'm just annoyed because my room (along with my entire hall since I'm the resident 'hey, call him!' computer geek and have patched everyone) is completely free of blaster and its ilk, yet I have to deal with the people who either don't know to patch Windows often, or don't care.
How about this one: What can a STUDENT at one of these schools do to help? I've tried teaching as many people as possible about computer safety (take a health classes' STD safety course, apply to computers basically), and I'm ineligable to become a dorm tech right now... anyone?
mac address registration + managed AV software (Score:2, Interesting)
1) Require all machines to register their mac address via nice gui or website. This way when you use all the rest of the stuff mentioned here (snort, etc) you can easily track the student down.
2) Run snort, router, acls, etc in a way to automatically blocks infected users. Or at the very least it should at least alert you of them. But blocking is best so that they dont spread the infection further on your network or to the internet via your fat pipe.
3) Buy a site license of the managed versions of Norton Antivirus for the dorms and hand one to every student as they walk in the door. Once they've installed it you can force the updates on to them.
Re:forcefully (Score:3, Interesting)
Great idea, but... (Score:5, Interesting)
I've found that to be very common (including the Uni that I'm typing this at) since it is MUCH easier to set freshman up on movein day.
Also, certain things do not work when you start logging onto domains. Example: XP's fast user switching. You'd have students complaining about the administration restricting their rights to their own computer, blah blah blah... then on top of it, automatically patching something. Legal nightmare. Works great for lab PCs, horrid for dorm PCs.
Good question (Score:3, Interesting)
I'm assuming you have each computer connected to a central switch, right? What I would do is block all communication between the PCs on the network. Allow each one to get out to the internet through the firewall, but block them from connecting to each other. That would give them the ability to browse the web, check email, instant message, etc., without needing to worry about them setting up servers, file sharing, and trading viruses, etc., between each other. It's heavy handed, but at least you're still providing the service you're supposed to (internet connectivity).
Just a thought. I'm not completely sure this is even feasible with a switch, but I would think so.
Re:Maybe give out some info to the people? (Score:3, Interesting)
Inspection (Score:5, Interesting)
This is of course great in theory, until a week later when someone formats, 'forgets' to patch, brings their computer home, gets re-infected and comes back to school.
Until patches become mandatory for many of these users, there is no way to prevent such a thing... short of finding the virus writers and skinning them alive during prime time, that might make some of these script kiddies think twice before doing what they do.
Re:Domain logons (Score:5, Interesting)
Re:morons (Score:1, Interesting)
I doubt this was what the original poster meant though.
Here is what we do (Score:5, Interesting)
This segmentation has helped dramatically. At one point, we were blocking nearly 800,000 icmp echo requests outbound/sec across all interfaces. Now? around 1k/sec. And that's over the last week.
Now if I could just get past the residents who:
1. Don't fix themselves because it was too much to read.
2. Don't know how to use a web browser
3. Don't know what a scroll bar is (!!!)
4. Don't contact us for help, but instead go to the President and Provost's offices.
Hang in there, segmentation helps dramatically.
try a LINUX FIREWALL for BLASTER PROOFING YOUR NET (Score:2, Interesting)
Public humiliation (Score:5, Interesting)
Naturally, if you're the BOFH type of network admin you can skip the first part
Burn, burn, burn those patch CD's (Score:3, Interesting)
Recently, we've begun deactivated the ports of people who we've been able to trace the worm back to, having them call us, pick up the CD, install the patch and then having an RCC verify that the patch is installed before reactivating their ports. We've also closed off the ports that the worm is known to propagate through. We've still taken damage as a result of it, but I think we've managed to minimize it somewhat. In the meantime, I've been trying to convince the Mac users I support that they're not at risk. If you say, "impossible" enough times in a row, they start believing you.
Re:Post lists (Score:2, Interesting)
We wouldn't have done anything to him, but network performance went up a little.
Anyway, I think that the list-posting idea is ten times better than any of the other suggestions I've heard so far.
To start with .. (Score:3, Interesting)
MAC address filtering would bring out at least one privacy advocate complaining about rights, and absolute Nazi like controls won't fly at a public institution.
Everybody seems to be advocating the staff doing stuff, do they have the resources to handle every little issue a student comes up with?
VLANs with heavily controlled QoS would help. I also like a script forcing certain patches.
Could the school get a license from an AntiViri company to cover all students, force everybody to run it as policy, script the updates, IDS to ban infractions by switch port or something with would f%$k the student because it might take a week to get around to turning the port back on.
Re:forcefully (Score:3, Interesting)
That's when you set forth the rules.
Windows 2000/XP only, if it's a Windows environment, or MAC otherwise. Any machines found online that violate the policy will be denied access, and the violaters fined.
I know of a couple of small colleges that are MAC only; they don't support Windows machines of any kind. To ensure this, you buy the computer when you start your term--it's part of your tuition and fees. This way, no one brings in anything unauthorized from home.
Re:Ban 'em (Score:4, Interesting)
Re:You could just... (Score:5, Interesting)
It all started when I helped a friend install Linux on his new computer. Unfortunately, in addition to installing a DHCP client on his machine, I had accidentally flagged the DHCP server to install as well. What happened was that the DHCP server software on his new Linux box was challenging the Windows DHCP server that the dorm was using, and his machine won -- even though his DHCP server wasn't properly configured to hand out IP addresses to other clients. So, all of these other 900 students would turn on their computers, which would send out a DHCP request, and they would get a response from his computer instead of the real DHCP server, thus causing their computers to give up trying to connect to the network. Ironically enough, his computer connected to the internet fine, as it was the only one connecting to the real DHCP server (I guess that explains his super-fast connection during that week).
Anyway, we had no idea that any of this was happening until we headed back to his dorm room one day, and found three network services guys looking in bewilderment at the computer (they had never used anything but Windows, so they had no idea how to fix it). They claimed that it took them a week to isolate the problem to his machine. They explained what was happening, and it then hit me that the DHCP server was also running on his machine, so I logged in, apt-get removed it, and the problem was immediately fixed. Not in their eyes though, as they made us talk to the head guy at network services... He gave us fair warning that if we did that again, our access to the network would be revoked (and rightly so!).
The obvious moral of the story is, whereas most OSes give you just enough rope to tie a knot, Linux gives you enough rope to hang about 900 people.
PFC (Score:2, Interesting)
set security acl ip WORM deny icmp any any echo
set security acl ip WORM permit ip any any
commit security acl WORM
set security acl map WORM 1 (or whatever VLANs you have)
If you have some other product for LAN switches, shame on you! Well, there probably is a similar filtering capability if you have the right components.
I've been involved in cleaning up after SQLslammer and Nachia on a rather large network. In both cases, I found that router filters were difficult to implement without causing the filters to kill the routers (except on a few very new high-end routers). The PFC claims to work at wire speed. In practice, I've had a hard time proving them wrong on that.
This filtering technique will allow you to drop packets as soon as they enter the switch. Basically your doing a L3 or even a L4/L5 filter (tcp/udp with port) on a device that is really operating at L2.
A couple things to note, you can't log the packets and once you put the filter in place you probably won't be able to determine who is sending junk, but you shouldn't be patching machines for a worm by going after the infected ones... every machine in the network needs patched before you lift filters regardless of whether the worm is still in your network or not. If not, it will be back!
Re:Simple... (Score:5, Interesting)
I guess that's what they get for forcing everyone to migrate to XP last year...
Re:forcefully (Score:2, Interesting)
The dorm network is slow not because... (Score:1, Interesting)
Here's what we did. not perfect, but: (Score:4, Interesting)
We did some testing with the Blaster patch before we encouraged our users to download it; I always check Bugtraq, personally, before I put anything on a machine I'm responsible for. Once we decided it wasn't breaking anything (at least it didn't break anything for us) we burned it to a whole bunch of CDs (with the Symantec removal tool, the Win2k patch, the WinXP patch, and the WinNT fix). Each RA/helpkid/tech also got a corporate edition of NortonAV on a disk (we have a site license) with instructions for students on how to update their virus definitions.
Each RA got this disk. Each help desk kid (there are about 15 student help desk kids) got one, and the other five PC/net techs (other than me) got one. We marched around campus for about a week wearing very visible "TECHNOLOGY SOLUTIONS CENTER" T-shirts and essentially infiltrated dorm life with our antivirus software.
Were there huge network slowdowns? Oh yeah. For the first day and a half when students came back there was little, if any, network connectivity. But the RAs were adamant about having the kids run the patches and install NAV. Did we use guerilla tactics, like disabling network ports or confiscating network cable? No, not at all. We just made help extremely visible, and with a horde of student tech workers getting $5/hr, it was not so bad for cheap labor for the college, either.
You might bitch and moan and say that a college kid with a virus will never go talk to his RA, but we had mandatory floor meetings for every floor for every hall across campus, and when you've got 20 kids and one RA, it's pretty easy to reach the end users. Users only understand that "my computer doesnt work", and you can bet that a college kid at a small, tech-oriented campus will go see his RA if he knows his RA can help him. (If the kids think the RAs are totally bogus, then there's problems with administration that have nothing to do with computing and is for another thread entirely.)
Do these tactics make Mac/Linux users feel discriminated against? I saw some whining in the comments about this, but guess what: Even if an RA is minimally intelligent in the realm of computing, he can PROBABLY tell a Mac from a PC. Mac users get left alone (like me.)
Full network connectivity returned at about 9 in the morning on the day after move-in. (you'd be surprised how fast 30 RAs and 21 tech kids can move.)
You might also bitch and moan and say that students shouldn't have L2 domain admins. Okay, I can understand that. One kid got forcibly removed from our staff last year for leeching software off a drive he had permissions to, so no, it's not a completely perfect solution, and a lot of trust is involved. But it worked okay for us and minimized a lot of headaches.
DHCP, ARPWATCH and managed switches. (Score:2, Interesting)
1. Only machines we want to have on our network are there. This usually means that we give out IP addresses in exchange for the basics - a MAC address and the location of the machine. Higher levels of management of clients has its costs, so that'll be down to the individual manager to decide (for instance - only machines running OS xyz, or only machines we have root/admin access to, only machines built with our spec/OS and connected to our auto patching architecture, etc).
This means that we can, in extreme cases, remove someone from the DHCP lists, and flag their MAC up in arpwatch. In the case of "students arriving at the start of term", there is quite a flood of applications at the start of term - combined with teaching them how to find their mac address (solved with a flier in their matriculation pack). After that, it slows to a trickle of applications.
2.With managed switches (and really, who DOESN'T use managed switches in large networks?) troublemakers can be sought and disconnected in times of strife. You have their IP address AND know which switch/port they're on (through the MAC/location registration process). It really is up to the user to come to the IT staff in the event that their connection drops. We have disabled specific ports on network switches in some cases, which is a far more useful solution than removing DHCP entries, but for public areas the DHCP block is what is needed (laptops in libraries for instance). Smart users will get around this, but it's not the smart users you're worried about. They know how to patch.
When it comes down to it, make one simple rule - network access is a priveledge, not a right. Our entire university wide IT infrastructure is built on this philosophy, and as a result the onus is on your users to behave in a responsible way.
--
VPN isolation (Score:3, Interesting)
Because you cannot control who uses the wireless zone, it's treated as potentially hostile or untrusted and users must authenticate to a VPN.
A nice side-effect of this is that the VPN in Windows routes all traffic via the VPN, letting them apply all sorts of policies "port 4444, I don't think so...". Blaster only affected users silly enough to bring in an infected machine.
Perhaps a similar setup for the untrusted wired network too?
Re:Simple... (Score:3, Interesting)
Re:responsibility (Score:3, Interesting)
Schnier (sp) has been singing this song (from a corporate standpoint) for a while: the only way M$ will secure their products, and the only way companies will think about secure networks will be if they are held accountable for damage they cause.
He argues that security will be forced not by laws, but by insurance premiums. You (big corporation) are liable for propagating virii (civil claims of contributory negligence), thus take out liability insurance; Run an insecure OS, and you get higher premiums. Thus, you tolerate less shit from M$, and they have to shape up.
Notice that he isn't claiming that M$ will be held directly responsible (Would make as much sense as holding Cox responisible for local exploits in the kernel), but that companies with eqv. of ISO-9001 security practices will get lower premiums, and the choice of OS will factor into those premiums. So in order to remain attractive OS choice, pretty icons and talking paperclips will no longer suffice.
I wonder if Billg did sense a change in the wind towards something like this, and thus sent out his famous security above all else memo.
Re:No more (Score:3, Interesting)
Of course this only works if you have managed switches/hubs, a bunch of spare 486's (pentiums would be better) and a day or so to set it up. The nice thing is that if the 486 fails (only one has so far), the network stays up.
This has stopped 99% of malicious traffic dead in it's tracks.
Timing the cable failures (Score:3, Interesting)
It's amazing how many students seem to have wiring problems after they crash the local nets on certain campuses. I just wish the same approach could be applied to home users.
Many of the worms and viruses that bog the net have had patches for months or even years. I say if the patch was out three months ago, cut the user off at their ISP -- permanently.
You can't drive without a license -- if you can't update, you don't know how to "drive" the internet. And no, I really don't care about the "rights" of the brain-dead to access public resources.
Even my techno-illiterate parents know enough to keep the virus files and patches up to date -- because they were taught before the machine was ever plugged in to the 'net.
Re:Timing the cable failures (Score:2, Interesting)