How Can You Screw up a Network?

aztektum asks: "Like a lot of Slashdot readers, I have setup my own home network. It isn't tricked out with all the fanciest hardware, but I do have a switch, BSD based firewall, I have configured e-mail (again on BSD), NFS and Samba, as well as remote access services like SSH and FTP. Now my line of work isn't networking or computer related at all. This is a personal hobby and a fairly new one for me (relatively speaking compared to others). I'm looking to learn more about managing problems with networks, but have no idea where to start. With such a small setup and only supporting two users (myself and a roommate) this isn't exactly enterprise level with enterprise level ups and downs. What are some ways I can screw up my network to troubleshoot problems and gain some insight? Also, what are some reference materials that you have found to be educational with relation to network administration?"

Clone some ethernet NICs

[HotNeedleOfInquiry]

With the same MAC number and try to use them.

Just wait, it'll screw itself up.

[dtfinch]

Eventually hardware fails, always. Notice the signs that something is about to fail so that you can replace it when in a timely manner and with little downtime, or none in some cases. If you know you'll have to take a server down, figure out how to replace it without data loss or downtime. With an MSDFS root, which Samba does well, phasing out a dying or obsolete server is relatively easy. Otherwise, you'll just have to fiddle with the DNS and maybe give the new server the same IP. You can also look into clustering, but the cost and complexity can be prohibitive for smaller companies, and possibly for home experimentation.

Always keep good backups. If someone comes to you and says they deleted an important file last week, be able to get it back without a full restore. Also, be able to do a full restore of a server, and know it'll work. If the server catches fire, have a plan to replace it within the hour.

Make some ethernet cables. Buy some raw cable, and end plugs, and put them together the right way. The ordering is very important. Not only must each end match, but the color coded twisted wire pairs must be arranged in a certain, non-obvious way or else you'll experience severe noise and crosstalk problems.

Mix older (bargain) gigabit hardware, different brands. Some card-switch or switch-switch combinations have slightly hard to diagnose problems. If you ping, you'll have zero packet loss. But if you transfer a file, sometimes speed will drop down to 20kb/s or so, and it'll only happen in one direction. I've seen buggy drivers cause this too. When packets are sent in rapid sequence, every other packet is lost, and the send window shrinks until it's sending only one packet at a time, and waiting for an ack before sending the next.

Get a really, really long ethernet cable and use it to plug a windows pc to a switch. Let it autodetect the speed. If it's long enough, it'll still detect 100mbit or 1 gigabit, and then fail to connect. You'll have to force it to 10mbit, or get better cabling, or use a switch, hub, or some other repeater to break it into two short connections.

Again, get a really long ethernet cable, and put a sharp kink in it. You do this by making a small loop, then trying to force it straight by pulling instead of carefully undoing the loop. Line quality will suffer dearly, even though you may still be able to connect. The best fix is often to buy a new cable. Any sort of sharp bend will cause problems.

Have fun with Windows name resolution. Windows PC's seem to be able to find each other pretty well just using WINS or broadcasts, but only after checking DNS first, which goes out to your ISP's servers if you don't have your own DNS server(s) set up. These requests tend to fail almost immediately without delay, so the issue can go unnoticed. This allows your network to be hacked a bit more easily from the outside, and also allows internet problems to translate into delays in local name resolution. This sort of problem is easy to create and easy to fix, and plagues some small businesses that lack experienced or knowledgeable IT staff.

The problem with managing problems...

[mysidia]

Is you need more nodes and more complexity -- your network is too simple, so there is fairly little that can go wrong compared to real networks.

Try reinstalling and switching your systems' OSes, especially the BSD firewall's -- provided your hardware and wiring are good, the OS is the most likely thing to mess up anyways.

I.E. Are you sure BSD is the best OS to use for that firewall? Maybe trying to run the fireewall of of VMS or something else could have interesting results.

Increase the demand on your network is the main thing; if you don't get to have problems, you can always try to tune for performance, stability, security, by switching things around and changing configurations --- try to find as many configurations that work as possible and figure out what works best.

Figure out the way to add as many units as possible and to make the network arrangement as complex and spread out as possible --- the more complexity, the more devices, nodes, etc, involved -- the more likely _something_ will go wrong; find a way to get 3 or 4 windows machines in there with serious demands on them, and something's almost certain to break.

How do I screw up a network?

[anticypher]

By touching it. There's always an assistant named Murphy looking over my shoulder, but she usually waits until I'm in the shower or leaving on vacation before "helping".

Your question is really "How do I introduce layer 1 and 2 problems into my home LAN, since all layer 3 routing is limited to a NAT box with a single default route?". The lower layers are a good place to start, since half of all your problems come from there, save the routing problems for a future ask/. question.

Others have already pointed out the joys of having dueling DHCP servers, subtly mis-configured DNS servers, overlength cables and the like. Keep an eye out for others throwing out bad ethernet cables with broken catch-tabs, frayed insulation, sharp kinks or intermittent wiring, and put them into critical places in your network. They may not fail right away, but will wait until you host a lan party at your place or you have a few hours to get a report done. Her name is Murphy, she's a bitch and she'll gladly pay you a visit when you least want her around.

Start to learn what kind of traffic is on your local network. Get ethereal, snort and ntop running, and see what the packets look like. Chances are you'll find some things that look suspicious, you'll learn a lot by figuring out how DHCP handshakes work, how often ARPs happen, what other protocols are on your net besides IP. Since you are running a BSD, you can pretty safely put the box on the outside of the firewall (it probably is the firewall) and watch all the constant crap scanning the internet. That's a great way to learn how to tune firewall rules by hand, and you will break things along the way.

To really start to learn how layer 2 networking almost works, grab some old cisco kit off of eBay. I've seen 2900 switches for US$20. Plug something slightly pro into your network, start simple, just get a cheap used cisco/hp/3com switch off eBay that can do 802.1q vlans, spanning-tree, and snmp. Your BSD ethernet card can be configured to do .1q, so there is a lot of learning there by creating multiple separate vlans, one for each machine. A single router and switch with 802.1q vlans can make some pretty complicated networking topologies without massive amounts of wiring. Then you can break your network by plugging a crossover cable into two ports and watching spanning tree open up one of them. Bonus points if you create a topology where by creating a spanning tree loop, your main gateway or server port is the one that goes into blocking mode (you need a minimum of two switches to do that).

To break things in subtle and non-obvious ways, try changing your address ranges from the usual 192.168.0.0/24 to something unusual like 172.31.255.16/29, doing the netmask/subnet/broadcast calculations in your head for practice. Then misconfigure the netmasks on each device, notice how one machine can ping another, but not the other way around. Try building multiple separate segments rather than multiple subnets on a single wire, this will force traffic to use your router, and really show netmask problems more clearly.

To really break things, instead of using reserved RFC1918 addresses behind your NAT box, use a public network range like 66.35.250.0/24. Sure, it will break one major site, but you shouldn't be wasting your time there :-)

Since you already have a BSD running, do you leave it on 24/24? If so, its time to start loading up the real tools like cacti [cacti.net], nagios [nagios.org], and smokeping [ee.ethz.ch]. It helps if you have an SNMP capable switch on your network, but configuring your own SNMP [sourceforge.net] can be quite a learning experience as well. With graphs showing what is happening on your net and the internet over time, you will start to see the cycles of congestion every evening and maintenance times every sunday at wee hours. The most frustrating problems in networkin

Combine your ssh remote login with poor passwords

[jonadab]

Your ssh remote login *will* get noticed by port scanners, and both dictionary and brute force attacks will be made against it, particularly if it is running on the standard port (22 IIRC). You can help the attackers to compromise your system if any of the passwords of any of the users who can log in in this fashion -- especially passwords on accounts the attackers can guess must exist, such as your own preferred username or an account that is usually present on most systems, and extra-especially the root account -- are attackable via dictionary or brute force. For instance, if one of these users has a password that is only ten characters long and contains only letters, that is a potential point of entry onto your network. On the other hand, if you want to *prevent* them from getting in, use passwords that are longer, contain non-alphabetic characters, and not based on dictionary words (but pronounceable so that you can easily remember them), e.g., passwords like Frolliga_Bruckenovich or grazzivian-CHOXXI or SpoyBan8CritNox or cetera. (I don't mean these specific examples, but hopefully you get the idea -- passwords that are hard to brute-force don't have to be hard to remember. The more paranoid you are, the more syllables you add, and remember that a certain amount of paranoia is part of any sysadmin's job description.)

Another thing you could do to allow attackers to gain access is to completely ignore security bulletins and never install updates.

Start from the bottom, and work your way up.

[amper]

Wow. Taking a brief look at the responses here, I can't believe how complicated most of the answers are.

You want to know what makes a network tick? Start from the bottom and work your way up. That is, follow the OSI Protocol Stack Model, and start from Layer 1, the Physical Media, and learn why it is that Ethernet (or your choice of PHY) works the way it does. Then move up to Layer 2, the Data Link Layer, which in the case of Ethernet would be CSMA/CD, then move up to Layer 3, the Network Layer, which in most cases these days is TCP/IP (though TCP/IP really sort of covers Layer 4, the Transport Layer, as well).

Allow me to suggest the many excellent books by O'Reilly that will tell you everything you need to know.

Do not use the Cisco or Microsoft books. While most of the information there will be correct, some of it will be specific to Cisco and Microsoft's proprietary implementations. I feel it is always best to learn the generic, standardized protocols before branching out into proprietary protocols.

Check out these books from your library, or buy them. Used or new doesn't really matter all that much, as the basic protocols have not changed much in the past 15 years or so.

1. O'Reilly - Ethernet: The Definitive Guide
2. O'Reilly - Internet Core Protocols: The Definitive Guide
3. O'Reilly - TCP/IP Network Administration
4. O'Reilly - Building Internet Firewalls

That will get you started. Then, you might want to know something about other types of networks:

5. O'Reilly - 802.11 Wireless Networks: The Definitive Guide
6. O'Reilly - T1: A Survival Guide

With those six books, you'll have a solid grounding in how networks network, and how internetworks, internetwork. Once you have that, you'll have a pretty good idea of how to screw up a network. You'll also have a pretty good idea of what more advanced topics you'd like to know more about.

One old book that is out of print and difficult to find that I highly recommend is Inside AppleTalk, 2nd. Edition, from Addison-Wesley. It's the definitive book for AppleTalk, and you might want to know about AppleTalk, even though it is falling out of favor.