Simulating Network Latency?

ixmo asks: "I've just come around an interesting problem: to simulate low-bandwidth network links without buying expensive WAN simulators, I can connect two old Cisco routers back to back with serial cables, and control the bandwidth via the 'clock rate' IOS command, but how can I simulate network latency? Is there some OS tool or patch (for Linux/OpenBSD) that allows for tuning of network delay? Any hints?"

FreeBSD Dummynet

FreeBSD's dummynet can quite easily do it. I suspect the same of openbsd and linux.

What kind of question is this for ask slashdot?

Re:FreeBSD Dummynet

Dummynet can absolutely do it. Put a PC with BSD & Dummynet and two ethernet interfaces in to simulate delay/loss/BW restrictions/etc. Very configurable. You can chose which packets are affected.

Worst problem: fixed delay, not bell-curve/whatever. You can roughly approximate delay variance by several rules of varying probabilities. Also, loss is random not bursty. For most testing, this is fine.

It can take a little while to get used to configuring it. Don't forget to make it act like a network in both directions!

Re:FreeBSD Dummynet

Dummynet can absolutely do it. Put a PC with BSD & Dummynet and two ethernet interfaces in to simulate delay/loss/BW restrictions/etc. Very configurable. You can chose which packets are affected.

You betcha. At a job I previously had, supporting TCP/IP for a large Unix producer, I had a FreeBSD box set up for just this purpose. It only had one ether interface, and was in a different part of the building from the test lab, but that didn't matter; I'd use PPP over TCP to connect to the test machines.

I'd use it to test how different OSs (including our own) would handle different bandwidth*delay products, packet loss, etc. For instance, one I found out that a customer was having problems with a lossy WAN connection to an NT server. I experimented with high packet loss percentages, and changed the rules to narrow down the problem. Turns out that NT won't retransmit a FIN packet, at least not back then.

When I left that job, my coworkers insisted that I show them how to set up that box to run those kinds of experiments.

Quite an educational experience, too. It's one thing to read Richard Stevens describing congestion avoidance algorithms; it's something else to watch them in action.

The solution is obvious.

Use a longer cable.

Re:The solution is obvious.

Use a longer cable.

Or do what my last job did: Use CAT-3! Oh, you're just testing... Too bad they weren't.

In related news

It's good fun telling people who come down here to visit (New Zealand) that the twists in our cat 5 go the other way - because the earth's magnetic field is reversed.

Made all the better when they discover that southern hemisphere monitors actually *are* different for exactly the same reason.

Dave

Re:In related news

Go type 'southern hemisphere monitors' into Google.

Download a copy of SP2

This may be the oddest use for this yet- but any large file would work >200MB. Set up your Serial port for a slow connection, say 110 baud, then start an ftp transfer in the background of SP2 from one machine to the other. That ought to simulate latency errors just fine.

dummynet

FreeBSD has dummynet [freebsd.org]; I'd guess OpenBSD has something similar.

We use Dummynet

We use FreeBSD and its "dummynet" capabilites. (Perhaps other BSDs have this as well?)

You can get m0n0wall and stick it on random hardware. I think you then have to recompile the kernel to enable dummynet.

We use a Soekris 4501. It'll only bridge upto about 50mbit of traffic, but if you want to simulate T1 speeds it'll be fine. Beefier hardware (the soekris box is roughly a 133MHz 486) will probably let you max out at wire speed.

Stand next to the router and...

Stand next to the router and simulate rough conditions... ... yank out the wire a couple times. ... play kick the router. ... simulate lightening by plugging a network cable into a 220V plug ... paint sunspots on your face and spew out some EMF pulses like the sun.

Re:Stand next to the router and...

Don't forget the most fun part of cable simulation: the backhoe!

Dummynet is great, but...

What if you want to simulate two multiplatform networks connected? I think this guy is looking for hardware because the commercial latency simulators are ~$2000.

Simple

Sign up with Comcast.

Linux QoS

Network emulator
CONFIG_NET_SCH_NETEM:
Say Y if you want to emulate network delay, loss, and packet re-ordering. This is often useful to simulate networks when testing applications or protocols.

single-shot delays

I have found kill -s SIGSTOP and kill -s SIGCONT on the server process useful for simulating a temporary network congestion / single packet-drop on a TCP connection.

OpenBSD

For openbsd you can throttle bandwidth right in PF [openbsd.org].

Just cap whichever queue you want at whatever rate you want.

My Thesis

My thesis (which I am only just starting) deals with networking, and one major part of it is in simulating different network conditions.

I was planning to write my own wrapper around the standard socket operations which will add things like latency, unreliability etc etc (for testing the robustness of a protocol).

However I am looking forward to seeing some of the answers here as maybe I wont have to do as much work as I previously thought ;)

NIST ATM simulator

The NIST ATM simulator [nist.gov] (public domain) might be useful. You need to provide some personal info to download it, but that isn't verified.

nistnet

Nistnet [nist.gov] is another tool that simulates delay.

-jim

Shaper

Howabout using a traffic shaper I should do what you want.

Shaper is a traffic shaper and a packet filter for a server and for a gateway. With only limited configuration information, that are to be supplied, this script can control which and how information flow through the box.

http://www.chronox.de/

There are other ones as well type shapper at freshmeat.

use a traffic limiter (shaper)

Use a bandwidth limiter that throttles your send rate. Someone else mentioned PF for BSD. On linux you can use the shaper module. My old notes go something like this, it might still work. Assuming eth0 is your output interface (the limiter works only on outgoing traffic, so you need the other end to do the same):

modprobe shaper
shapecfg attach shaper0 eth0
shapecfg speed shaper0 9600 # in baud
ifconfig shaper0 <eth0-address> netmask <eth0-netmask>
# remote automatic route added by above
route del -net <eth0-network> netmask <eth0-netmask> dev shaper0
# add routes to be choked
route add -host <otherhost> dev shaper0
# or if going through a router
route add -net <othernet> gw <router> dev shaper0

...you get the idea. I wonder if iptools makes this a bit less hacky.

Use Google?

Google is sometimes a very good tool for finding things like this out, without having to submit a question to Slashdot. A Google for simulating network latency [google.com] and a click on the first link turns up a thread on a FreeBSD mailing list that provides the answer: Dummynet.

Found a hardware solution for $10, Ricochet. :)

Connect the machines using PPP over a pair of Ricochet modems, available on eBay for a song. They include a neat little command for developers:

AT~I13 -- WAN Simulation Command and Information Display
This command enables the Ricochet modem's WAN simulation feature.
Syntax:
AT~I13
You can use this function to test various transport protocols in the presence of network delay and packet loss. This simulation only affects the modem's transport modes, i.e., LIGHT/PPP/SLIP/STREAM. If you are going to reset the WAN simulation values, then you should reboot the modem because it is not built to reset and process incoming packets at the same time. WAN simulation affects the processing of received packet, therefore, when testing the simulation needs to be set at both ends of the connection.

The incoming packets are processed in the following order. First, the drop percentage value is checked and the modem drops that N% immediately. Second, the base delay is added to a random percentage of the variable delay. Then the packet is inserted on a time ordered delivery queue. If the variable delay component is great enough, a large number of incoming packets will be reordered.

Note:
In WAN simulation, there are fewer (Time to Live) TTL expirations than in an real network because packets ending up on the delivery queue is not expiring based on the TTL value.

Two Modems...

Two Modems, Two phone lines, dial them either straight to each other, or better yet... into an ISP.. that will give a great simulation. Perhaps even dial up one of them to an East Coast number, and another to a West Coast access number.. Should do the trick

link in Netkit.org

I've seen a tool "delayer", in a paper on www.netkit.org, used for testing TCP windowing negotiation
http://www.netkit.org/docs/netkit-clark-and-nagle- 01.pdf [netkit.org]
JurgenK.

Nistnet

http://snad.ncsl.nist.gov/itg/nistnet/

nuff said...

Here's something that may be useful...

I know dummynet has already been mentioned, but here's some detailed info on how to set it up from comp.unix.bsd.freebsd.misc.

Article: http://groups.google.ca/groups?hl=en&lr=&ie=UTF-8& selm=ucggua1ghi9ic1%40corp.supernews.com&rnum= 12 [google.ca]

config NET_SCH_DELAY

I'm shocked no-one has posted this!

It's been in the kernel for while, though I don't know much about using it. I never bothered even looking at it (had no need) until a coworker wanted to use it (on Thursday) to do some testing and asked me about it.

Here's the chunk of Kconfig:

config NET_SCH_DELAY
tristate "Delay simulator"
depends on NET_SCHED
help
Say Y if you want to delay packets by a fixed amount of
time. This is often useful to simulate network delay when
testing applications or protocols.

To compile this driver as a module, choose M here: the module
will be called sch_delay.

Please reply to this if you have been able to get this working... the tuning parameters to tc we found give errors (and yes, we built installed the latest iproute2 tarball).
Then again, we only spent a few minutes playing with it (he had to leave).

VPN out and back in

If you've got an offsite broadband connection you can stick a PPTP server on (I use PopTop, http://www.poptop.org or http://poptop.sf.net ), try making a VPN connection out to that server and then back in. I've found that operating stuff over VPN at work tends to introduce significant latency (more than running over an 802.11b wireless bridge, which is also quite noticable compared to a hardwired Ethernet connection), so if you can VPN out and back in, you should have a fair amount of latency involved. If out-and-back-in doesn't work, just in (i.e. test machine A offsite, operating via VPN, talks to test machine B which is onsite) should still introduce noticable latency.

(This does, of course, assuming that you're testing with routable protocols)

Do what I do

1. I put a linux box acting as router, firewall, apache server and internet connection on the network.

2. Then I post a story on slashdot with a link to the apache server.

3. ???

4. Instant latency!

(And by carefully choosing the size of the documents/jpegs/mpegs I can actually simulate different kinds of latency!!!)

hub

My idea is this: put the server and the client on a hub (not a switch), and put another couple computers on the hub ping flooding each other. The collisions will make the average latency increase.

I don't know if a bunch of collision will affect the latency, or just the bandwidth though.

Cisco Routers can do this too... but...

It is kind of a pain. You need to adjust QoS settings to cause congestion or delay. 2 ways: One mark the traffic you want to be delayed and then pump a bunch of traffic through the pipe at the same time. Apply QoS to the unmatched traffic (say a ttcp stream is good). This will provide a real world latency and will be hard to predict how much latency. The second way is to manually mess with the low latency queing settings for an interface (decrease queue size is one way), beware this will foobar your performace so keep a backup config or don't accidentally put the router into production. This will let you do more predictable latency tests, but you still may need a stream of data to really bork up the latency. You can always do the bandwidth command on an interface and introduce pinhole congestion as well (may add latency), but that may not be what you are looking for... and I'm sure occured to you already. -Andrew more info available on www.cisco.com http://www.cisco.com/en/US/tech/tk39/tk824/technol ogies_configuration_example09186a008009461f.shtml and search for QoS as well, and then use the techniques in an inverse manner.

IPTables + Userland Queuing

Depending on the precision you need, you could use this combination to create latency and generate losses. I've written a quick hack like this for some tests I had to do, and it works (I did it a bit hastily, and it's sure to to be buggy, but, oh well..)

Does CONFIG_NET_SCH_NETEM work for IPV6?

I am also looking to simulate a network delay (satellite link) and was interested by the CONFIG_NET_SCH_NETEM for linux. Does anyone know if this works for IPV6? If so, is there documentation for it anywhere on how to use it? Or is there any other free network delay simulator for linux/freeBSD/or solaris that will work for IPV6?????

Re:FreeBSD

"BTW....wouldn't have taken more than 2 minutes to Google that!"

The point is to discuss it, not to be a human Google. Read some of the other posts that were modded up.