Phase Changes While Forking Servers in Perl?

For those of you who think Ask Slashdot is lacking in the technical questions department, I hope this query from (yet another) Anonymous Coward will spark your interest: "I have written a forking server in Perl that is exibiting a phase change in its response characteristics around 10 simultaneous connections. As the number of simultaneous connections passes through 10 connections the connection time increases by two orders of magnitude and becomes very random. At the same point the transmission times drop slightly and continue to have small standard deviations. Although I posted this question to a variety of Perl and Unix usenet usegroups over a week ago, I never received a response. Since the most important part of the post is the performance plot, I placed it and relevant code snipits at: http://www.geocities.com/nawkboy The memory, io, and cpu usage never exceeded more than 80% at any time during the tests and were seldom above 50%. The test results were very similar between remote and local clients. The server is running on a Sun Enterprise 420R running Solaris 2.6. I do not understand the behavior and would appreciate any guidance in understanding it."

Relevant questions for you.

[AtariDatacenter]

Here's a number of things. How many CPUs does this 450 have? How much RAM? Are you using "lockstat" to look at the locks during all of this? How much are you swapping to disk during this? How much of the CPU time is system time, and how much is user time? If the system time is high, get a copy of the "SE Toolkit" (available at the Sun website) and monitor your mutex [multiple exclusion] contention within the kernel. See if it is excessively high. If you REALLY want to do something painfull, run a "truss" on your server process to see if you can gleem anything useful. Same thing with a "snoop". See if you can use it to identify a point of delay in the communications. Probably not an issue, but go ahead and use "iostat" to look at your disks.

There are lots of things you should be able to do to pin-point it down a bit. Hope this helps?

Re:Relevant questions for you.

[randyflood]

I would look into memory used as my primary focus. Have you tried running it on another machine to see what the results look like there?