Measuring Usage of Distributed Resources?

An Anonymous Coward writes "I work in a distributed development environment with some 1000 engineers running Solaris on different hardware platforms. We are initiating a project to move to a centralized environment. One of the biggest problems in this project is to identify the amount of resources (i.e.: RAM, disk space, etc) that the centralized environment should provide to the users. For example, each developer is currently building his code in a different desktop on the network. How could we effectively monitor how much resources are been consumed from each desktop during each build? If we find a way to capture this information, considering that the results were gathered from different hardware platforms, how could we normalize the data to get meaningful metrics that would help us define a solution?"

More info, for interest sakes...

[JMZero]

Sounds like a very ambitious project to me. Any chance you'd explain more about the situation? Why the move? What are you going to do with all the other hardware?

Back in University we shared Solaris machines for development. Right around project due dates, things really dragged as everyone was compiling (and someone was playing Doom on the console).

At least as far as processor time, compiling seems extraordinarily "all-or-none". If your company is like ours (only much larger), you'll have some real spikes when everyone makes one last build just before 5:00. But I guess that doesn't really hurt anything, except perhaps dinner will be a little colder for somebody.

We've tried to guess how much(bandwidth/processing power/memory/disk space) we'll need a few times - and we've tried to be thorough - but the only way we've ever got within a factor of 2 is by talking to someone with a similar setup already running.

That may be harder for you to find.

Good luck.

.

Easy Way Out

[Jester998]

There's a very simple solution to this problem: Imagine your 'dream box', then tell management you need that.

"Yes, Mr. CEO, we need a system with 16-way SMP, 64GB RAM, 2 or 3 TB hard drive space... actually, make that solid state storage... we need the speed. It'll also need 10 gigabit Ethernet. Oh, and don't forget my GeForce 4. What? Yeah, we need an GF4 to play Qua... to render those architectural drawings quickly.
....
I, did I say we needed ONE of these systems? Sorry... I meant THREE."

=)

Time it :)

[bpb213]

use an alias to bind gcc to a script that runs time on gcc and then sends the data to a analyst comp where its graphed and whatnot for blokes like us :)

WTF cares?

[SuiteSisterMary]

You don't want to know anything about your current set up. It's got nothing to do with what you want to be doing. What you want to do is figure out the work load that this new box is going to be doing, multiply those requirements by 1.5, and buy a box that, half loaded with equipment, can do that. It's called 'needs analysis.'

Test a smaller case?

[cperciva]

Given that software never works properly the first time, you're going to have to get the bugs worked out with a small group of developers before moving everyone over to this new system. Why not use that to your advantage? Measure the utilization of a box which has ten developers, multiply that by 500 (in order to give yourself a good safety margin), and that's what you need to buy before you move all 1000 people across.

You're trying to measure the wrong thing.

[hamjudo]

Only look at the heaviest users on the biggest, fastest machines. Once developers have easy access to faster systems, they will use them much more than the small slow systems. The only developers whose use usage won't go up much, are the ones who already have machines that fast.

See what resources the folks on your fastest systems use, and assume everyone will use that much.

Expect it to ramp up to that usage over the coarse of a few months. Expect the usage to keep going up forever.

Bigger issues with centralized development

[tshoppa]

I understand your desire to quantify the hardware needs for moving to a centralized system. But IMHO there are many bigger questions that have to be answered first:

• Policy - Exactly what is being centralized and what isn't? Are browsers going to be run on the central machine and displayed on X-terms? Or are they going to run on the satellite machines? You have to answer (or guess the answer) to this question for each and every application you may have. And the answers will not only determine the server CPU/storage architecture, it will also be vitally important to the network infrastructure.
• Development - What sort of development environment will there be? Will all software - from the littlest dinkiest shell script up to the giant mega-app be forced under CVS? Will you allow "checkout" to remote nodes, or only on the central node? etc.

All that said, as for remote CPU utilitization the ruptime [olemiss.edu] command is a start.

Distributed resources != centralized resources...

[skippy5066]

A problem you're likely to encounter (and may have already considered) is the fact that a the resources you require to do work in a distributed environment will not necessarily be representative of the resources you will need in a centralized environment.

A good multi-user system will be efficient in handling system resources, sharing CPU cycles, managing RAM and sharing when possible (using copy-on-write, etc), and all these things are unneccesary in single-user systems.

Probably the best bet is to come up with vague numbers for systems in different departments (developers need X amount of processing power, Y amount of RAM, while marketing needs X/100 CPU and Y/50 RAM, for example) and then find a company that's already done this kind of migration - perhaps through your solutions vendor - and see how their numbers for disctributed systems matched up with hardware for the centralized system.

Hope that helps,
-Jeff