What Do You Look For in a Big Iron Review? 262
ValourX writes "We're starting to write more reviews of enterprise-class hardware and software and although we've done pretty well with our reviews, the high-end products are a lot trickier when it comes to testing and evaluation. Obviously it is not possible to build an enterprise-grade 'your neck is on the line' production environment just for writing reviews, but maybe we can do something smaller, just for testing purposes. What do you as an IT professional want to read in a review for a server OS or a high-speed switch, or a big iron server or proprietary workstation? What tests should we run? What results and feature comparisons are going to be most meaningful to you?"
Vendor-Specifics (Score:5, Interesting)
Basically, none of these purchases happen in a vacuum. The merits of the technology matter, but "playing nice" is a dealbreaker. If this causes ANYTHING to break, forget it for now. et cetera.
True costs (Score:5, Interesting)
Just my $.02... oh, also just plain reviews of support companies on different hardware would be good also.
Scaling claims & Installation complexity (Score:5, Interesting)
Second, install the application yourself. Don't let the vendor do it for you. And when you install it, install it as an enterprise would. That is, if it's an n-tier application, or has multiple components, don't take the "default" installation and put all of the components on one system. Of course this will work. Try distributing the components over multiple systems like an enterprise would. Often this is where the complexity comes in and products falter.
One company I worked for purchased some software from Tivoli. After 6 months, and a team of engineers onsite from the vendor, they still couldn't get the components to talk for more than a day without problems (after weeks of installation), and still couldn't get useful data out of the database due to its size, so we took our $500mil back and bought something else. Having an evaluation that would've tested this would've saved us a bundle.
Environmental Factors (Score:2, Interesting)
From a Network Admin perspective... (Score:5, Interesting)
How easy is it to install? How easy is it upgrade? How easy is it, if its a different architecture (ie, Windows, Linux, Mac), to migrate big programs (Exchange, databases) from one to another? How well does it gel with existing servers? Do they recognize one another? Do they acknowledge? Can they fit into existing Active Directory-type listings effectively?
Most to all shops are not created overnight. They are built on mistakes or tried-and-true methods that are (usually) quickly outdated. The problems arise when you try to "fix" the existing problems by bringing in more robust OS's and capabilities. It is the meshing of these that is more important to Network Admins that tales of how well this server did on a single machine in a non-network environment.
** High-speed switch
Does it scale (how easy is it add one to five or more on a single chain?)? How is the admin interface? Is it web-based? Console (ie, serial port) based? Does it have both in case console is all that's available? Can you break it or overrun it with traffic?
** Big iron server or proprietary workstation?
Someone else has mentioned scale so let me throw in something different: How easy is it to recover? Does it have Raid? (Well, it should obviously) Break it, remove a disk and see if you can recover from it easily. "Lose" a driver and see how quickly you can recover.
Something I'd love to see is a review that includes a call to the tech support of that server. Don't tell them you're a reviewer, just tell them you got a problem. See how quick they respond, how informative they may be, how far does it have to go before they call in reinforcements? (ie, higher level support)? Will they call on-site repair? If so, how long did you have to troubleshoot before they determined it? Sometimes a card or piece will break and front line support will make you bleed through their ignorant manuals step-by-step when its clear that Piece A is broken and need a on-site tech with experience with that hardware to come and replace it.
** What tests should we run?
Stress, along with installing/upgrading hardware.
** What results and feature comparisons are going to be most meaningful to you?
I believe that over the course of this comment writing and thinking back over my dealings on big iron hardware, that comparisons in regards to tech support, informativeness, and responsiveness are something that can immediatley be added to the review process.
Something more long-term would be how long did the server run before downtime, problems, burnouts, or hardware failures.
Big Iron - Devaluing the Brand (Score:5, Interesting)
In particular, big IBM mainframes (s/3x0) running something like MVS (maybe VM at a push).
Anyone else think the term "Big Iron" is used innapropriately to describe a bunch of piddling little boxes that don't even need an air-conditioned datacenter equipped with an automatic Halon fire extinguishing system?
Realise what they are designed for. (Score:1, Interesting)
A Mainframe looks like a dinosaur when you grade it by PC standards, but when you actually see what they do and what they are designed to do you quickly realise that no PC or PC cluster could be made to do the same things at anything close to a reasonable cost.
For example take I/O operations for instance.
You have your standard PC PCI slots that run a 66mhz and are 32bit. That means that the a PC has about 120-130MB/s worth of bandwidth to move information from one device to another. Give or take.
Now when you look at a Mainframe enviroment you notice that it's very distributed by comparision, a modern top of the line Z Series has a theoretical 26TeraBytes worth of I/O operations at it's disposal.
Completely blows anything in the PC or workstation or server world away. There is no way you could create with a PC cluster a cost effective and reliable and backward compatable way of doing what a Mainframe can do and still be in the same price range.
So when testing computers test them for what they were designed to do and the enviroment they were designed to operate in and avoid making meaningless connections in between things like a cluster of PC servers aggragate SPEC CPU score vs a Mainframe's.
Or compare the $ per cpu power of a Itanium proccessor vs a Power970 (mac g5) proccessor. It's mostly pointless and meaningless except for curiosity sake.
Re:Not Speed (Score:3, Interesting)
Not so fast.If I am running 3 processes that don't need to communicate, the single CPU system will keep thrashing the cache while the 3 processor system won't.
That's why a big iron system may feel ponderous even if you're the only user online, but with 1000 users, it feels no slower while your disktop feels snappy and responsive but 5 people logging on with you can bring it to it's knees.
Re:Big Iron - Devaluing the Brand (Score:5, Interesting)
For extra points, the Lamp Test switch should be located at elbow level, so you can nudge it while walking through with the regional manager.
LEDs are fine, but they can't be blue. Anything with blue LEDs is probably still in diapers.
Seriously, failure isolation is a big thing. The best test would be to get a bunch of failed boards from the factory and install them in various combinations, to see if the system can puzzle it out. The manufacturer isn't likely to assist you with this test, however.
How does it handle spares? Are important parts protected 1+1 or n+1? How long can it operate with a fan unit removed for replacement? Do the air filter trays like to come unlatched and snag cabinet doors as they close?
Also, since my definition of "big iron" means "equipment which justifies employment of a Floor Space Planner", let's talk about cabinets and connections. Some of the better gear I've worked on uses fiber links between pieces, letting you locate them on different floors of a building if that suits you. And since all the links are redundant, you can move and replace link cables without taking a hit.
That same equipment, by the way, had a slight bug in the interface. If one sent too many commands over an administrative link in a short period of time, it would reboot. Oops. There's supposed to be a graceful rejection process when the buffer's full, and they must've forgotten to QA that part. (As far as I know, the bug is still in current versions of the software, because nobody runs up against it but me.)
Re:sorry to be blunt, but... (Score:3, Interesting)
I agree with the above post for all the reasons he mentioned. You don't drop $1M on a product because it got "5 thumbs up" in some magazine.
However, to offer some constructive criticism--what you could do is do extensive technical and performance analysis of a working system in a production environment. Instead of being able to sit at your desk and run pretty little tests, you would have to interview customers of a product, and ask them:
- why did you choose this product?
- did the hype live up to the delivery?
- What infrastructure does it interop with?
- what are real-world performance numbers?
- what are you the happiest about in terms of it's use?
- what do you find lacking?
Not nearly as easy to be sure--it'll take finding real customers of the product who have the time and/or inclination to let you do interviews and look at performance metrics. However, I for one would be interested to read case studies of other folks; although I would understand that not every thing would apply to my environment, I would be really interested to learn about the gotchas and real-world performance.The Ultimate Test (Score:3, Interesting)