Slashdot Log In
How Would You Make a Distributed Office System?
Posted by
ScuttleMonkey
on Mon Jan 21, 2008 05:35 PM
from the ip-over-avian-carriers dept.
from the ip-over-avian-carriers dept.
Necrotica writes "I work for a financial company which went through a server consolidation project approximately six years ago, thanks to a wonderful suggestion by our outsourcing partner. Although originally hailed as an excellent cost cutting measure, management has finally realized that martyring the network performance of 1000+ employees in 100 remote field offices wasn't such a great idea afterall. We're now looking at various solutions to help optimize WAN performance. Dedicated servers for each field office is out of the question, due to the price gouging of our outsourcing partner. Wide area file services (WAFS) look like a good solution, but they don't address other problems, such as authenticating over a WAN, print queues, etc. 'Branch office in a box' appliances look ideal, but they don't implement WAFS. So what have your companies done to move the data and network services closer to the users, while keeping costs down to a minimum?"
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
erm.. (Score:5, Funny)
If you solve that one let me know...it's been bothering me a while too...
So, here's your answer: (Score:5, Insightful)
Exactly what costs were you thinking of saving by consolidating? If it's just the cost of building and maintaining those physical servers, then here is the cold, hard truth: You are paying less for less service. Put servers at each branch office if you'd rather pay more for more service.
You get what you pay for.
Now, if it's other problems that are keeping you from setting up those dedicated boxes, realize that these are other problems. Identify them, and bring them back to Ask Slashdot. We're Slashdot, we're not psychic.
If it's your outsourcing partner gouging prices, dump them for an outsourcing partner which doesn't gouge prices, or do it in-house.
If it's the inability to manage all those servers, get them to talk to each other, etc, that's a more interesting technical problem that Slashdot might be able to help solve.
There are a few exceptions -- you might be able to get away with something like Coda or AFS, though I don't know how well that scales to crappy bandwidth. But if so, that would imply that your only problem is managing strictly filesystem data -- it doesn't help at all if the problem is access to, say, an intranet webapp. So again, we need details, if we are to find the clever exceptions.
Otherwise, upgrade your bandwidth, and/or outsource your actual application servers to someone who can scale. If it's just web/email/docs, Google can do that. Otherwise, find someone who specializes in what you're doing (our SVN is run by cvsdude.com), or bite the bullet and buy some virtual servers.
Parent
Re: (Score:2)
Re: (Score:2)
Re:So, here's your answer: (Score:5, Funny)
I'm an executive in IT with almost 20 years in. I have learned, without a doubt, that in IT what one pays is usually quite unrelated to what one gets.
Parent
Re: (Score:3, Interesting)
that worship Microsoft. If that is not the case, then
maybe you don't get what you pay for because you don't
have the budget to hire good people.
Re:So, here's your answer: (Score:5, Informative)
In my experience the only way to ensure value comes down to the processes involved in the planning, acquisition and implementation of any given project.
Ensure you have a process for identifying the requirements of any new service or equipment acquisition and do it without focusing on a specific system or product, if you limit yourself initially because you have formed a preconception of what you think you need, or you simply copy what others have done before, you will not get a solution that meets your needs.
Acquisitions of any type should always solve a business problem, whether you are addressing poor or suboptimal communications, the lack of external access, the rigidity of an existing system, scalability, security or stability issues or the lack of proper redundancy and disaster planning. You should not be buying things for the sake of it, or because someone simply thinks it might be a good idea, most of all don't buy things because other people have them. Justification is everything, otherwise you end up with things you don't need or want (but need to support) that don't provide business benefit, but do drain budgets which in turn makes it harder to address real issues. The identification of problems should come from within the business (that's what management is there for to a degree) or from independent consultants brought in for that purpose, it should never come from a vendor who (as it happens) also provides a solution. If a vendor makes a suggestion then assess the need and see if there is a business requirement, but do it independently.
Make sure you have a decent tendering process when you are sourcing equipment or services (for smaller businesses, that basically means you need to shop around, and tell your existing suppliers that you are doing so). Make sure that there is input not only from management and finance but also from end users and IT staff (sounds basic but not always the case...). You should also have a well thought out budget (after all you are solving a problem and problems should be quantifiable in cash terms), stick to it.
I don't even want to think about the number of times I have seen needless upgrades, additions and total changes to IT infrastructures for no good reason and more importantly with no real benefit. Resist it if you can (but don't resist change for the sake of resisting change, that is just s bad as doing the opposite.
As the parent suggests, price is not an indicator of performance. If your specifications and requirements are met, and you are within budget then great, if you are under budget then you are ahead of the game! With that in mind though, do thoroughly check out your suppliers (its inexpensive and easy enough to do), if a supplier is cheap and has a bad reputation then avoid them, make sure your suppliers can deliver before you sign contracts, sure you may be able to sue them (if you have all the information and the budget to do so) after the event, but it will be much cheaper to get it right first time.
Finally, I have found that the law of diminishing returns seems rather applicable to IT, as things get more and more expensive, the benefit from obtaining them becomes less and less. For example, a email system of some kind in a necessity in most businesses and generally speaking they are fairly inexpensive (relatively at least), whilst electronic whiteboards (my per hate) or upgrading cat5 to cat6 cable (without changing anything else, - something suggested to me by a vendor recently to improve network performance..) bring only marginal benefits but are relatively expensive.
Hmm, that was probably all totally offtopic - never mind.
Parent
Re:So, here's your answer: (Score:4, Informative)
Then 6 months later, we have a T1 outage to one of our larger offices, that office grinds to a halt. No BDC, file server and print server mans that as long as the T1 is offline that entire OFFICE IS OFFLINE. zero work is getting done, we spent 5X what we spent to consolidate to undo what he had us do.. It is the wrong thing to not have servers in every office. you have to plan for outages, and performance of having a server local can not be beat. (well you could have OC3's installed to each office, or have fiber ran to every office from your central location, 1000Mbit fiber point to point connections would do it...
Parent
Re:So, here's your answer: (Score:4, Informative)
Here's how we're moving ahead with centralization in a large distributed environment with about 50,000 users and 1,000 branches. We're reducing the server count by about 40%, and the cost by 70% versus a couple of years ago:
- Most sites with 10-75 people get a headless, stripped down box (~$2,000) that runs our desktop management software
- Medium/Large sites (75+) get a file server, which fulfills some other roles as well
- Large and VIP sites get a domain controller, mainly for availability purposes.
- A few "very large" (800+) sites get a 100MB WAN connection and use the data center services.
We looked at a few other solutions, with mixed results:
- WAFS/WAAS looked great, but the solution cost was almost the same as rolling out servers. Additionally, most of our applications are "thin" already, so we weren't really gaining much.
- Distributed AD servers are purely an availability play. (If your circuits/core servers are sized correctly)
- NAS also looked promising, but the cheap solutions weren't very manageable at our scale, and the manageable solutions weren't cheap.
- No backups are done on site, we're rolling out a distributed backup system that we de-dupe the data globally and backup to a data center. If you're using old backup software like TSM, Legato, etc, you MUST go shop around, the newer solutions are way way better and probably have lower administrative costs.
- Networks are getting faster and cheaper. We're seeing 3MB connections available to replace 512k frame relay connections at a slightly lower cost. We'll be switching as our network infrastructure gets upgraded.
- If your network supports it, multicast can make it much cheaper and easier to provision your workstations. Most management tools (Altiris, SMS, Tivoli, LANDesk, etc) support it.
Parent
All this is too complicated (Score:5, Funny)
Parent
Global file system (Score:4, Interesting)
Something like coda might be nicer but progress on global filesystems seems to have pretty much stalled.
It's a dead FS (Score:2, Informative)
Re: (Score:3, Interesting)
Re: (Score:3, Interesting)
Two words come to mind.. (Score:2, Insightful)
Another two words (Score:2, Insightful)
The only responsible answer to this question is to get someone in that has a track record of fixing problems like this. Don't expect to get a reasonable answer from a sketchy problem definition in a place like slashdot.
Re:Another two words (Score:5, Funny)
I'm sorry that my post is not more positive. But your post was so full of bullshit that I had to call you on it.
Parent
No Good Solution (Score:5, Interesting)
You can try the application accelerators that are out there now from Cisco. They basically use smoke and mirrors to keep traffic off the WAN and act as local proxies for different services.
Otherwise, your choices are limited. Citrix servers would be good for some apps, but get god-awful expensive fast. And an organization too cheap to build out a decent system to begin with isn't likely to make the investment in writing efficient apps.
If you're running on slow lines, bump them to at least fractional T3.
It sounds like the system was designed to serve 5 gallons of water through a swizzle stick. Ain't gonna work unless something is radically changed.
Or better....
Fire the outsourcing partner and the management that buys their bull, and build out a proper distributed archetecture.
Re:No Good Solution (Score:5, Insightful)
Parent
Re:No Good Solution (Score:5, Interesting)
Parent
Riverbed is a decent Solution (Score:5, Informative)
Parent
Re: (Score:3, Interesting)
Re: (Score:3, Insightful)
Yes fire the damn outsourcing partner. They obviously did not have your needs in mind when they suggested it. Most likely they thought they could save themselves money by having 1 location they have to go to when shit goes wrong.
Re:No Good Solution (Score:5, Interesting)
Parent
Re: (Score:3, Funny)
Your Senator...
Not enough information. (Score:5, Insightful)
The reality is other companies, such as yourself, exist and function probably better. If that indeed is the case, perhaps a friendly lunch with another IT staff member might help you.
I've consolidated offices and I've also pushed out servers to remote offices. It all depends on the need of the client. Examples
1. Client wanted 99.999% uptime and the only way I could get that was to have their servers in a data center. We moved them and uptime has been great.
2. Client wanted fast file access. We setup DFS with WIndows 2003 over a WAN link (T1) the client has never been happier.
So, to answer your question, it depends on your needs.
Re: (Score:3, Informative)
Hmm (Score:5, Insightful)
Find a new partner.
Re:Hmm (Score:4, Insightful)
In short, this guy better tell the management to get out their chequebooks, because the stupidity of trying to save a buck by cramming a Buick through a pinhole was a costly mistake with only one solution, inputting lots of money.
To my mind, unless the branch offices are really small, I think servers in each are in order.
I'm the network admin for a company with three offices; a main branch with about 25 workstations, a branch with 7 workstations and one with a couple. Because of the flakiness of connections, I can't rely on VPN. In the larger branch I have a Win2K AD domain controller running all the local apps, with some mirroring of the file store. Still the branch office can function even if the VPN goes down. For the smaller office, we have some Terminal Services licenses. It does mean if the VPN goes down, they're hosed. If it gets bigger, I'll put a server in. To keep costs down, I'll probably just put a Samba server in place.
Parent
WSUS severs (Score:2)
Re: (Score:3, Insightful)
Amazing (Score:5, Insightful)
IT costs money. I'm sorry that your outsourcer had some bad ideas. But your management must understand that IT services aren't free, and the health of your company depends on it's infrastructure.
Without knowing the specifics, the only low cost suggestion I can provide is converting desktop PC's into Linux servers, thus providing you with the distributed server network you need. Of course, the boxes will be underpowered and fall over all the time (yay desktop hardware), but if you really want to cut costs, there you have it. For backups, put in extra hard disk and backup to disk, it beats nothing at all.
Pixie dust (Score:5, Funny)
But whatever, you do, don't fire your incompetent outsourcing partner or actually invest in beefing up your IT resources. Both of those paths are DOOMED, DOOOOOOMED, I say!
What traffic, exactly? (Score:4, Insightful)
Print queues over WAN is taking the consolidation thing a little to the extreme, isn't it? Login authentications and print jobs really want to be local. Sorry about your predicament but you're going to get a lot of comments telling you to switch outsourcers or bite the bullet on their prices. What is the other traffic (as if that isn't bad enough): one assumes email, but are there big apps hosted on remote servers with lots of data traffic to db servers and the like? Simple document file sharing shouldn't be that much of a problem, or is it? You're going to get a lot of guesses without knowing the exact needs of your remote traffic. Good luck!
Re: (Score:2)
We've all got the excuse that we don't know what exactly this guy or his company needs. The question I'd be posing is why the partner didn't, because, regardless of what the next step is, I'd be giving them a swift, unceremonious kick out the door.
Having your Cake (Score:2, Insightful)
and eating it too? Is it just me, or is this one of those situations where upper management makes a design decision from something they glanced over in some IT mag, then decided to implement without consulting anyone with any IT background?
I don't see how you can create an insanely diffuse network, then turn around and expect it to perform like a network that has a centralized "HQ" with file services etc and a fat WAN connection.
Of course, you could just ask the execs to spring for ~100 WAN accelerato
There is no cake (Score:2)
Too little too late (Score:5, Interesting)
It's Easy! (Score:2)
Any application that won't run in a Firefox window is unneeded and merely distracts from the company's core mission. You won't believe how much of a performance boost you will get when you shut down those apps.
This problem could do with better definition. (Score:2)
We don't know how much data needs to be made available to each office - is it everything? Or is it just a different subset of the total in each office?
We don't know if you're talking about megabytes, gigabytes or terabytes of data. We also don't know how much that data changes on a daily basis.
We don't know if there are any existing factors to consider - be they political or technical (eg. "management almost c
WAN Accelerators (Score:4, Informative)
Not cheap - but easy.
Re: (Score:2)
Packeteer iShaper (Score:2)
Basically it is a WAFS box, with WAN traffic shaping, caching, etc, plus it acts as a Domain Controller, print server, authentication, dns/dhcp, etc.
If it works like they say it will it would be a good solution for you based on the problem description. Basically it is a server, plus WAFS, without being a server...
I wonder if anyone here has some hands on experience they could share?
Citirix? (Score:2)
Thin Client (Score:2)
What We Do (Score:2)
RCA of your situation... (Score:4, Insightful)
1. Propose a WAN-based solution.
2. When that slows to a crawl, propose a branch server solution.
3. When that proves to be too expensive to administer, propose a centralized solution.
4. When that proves to be difficult, unproductive, or slow, propose a branch office solution with accelerators, DFS, and all the goodies.
5. When that proves too expensive to administer, propose a thin client/remote app solution.
6. Repeat steps 2-5 as needed, substituting current technology for at least three iterations.
7. If you still have this client, you may now feel free to propose ANYTHING, including cans and string, or gerbils. They will buy it. Change your technical onsite staff every 6 months, rotating in fresh and untrained candidates. Rotate out those who show promise to be re-deployed at newer clients who are at step 4 or earlier in the process.
It's kinda sad. Consulting outfits can rarely make a living by doing right for a large client. Sooner or later, they either get replaced when the client starts 'analysing' the operation, or get replaced when some other outfit has a stronger line of bull to offer management.
Of course, there's incompetence, but my former boss isn't involved. He's busy screwing people in a different business, when he's not busy screwing his employees.
Published Apps or WAN Accelerators (Score:2)
If you're in a Windows environment, look at getting Citrix (or something similar) set up. Centralized files, centralized management, and it works very well. The one major issue is printing, although we use a product called Uniprint at work that is fucking fabulous. We went from 60% of helpdesk calls being "reset print spooler" down to 0% when we rolled out Uniprint. Very impressive stuff. We use Citrix at work primarily for our DB-intensive apps (so we
Samba and rsync (Score:2)
Well, how about just an old workstation at each remote site to run Linux on with Samba (assuming you're supporting M$ clients) and CUPS for file and printing services, while using rsync to synchronize the data with your centralized servers? You can even make additional automatic local backups to disk with things like faubackup or dirvish. It worked for me and you don't have to use such cheap hardware as long as I did.
But seriously, it so
Welcome to the cross roads... (Score:5, Informative)
Your company is trying to cheat their development model. Rather than setup a distributed IT application they have simply tried to distribute a small office network worldwide. If you look back to the tried and true OSI model. 7 layers. The 7 layer model doesn't speak of Network File Sharing, it speaks of Hardware and Application. TCP/IP (which we have taken quite for granted) is around/below the application level. If you have an application that runs at the TCP/IP level you are good to go.
I have setup distributed systems for several ISPs in the late 90's. We didn't think about what we were doing or why it worked. It looked like we could long haul anything we wanted. A little lag in sending mail or a few extra milliseconds to authenticate LDAP is no big thang. The Internet is distributed by nature. Sometimes DNS was a little slow but that was acceptable for 56k modems and DSL customers. But we spent 2 years working on a central web based administration/billing/customer support application with 1 SQL base in the center. We didn't distribute the application and have it write to the SQL base directly or move files around.
But you can't distribute the file layer. SANs in a local building have had some of the same problems. Any lag affects all applications and you solve it by throwing a big fat fiber backbone in the local building, but it break downs when you try to long haul over WAN links.
If your company is thinking it can sneak around coming up with a decent workflow model, and then implementing that in an application by simply given MS Office and Exchange (or whatever they have employed) to everybody they are sadly mistaken.
But worry not. You are not alone. Many business execs scratch their heads as to why the simply can't share out MS Project and their Excel Spreadsheets to 25 plus people teams and it will work fine. You still need to do the leg work of figuring out the work flow and reducing that to a transaction based system centrally located. That's it. All we've done in the last 20 years is replaced printouts with emails and spreadsheets, and the night operator (a job I used ta do) with scripts (or procedures) that dynamically update or run every 10-15 minutes. You still need a central system and then distribute parts of it, or have slim down interface that everyone can use remotely. Look at how a bank does it, just good ole dumb terminals.
No magic bullets yet. We need faster broadband and much lower latency before you can share out at the file layer using a network stack meant for transaction based appilications.
Let yourself off the hook. No mortal IT person can turn this tide....
You need local servers to reduce the latency. You need some decent thought on the application, not the OS and Office Suite. Good luck!