Glitches in Massive Government Databases? 310
HBergeron asks: "Rather then post this as another YRO in the litany of new government datamarts there is a more fundamental question for all the coding Slashdot readers out there. This story, in Government Executive magazine, outlines the range of programming glitches in what is a relatively simple database. As a matter of public policy (and taxpayer money) is this level of non-functionality to be expected in these sorts of projects? Is the contractor just ripping off the taxpayers with bad code? How hard is it to write software like this that works?" The article focuses on the SEVIS database, but have others noticed similar trend in other government information systems?
All software has bugs (Score:5, Insightful)
It's like they say, you get what you pay for. Cheap prices are only cheap if your time has no value.
Re:All software has bugs (Score:3, Funny)
Wait! But what about Linux?
Time to end the sarcasm for the day..
What about Linux??? (Score:3, Interesting)
Time to end the sarcasm for the day..
Linux is not necessarily "cheap". It requires a substantial investment of time and effort in learning how to install, configure and use it well. My time is worth a lot of money. And I have very much gotten great value for my investment of time in learning Linux. In the end, Linux is certainly a bargain, but it ain't "cheap".
Re:What about Linux??? (Score:3, Interesting)
Re:All software has bugs (Score:3, Informative)
Re:All software has bugs (Score:5, Insightful)
Inexperience may be part of it, but often government systems are subject to a lot of competing interest and tying together existing diverse systems such that simple requirements in isolation often balloon into complicated situations. As a contractor your hands are often tied WRT cleaning up existing bad processes and odd requirements to solve needs of competiting agencies, departments, etc.
It is often diplomatic issues that cause the messes, not technical ones.
Re:All software has bugs (Score:4, Insightful)
Example: I have the database that I put together for my (small agency). It took a few months to do, but not many... perhaps three, and that's allowing for a bunch of mistakes. Then it was refined for a few years.
Three years ago we were told that a State Agency would be assuming the duty. They were legally obligated to begin about a year ago. They still aren't doing it. But I built it myself, but when I talk to the State people:
1) it's a team of people building it
2) the people don't stay at the same job
3) sometimes it's being done by people in a different department.
When I went to give them my data, they wouldn't even take an electronic file. They wanted typed forms. (I cheated and printed them out on a laser printer...but the spacing had to match that of a typewritten form.)
Now this was about 4,000 sheets of paper that they were going to need to enter by hand. My *guess* is that they were entering it into a 3270 based system (i.e., old IBM mainframe) and were then going to convert it into a PC based system. They were implementing it in FoxPro...probably. I don't really know. I never ended up talking to anyone who was doing the real work.
Personally, this was what I consider my "small test system". It's the one I traditionally converted from system to system during the process of learning the new system. I'm told that they'll be on-line with it in a month. (They'd better hurry, or I'll retire before they get it running... not that it matters.)
So I sure understand about the slowness, and I have a few theories as to what causes it: You need to take the cross-product of the Parkinson Laws with the "Mythical Man-Month". Then add a bunch of Dilbert for seasoning. Gourmet crusine to revolt the finest palate.
MS BS (Score:5, Insightful)
Any competent manager would know that experienced coders are usually FAR cheaper than inexperienced ones because they make fewer mistakes due to ignorance or indifference ("it works for me, so it's done!"). That gets you to the point of dealing with the more subtle and intrinsic bugs (e.g., due to conflicting requirements) quicker and cheaper, and the apparent cheaper cost of inexperienced developers is only achievable if you plan to release after coding is finished, not after testing is completed. Which is pretty much every MS *.0 release, now that I think about it -- got to get to market first, even if it's pure crap!
All software has bugs (Score:4, Insightful)
An excuse after the fact, the canard seems infantile.
All software has bugs. They are most certainly not created equal. Just because there is no way to expose them, doesn't mean they're not there.
("it works for me, so it's done!")
OUCH!
There is a big difference between partially working for a few things for a few people and never failing for everything for everybody. Some of the critical knowledge lies in the application area and is neither achieved readily or even necessarily consistent.
Re:All software has bugs (Score:2)
Oh, goodbye student loan payments, haha - Snake
Re:All software has bugs (Score:3, Funny)
If they go with the lowest bidder, why do they choose Microsoft over Redhat?
Re:All software has bugs (Score:5, Insightful)
Re:All software has bugs (Score:4, Interesting)
They are all right, except for the mathematician, of course (just kidding...well, mostly).
My own personal list of priorities would be (from high to non-existant):
1) Domain knowledge.
2) Solid data modeling ability.
3) Most everything else.
1,000,000) Buzzword compliance and/or magazine-cover conformance.
Unfortunately, most self-proclaimed prophets of the software world work in the exact opposite order, which is why nearly every project is a pain in the ass.
programmers =~ lawyers (Score:5, Interesting)
I think that, for most managers, the analogy which would most make sense to them is that Programmers are pretty much the same as Lawyers:
The goal, in either case, is to take a set of rules -- often arcane and ancient (programming languages and operating systems Vs. rules and laws), and combine them in such a way as to allow the client to achieve their wanted ends.
The judge would be the rough equivalent of a wetware execution unit.
Once they accept the lawyer analogy, then you can ask just how reasonable it would be to expect a lawyer to accomplish a lawsuit according to a tight schedule. Although it is doable, the tighter the schedule, the higher the price (often exponentially so).
I came up with this analogy because I ended up, a few years ago, self-representing myself in a reasonably complex lawsuit (It was about 4 years old by the time I got pulled in). With a couple of months heavy research I was able to do well enough in the courtroom (before the chief justice, and later at the Court of Appeals level) to reasonably impress just about every lawyer I dealt with in court.
I achieved this by pretty much applying my programming experience almost one-to-one. I simply treated the legalese and rules of court as a programming language. Old precedents were treated much like code snippits.
If you look at old slashdot postings, I think you can see that good programmers don't have that tough a time with laws and even court decisions. I submit that it's because the paradigms aren't really that different.
Re:All software has bugs (Score:5, Interesting)
How hard is it to write software like this that works?
Wow! Well said! My grandma couldn't have done better. In other words, please define 'works' for me. How many blue screens a day constitute 'works' and how many are too many?
Also, since we are at it, I want to reflect back on the latest project we have done. Incidentally, for the government. Before asking if a vendor is ripping the taxpayer off we need to consider how the government mismanages the resources it has. Consider the facts:
1. The project itself was fairly small and simple. I'd say it would normally take about 2 months to develop and deploy, but it needed to be done before the end of the fiscal year, so it was a 'now-or-never' situation, and was a horrible time-crunch. We had slightly more than half the time necessary to do it, but they won't even try to install it till probably the end of the year! The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.
2. They tried to keep tabs on the development by scheduling 'technical meetings' over the phone. While there is nothing wrong with that per se, in a time-crunch that was a horrible waste of time. The smartest things we've heard from them were questions like 'Are you using hungarian notation?' or 'is your code well-documented?'....
3. They insisted on
4. Their IT is kick-ass. As in 'their ass needs to be kicked real hard'. Installing a a windows server is a mountain of a task for them. Installing
Yes, it is true that some contractors will rip off the government (and it is really the government's responsibility to make sure that doesn't happen! But that's not the point). The point is that even if they have a perfectly good product developed by honest people, they are still remarkably talented at screwing it up. Bureaucracy and lots of idiots in charge of hiring people are to blame.
Brief Rebuttal (Score:5, Insightful)
>but it needed to be done before the end of the fiscal year
This is how it works: The USAF has a budget. Each area gets a small slice. If filters down to each office having about $10k ~ $30k for operations that year. That money has to last all year. About 20% of that is kept in reserve funds. If that money is not needed by August, we are free to spend it. At that point, we develop a wish list and try to get that aproved. By time all this happens, we have about 5 weeks to spend the reserve money.
No one in the military likes it. All our contractors hate it. If you want it changed, write your congressperson and have them change 50+ years of bad management practices...
>The quality of code would've been greately improved if we coded, say 40 hrs/week instead of pulling all-nighters.
I have spent countless days and nights working overtime. So have a lot of my coworkers. In times of exercise or, God forbid, a war, we go to 12+ hour days. 6 days on and 1 day off are common during exercises.
Contractors always make fun of us for sloppy wiring, half-assed installs, unpatched servers, etc... When new equipment arrive, we usually have a few hours to determine where it will go and when. We are usually told that the old equipment stays in place until the new stuff is operational. This leads to massive misuse of rack space. and cluttered wiring.
Also, just like your code suffers from 40+ hours, my wiring suffers when I have to spend my Saturday morning connecting a new router.
No one likes to work overtime. Your work suffers just like mine. You may lose a contract because of your bad code. People could lose lives because of my bad wiring. Let's both work harder to keep our shit straight, regardless of hours worked.
>They insisted on
This is becuase we have a very nice license with MS for their stuff. We get good support, including semi-annual "Best Practices" reviews by MS inspectors. The US Gov paid for MS tools, we should use them. If you don't like it, write your congressperson. Personally, I'd love to be able to use Squid on Red Hat. Unfortunately, we don't have the money to spend on more software licenses after we bought MS stuff.
>asked if it would work with a win2003 server as opposed to a win2k
Our upgrade paths are fixed by MS. This absolutely sucks. Our systems require specific patch releases from MS. Once they stop supporting those patch paths, we have to upgrade. Agian, if you don't like it, write your congresscritter.
>but they didn't even know how to install windows
I'm throwing a bullshit flag on this play. I find it difficult to belive that no one knew how to install Windows. In the USAF, we have a NCC department that does nothing but install, configure, and maintain Win2k servers.
There may have been an internal power play based on getting Win2k3 server training. That is an ongoing military issue. Your boss tells you to do something. If you do it and screw it up, they ask if you were trained to do that thing. If you were not trained, then you go to federal-pound-me-in-the-ass prison for working on something without proper training. If you were trained and you screw it up, then you get in trouble for not folowing the training guidelines for whatever it was you broke.
Everyone working in a military NCC can install Win2k Workstation and Server. Many of them are MSCEs or higher. They could probably install Win2k3. They just wanted official training on that product before they tried something and broke it.
>Installing a a windows server is a mountain of a task for them.
No it isn't.
>Installing
Re:Brief Rebuttal (Score:3, Interesting)
Installation documentation is for the lazy. They should've had a proper MSI package that w
Re:Brief Rebuttal (Score:3, Interesting)
Sorry if I offended you, but don't take it personally. Besides, we were not working with USAF, in fact, not with the military at all, so most of my statements probably do not apply to you by a long shot.
The thing I was really mad about was that they hire someone without any knowledge whatsoever to do their IT. When I was saying that installing windows was a mountain of a task, I meant it. Not kidding. For that par
Re:All software has bugs (Score:5, Interesting)
> And the government system of going with the lowest bidder is bound to cause some problems
I worked in a state agency, and the fact that we were required to take bids didn't really change who we purchased from. We just chose the vendor we liked best and justified it by writing the project needs around that vendor. They did that with employees too. When a new job opened, they often had someone in line for the position. However, equal opportunity required that the do interviews for the position. To justify the person they desired, they would write the job description and requirements based on the skills of the individual they liked. They would then schedule interviews even though they already had someone chosen for the position, just to meet requirements. I suppose they could have changed their mind if they found someone who was absolutely fabulous, but it's hard to convince an employer how great you are when in the back of their mind they don't think the interview is going to matter anyway.
Re:All software has bugs (Score:4, Insightful)
> bidder...
So you want them to give the contract to the most _highest_ bidder? The brother-in-law of the contract officer?
They give the contract to the lowest _qualified_ bidder. Doing otherwise would be stupid.
> It's like they say, you get what you pay for.
Bullshit.
Re:All software has bugs (Score:5, Insightful)
I've done some work with government organizations (not a lot so others way have had different experiences than mine) and from what I saw the biggest problem wasn't that the work was done by the lowest bidder, it was that the requirements were often created by people other than those that know the situation the best. Very little thought was given to designing something to be as usable and efficent as it can be, and more focus is given to making sure it gets finished by an arbitrary date. If it works, great, if not, it is ok as long as some department chief can say they are compliant with something by the required date. I've seen so many little problems that could have been fixed, but time involved in getting approval would have been more than then it was worth. I can't imagine the cumlative effect of all those little problems that get overlooked.
In the end you wind up with a big mess, tacking on or changing just enough to meet some kind of regulation. If you see something beyond the immediate scope of the project that would make things a lot easier and efficent, but it would require time/money or cooperation from someone else's department/division/agency it will be shot down as it won't be 'in the budget.'
Ok, that was my little rant. One time I had to sign a form to get the air conditioner turned on before the 'approved' time in a federal building, and I've been pissed off at bureaucracy ever since.
Re:All software has bugs (Score:2)
I work for a gummint contractor, and I am in 100% agreement with you. So are all of my co-workers, and a buttload of other
DingDingDingDingDingDingDingDingDingDing!
I think we have a winnah here.
Comment removed (Score:4, Interesting)
Re:All software has bugs (Score:5, Insightful)
And how do you deal with the customer whose specs say (in effect) "just throw a log across that creek because all we need is a footpath for the weekend" and subsequently declare your work an extension to the InterState Highway System and in non-compliance (substandard) of rule Blah, section Blah-blah, part Blah-blah-blah, paragraph Blah-blah-blah-blah?
From the article: The pilot was not designed to become a national system, however. The INS had intended to examine its results and then build something new, school officials who participated in the test say. It was a "throwaway project," says Johnson. "It wasn't supposed to become something bigger."
This is one of the most common causes of failure that I have seen over the years. A refusal by management to see the difference between a "proof of concept" project and a "production" project.
Attention programmers. Learn this now and learn it well. There is no such thing as a "quick and dirty" project. Anything you write for hire can and probably will be pushed into production. And if you "assumed" that you could "get by" with single user code with (for example) no record locking, error testing, logging, transactioning, or provision for remote monitoring or backup, you just screwed the pooch. The minute you check that code into CVS, it's heading for production with hundreds of incompetent users who will expect 100.000% uptime. And management will quickly point you out as the author of the failing new product and your reputation is shot, your future with the company is shot, and you have given programmers everywhere another black eye. Gee, thanks.
People, what it is, is that every piece of code that you write for hire has to be the very best you can create. Because, while your customer may have only asked you to throw a one-log footbridge across the creek, s/he is expecting an eight-lane interstate highway structure.
There are times when this is a good thing (Score:4, Insightful)
Re:There are times when this is a good thing (Score:2)
Surprising? (Score:5, Insightful)
Granted, this needs to change, but this isn't the first time the government has failed to provide adequate information regarding lists of people.
Re:Surprising? (Score:5, Informative)
Then don't live in Florida. It was a system designed to try to help Florida local governments enforce a Florida state law. This has absolutely nothing to do with the federal government, which is what the federal constitution mandates.
The way you genericly complain about "the government" shows one of the problems in these information systems: a total lack of knowledge of how "the government" works here in the US. We don't have one monolithic government (yet), we have a federal government, 56 state/commonwealth/district/territorial governments, and 3,066 county/city/parish/borough governments. Each of them have different rights and responsibilities spelled out by constitutions and charters. Each of them operates in a slightly different manner from all the others to reflect the wants and needs of its people.
I'm reminded of a story here on Slashdot a few years back about how France's government went "online," giving internet users the ability to do things like get a driver's license or enroll in public schools on a website. There were a great many people who complained about how the US hasn't done anything like this, all by people who didn't seem to realize that just about everything this new French site did are the responsibilities of state and local governments here in the US.
The only way things like this will get easier is if the US shifts away from the federal republic model and become a monolithic republic ala France. This would involve disemboweling the federal constitution and burning all state constitutions. Personally, I'd rather not see that happen.
Taxpayer $? (Score:4, Insightful)
They've been doing that for years. Toilet seats for $10,000; hammers for $7,000? Not only that how much money is wasted on the old "that's exactly what I asked for, but not what I wanted"?
Re:Taxpayer $? (Score:3, Insightful)
Re:Taxpayer $? (Score:2, Interesting)
Say I have a project that's going to cost a little less than $100,000. It's something I can't get funding for because the Great Gods of Paperwork have not dec
Re:Taxpayer $? (Score:4, Informative)
I spent time in Uncle Sam's Misguided Children, so yeah I knew there is/was fudging. Also [personal experience and no I was not in S4, logistics and supply] if a unit/department did not spend its entire allocated budget by the end of the fiscal year said budget was reduced. There were massive buying sprees of barracks supplies, spare parts, office furniture, etc. While not wasteful, unless it had a short shelf life, we wound up buying all kinds of crap that sat around for months before it was used. I'm not disagreeing with you; I'm just adding another expense theory.
Re:Taxpayer $? (Score:2, Insightful)
P
Re:Taxpayer $? (Score:2, Funny)
Re:Taxpayer $? (Score:3, Interesting)
1) A government employee(s) received a very large kickback from contractor (vendor) by accepting to pay the outrageously inflated prices.
2) The accounting department doesn't want
Re:Taxpayer $? (Score:5, Informative)
Government Waste (Score:5, Funny)
Had to. (Score:5, Funny)
Morpheus: What? What did you see?
Neo: I saw the same Bush pass by twice.
Morpheus: Was it exactly the same Bush?
Neo: I dunno... could've been some kind of father son thing.
Morpheus: A deja vu is a glitch in the database. It usually happens when they change something. Particularly, votes.
decimal to blame? (Score:2, Funny)
EDS? Explains a lot... (Score:4, Insightful)
EDS do a lot of systems that don't work, or don't work properly, or run massively over schedule and budget, here in the UK as well.
I just can't understand why governments insist on using them with the track record of cock-ups they have; they're not even cheap.
Re:EDS? Explains a lot... (Score:2)
Please replace the string "EDS" above with a variable that can take on the value of most any contractor in the world. Even then, your statement remains true.
I think it is also worth saying that it appears these pork-barrel database projects get the sludge at the bottom of the contractor's talent barrel. Consider that there are a lot of really talented and brilliant engineers doing things like designing aircraft, radar systems, etc. and wh
Re:EDS? Explains a lot... (Score:3, Informative)
Re:EDS? Explains a lot... (Score:3, Interesting)
If you want to believe everything the media reports and not read between the lines, go ahead. Just don't try and peddle it to the rest of us.
Re:EDS? Explains a lot... (Score:2, Insightful)
Okay, let's engage in a little thought exercise here. It's quite likely that EDS projects <insert epithet here> because they skimp on actual project implementation. Yet the money from the contracts must be going somewhere.
Hmm. Could it be they funnel some of the excess into lobbying and other less-tasteful forms of political influence?
In the blurred lines of the public-private sector,
((Product == Crap) && (Contract == Lucrative)) ? EngagePoliticalMachinery(Future = Lots more contra
Ever try bidding on a government contract? (Score:3, Interesting)
The reason the same gang of idiots keep getting work from the government is that they're the only people willing to bid on it. Government procurement is so bizarre that unless you have a team of specialists putting the bid proposal together you have no chance of getting it. Every large company I've worked at had a special "Federal Systems" division whose job it was to deal with that sillines.
25 Years of Government (Score:5, Insightful)
This isn't really news (Score:5, Informative)
Don't they have some donuts that need eating?
'Lowest Tender' syndrome. (Score:5, Insightful)
Government departments are pretty-much obliged to go with the lowest tender, even if the people running the tender know that the winning bidders are a bunch of incompetents who couldn't organize a fsck in a brothel.
So, the lowest bid wins, and then even if they actually are well-meaning and try to do the right thing, they have such limited resourses that they usually have to resort to working too few staff too many hours.
The result will not be quality code.
What is this a surprise? (Score:4, Informative)
Besides, why bother doing a good job if you know you'll get paid either way? That's what the tax collector is there for!
Anyone seen "Brazil"? (Score:5, Interesting)
I'm not a libertarian; the government indirectly pays a large portion of my salary. However, the extension of government power worries me, because the more control they have, the more opportunities to fuck our lives up.
Re:Anyone seen "Brazil"? (Score:4, Insightful)
The real danger of John Ashcroft is that he's a fundamentalist autocrat and an incompetent fool.
and by the way.... (Score:3)
Not suprising. (Score:5, Insightful)
Harder than you imagine. If you remove the pork-barrel politics, directives of what technology to use coming from the clouds, and the recently potty-trained project team members, there isn't much left to give the project a chance at success. Most of the project's time is probably spent learning the difference between JDBC and EJB or at meetings discussing the differences between JDBC and EJB. The remaining time is spent accomplishing little by discussing the well-presented but vacuous system requirements for the project. Whenever I see a job posting for a database project for a government agency, I pass it and look for projects worth doing. If it mentions
I don't like this conclusion, but I've worked on, interviewed for, or heard about enough of these projects to realize they are pretty much all the same and for what seems to be all the wrong reasons.
its about "now" (Score:5, Insightful)
The problem is not so much about "how hard is it to write software that works"
When governments sit down to write software, its usually done through private contractors. So, a group of beaurocrats have a pow-wow and come up with a spec that generally reflects the type of work that the agency is doing "now", without much future consideration.
15 years later
Complicating the issue, "upgrades" are usually in the form of applying a new "layer" to the system somehow. As of 2003 in this unspecified state, the typical motor vehicle registration passes through 4 different systems before arriving in the central (OLD and limited) database at the state.
Complicating the problems even further are the many new layers of regulatory bloat -- meaning, the BMV is using software that met their needs in 1968, but doesn't meet their needs now. For example, (and this is how data goes bad), they're required to track whether or not somebody's registration is under suspension. However, back in 1968 registration suspension wasn't even a blip on the radar. To handle the problem after the "registration suspension legislation" was enacted, an "exception" had to be built into the system... if the street address field contains a special message, it indicates that the registration is under suspension. Ultimate problem... fields in the database are being used for purposes they were never intended. The age of the system does not allow for it to be updated properly.
Re:its about "now" (Score:2)
At some point when a technology is mature, open standards should be developed and the open version should be used for long-lived government applications. This is really the only way that technology can ever last. It's a shame that indu
Re:its about "now" (Score:2)
It is hard to imagine a better example of an open specification than a government project. The documentation is open, and the specifications are designed by the customer to meet its requirements. More than that, the customer in this case has a LOT of clout -- governments aren't likely to be going anywhere anytime soon.
That's fine -- but the cost of reengineeri
Re:its about "now" (Score:3, Interesting)
If the project takes 5 years to complete (propose, fund, define, develop, test, implement), you end up with a system that's 5 years out of date. You'll have a bunch of desktop systems with Pentium 90 CPU's, running Windows 95 at best.
And getting any improvements implemented literally takes an act of congress (or "creative accounting" - which several of our larger contractors have been bitchslapped for in recent years).
It's okay in the Military
Funny how you never hear ... (Score:5, Insightful)
Somehow "they" have had UFO technology which would make petroleum obsolete since the '50s, conspired to kill JFK to keep it a secret, brainwashed Chapman to murder Lennon, created a secret government database tracking everyone's cash transactions, control us by putting chemicals in our water and thought patterns in satellite broadcasts. Oh yeah and "they" also were behind the 9/11 attacks as well.
Yet "they" can't even figure out how to keep track of whether or not foreign students went to class or not.
Oh yeah? (Score:3, Funny)
That's exactly what the CIA wants us to believe! Saaaaaay...aren't you the same Professor D who was involved with the faked moon landings!
Canada's National Gun Registry Anyone???? (Score:3, Informative)
After living through that I no longer have any confidence at all in the government to be able to implement any IT project competently. One billion (with a B) dollars to develop a database that essentially matches a gun serial number to a name and address. And they're still not done. HOW? Somebody please explain that one to me...
Re:Canada's National Gun Registry Anyone???? (Score:2)
I sat down and tried to figure out how to spend a few hundred million dollars to create such an application and a department to handle it. Obviously I cannot be a government employee, because I couldn't figure out how to spend anything more than a fraction of the money...
Re:Canada's National Gun Registry Anyone???? (Score:5, Funny)
One billion (with a B) dollars
Canadain?
That would be like what, like US$ 150,000?
Re:Canada's National Gun Registry Anyone???? (Score:2, Funny)
Actually, it's more like 733,360,738.55 USD.
You Yanks pay that for Pentagon toilet seats, right?
Re:Canada's National Gun Registry Anyone???? (Score:2, Funny)
Not necessarily bugs (Score:3)
That said, the 'bug' mentioned (that of getting a non-related set of students) smacks of shit data entry rather than a programming failure.
Lord help you, America, if you acquiesce to this sort of intrusive crap.
Before you know it, you'll be as bad as Europe, and soon after, as bad as Britain.
You'll be being persecuted on the basis of bad data, bad laws and stupid politicians.
That's our job.
Stop it.
Sounds like poor relational design (Score:2, Insightful)
Foreign students KNOW (Score:2, Funny)
EDS business model (Score:4, Interesting)
This is good (Score:3, Interesting)
Of course it's a taxpayer ripoff. Thank goodness we're not getting all the government we're paying for.
And another thing... (Score:3, Insightful)
Re:And another thing... (Score:4, Interesting)
No suprise... (Score:4, Informative)
After September 11, it was decided that the system had to be used starting early 2003. This was years earlier than in the original plans.
Re:No suprise... (Score:2)
My guess... (Score:2, Insightful)
System doesnt work (Score:5, Insightful)
There is one database used for payroll on which millions (if not tens of millions) were spent, and the end result is a system nobody is completely sure of, and which requires all the deparments involved to completely change all their procedures. And its even less flexible and problematic than what it is replacing. AND this is a custom application!
Until government starts paying tech people what they are worth in the private sector, you will always have poorly skilled bullshitters pedalling their wares to the public sector, who is suffering from the illusion that throwing money at a problem will make it go away.
Q:Elepehant? A:Mouse built to government spec. (Score:5, Interesting)
A user who has never written a *COMPLETE* system specification, acutal has no idea what that is, who only knows what he/she does not want.
Software developers/coders/bodies who are not SME's (subject matter experts), making system / software decisions without either the knowledge or guidance to understand the ramifications of those decisions.
Neither users, nor software development companies want to deal with these issues, they would rather just get the money.
That is why most large software development/ service companies have such bad reputations.
According to SEI, (Software Engineering Insitute) over 70 per cent of all software development projects are terminated as failures.
Paper (Score:2)
This is terrorism! (Score:3, Funny)
Much easier to wreck havoc on a government project and cause disruption through buggy software than to take the time to learn to fly and then hijacka nd hit public buildings. The most positive thing is ofcourse that you, the terrorist, is not killed
If the US really wants to get rid of all potential terrorists, they have to evict everyone of a certain religious and cultural group and then close the borders and let no-one in and out. This is ofcourse not possible, hence all these measures taken by the gov is virtually useless.
Governments and Corporations - Clients or Children (Score:4, Interesting)
My experience with the problems of these sorts of situations is as follows:
1. Sales droids underbid each other to get the job and commit to ludicrous time frames
2. Project teams end up with short development time and are always pushing to reach the deadlines in time.
3. Client changes their requirements, but will not change their expected delivery date. Either they refuse due to business need, or they do view their change as an actual change. More often they view their change as a "clarification" - even if it contradicts what their specifications orignially said.
4. Agressive job market has Project Managers kowtowing to Client demands.
5. Multiple departments are clients, but pay different amounts into pool. Each department seeks to maximise their benefit at the cost of other departments (despite fact part of same organisation - politics)
I mean, really, the problem exists in the fact business units will often not sit down and commit to producing clear, unambigious details of what they want & need. Bugs creep into the process when your dev's are working frantically to meet the deadlines and handle the unexpected change request.
And now a pithy little quote to put on your wall:
-----
Programming to Requirements is like walking on water.
It's easy to do when everything is frozen.
Legacy systems (Score:2, Insightful)
Many campuses (at least state campuses) are running really old student records systems. Imagine that the web front end for the users is built from screen-scrapes of old mainframe applications. To make it worse, now try
Wonder what the SEVIS site is running? (Score:3, Interesting)
Sometimes, SEVIS crashed under the stress and expunged the day's work. The delays and headaches led some schools to close their student offices and ask employees to work nights and weekends, when traffic was lighter.
That's as good as putting a logo on it. So I went to Netcraft and checked the SAVIS web site, egov.immigration.gov. And sure enough: The site egov.immigration.gov is running Microsoft-IIS/5.0 on Windows 2000
I bet that the database is also running on Win2K...
Oh golly, you mean that when you put a high-volume site under Windows, you get (gasp) crashes and data losses? No way! Who would have thought that? Obviously, the poor Dept of Justice was a victim of an unheard-of, unexpected problem.
Not.
Another budget blow-out... (Score:2)
from the government managers hiring consultants to develop their requirements.
Quite often those consultants can charge almost as much as the entire development team charge
for the project.
Bureaurcy and the lowest bidder (Score:3, Insightful)
Yes.
Look on the bright side, if they can screw up simple projects like this how far do you really think TIA (Total Infomation Awareness) is gonna get?
TIA Alias Search: Commander Taco
Output: Mexican terrorist. Leader of collbration site for social dissidents.
Just far enough from the truth to get somebody sued....
Old news. (Score:3, Informative)
Blame DeVry. (Score:2)
It's the environment (Score:4, Interesting)
I think there's a few issues involved:
- a lot of the code is written by people with skill levels that wouldn't be accepted in the outside world. I've got no doubt that there's highly skilful coders working for government, but there's a lot of duds as well and often your code is only as good as the weakest coder in the team. Furthermore, a lot of the weakest coders are the designated "experts" for legacy infrastructure, such as databases, and you find yourself having to rely on their input far more than you'd like
- a lot of government stuff has been outsourced, and generally the outsource partners seem to be less diligent for government contracts
- governments are forced to send everything out for tender, and may need to give work to the lowest bidder or face internal inquiries. In the private sector, you get to eliminate a lot of tender responses, either because you know the respondent is incompetent from past experience or the price is so low that the respondent couldn't possibly deliver a quality service; in government, you don't have the same luxury of filtering out the crap
- in the private sector, a lot of mainframers moved across to Unix, then Windows, in response to changing demand. While these people may not be expert in all 3 areas, they at least know enough to be able to hold a technical discussion with someone else from another area. In government, a lot of mainframers have been doing the same job forever, and you need to tie everything back to their narrow view of the world to get anything out of them whatsoever. You'd better not try to talk "Web Services" with these guys... There's a level of financial support and job security for these types of people in government that allows them to keep doing this, that simply doesn't exist in private enterprise
Now that I've pissed off everyone working in govt IT, I have to reiterate that some of them are extremely good. The issue is that the system doesn't work on a "survival of the fittest" system such as applies in private enterprise.
The key is in the rate structure (Score:5, Interesting)
Unfortunately, It's a rule of thumb in this industry that a few good programmers are a lot more productive than many unskilled ones. The result is that many government IT projects are shoddily built by well-meaning but inexperienced developers who are put in that position by a contracting structure that fails to recognize the realities of the IT industry. Contractors are just responding rationally to the incentives that are presented to them.
These numbers are examples -- in fact the situation may be even worse. Federal government contracts vary in their rate structures, and many are stingier than this. It may well be impossible to bring on a senior developer as a subcontractor because the maximum hourly rate that the government will pay on a project is lower than the cost of the senior developer.
A prime contractor that I worked with staffed a large WebObjects project for the Department of Defense with a dozen or so low-paid, fresh out of community college drones. Every six months -- when a project review was due -- they would bring us on board as subcontractors for six to eight weeks. In that time, two or three of us would take the code base from where it was four months ago and bring it close enough to the required progress to get the contract renewed, and then the prime contractor would say "goodbye" and toss us out. Four months or so would pass by, with their people making little meaningful progress, and we would get a panicked call for six or eight weeks of more work to get by the next project review. (Did I mention that the prime contractor didn't pay the bills for one set of work until they needed us for the next project review? It got so bad that at one point we had to treat them as though their credit rating was zero, and demanded that payment for each week's worth of work be deposited in an escrow account before we would continue.)
By the way, this rate structure is not unique to government IT projects. Other types of government projects display the same professional services rate structure. When I worked for a (then) Big Six accounting firm as an economist, most consulting projects for corporate clients were staffed with a ratio of one partner and two or three senior managers to six or eight associates. However, the Federal government group was staffed with a ratio of one partner and one senior manager to twenty or so associates. I talked to the senior manager, and he told me that (a) the associates in the government contracting group were paid much less than we were on the corporate side since they billed out at a lower rate, and (b) the only way they could make money was to use lots of cheap associates because senior people could only break even at best at government rates.
Ya know, it'd be nice to see a GSA person squirm over this sort of thing in front of Congress some time. Then again, Congress may be part of the problem, as they'd rather generate lots of jobs for constituents, instead of a few.
--Paul
How hard is it to write software like this? (Score:5, Informative)
I see several possible problems here. First, it is possible that this software was rushed into use before it was ready. Given the political pressures involved, I suspect that is part of the problem.
Second, I doubt that all of the programmers involved are of guru caliber. I don't intend to malign them. Even assuming that you have nothing but above-average programmers, when you build a huge project with lots of designers and coders, there are going to be miscommunications and some details that just aren't communicated.
Third, I would bet that this project has so much design documentation done up front that it is impossible for anyone to wrap their brains around the whole thing. This is, at best, a 1.0 release. And there are going to be design flaws in it. And the guys writing the code aren't likely to have a broad enough overview of it to spot them all. They also undoubtedly tripped over a lot of things that weren't specified up front and should have been. It is the nature of the game. But they weren't free to just choose a good solution when the questions came up.
The projects I cited at the beginning were developed by small teams with a vision of what they wanted to build. Within the constraints of the tools they had to work with and the general idea of what they were building, they were free to change the rules. They could refactor to their hearts' content. That is not going to be the case on huge government contracts.
Everything that we know about open source, agile/extreme programming, etc. doesn't apply to this kind of project.
It's 'Unit Tests', stupid! (Score:3, Insightful)
The key to quality software; flexible, extensible, fault-tolerant, maintainable, and all of those other adjectives that 'good' software is supposed to have is very, very simple.
It's called Unit Testing. It's not brain surgery. I have worked on several medium to large scale projects (500k-3.5m lines) in several languages and environments, and I've yet to bomb one hard using this methodology, despite the usual client shenanigans.
Every time I write a functional subunit, I start by writing a series of tests (based on the spec, hopefully) that define 'doneness' for that subunit. Every object in the system has it's own set of tests. The test harnesses are chained together, so I can hit a button, so to speak, and run all of the hundreds to thousands of tests at once.
Whenever I check in new code changes, I run the test suite. If a test fails that previously worked, then I broke something. This plus good OOP practices (low coupling, high cohesion) allows you to make changes on the fly without the kind of 'The Money Pit' syndrome (fix one thing, another breaks) that is described in the article.
I am certain that the system in question was NOT developed with these methods. Most development organizations that I come into contact with pay lip service to the concept, but don't want to spent the perceived extra $ up front. The thought of all those developers writing TESTS when they could be writing CODE scares the willies out of them. But it pays for itself. It really does, every single time. Don't tell your boss, and try it on your next project. This is old news - google has a ton of info on it, and there are some good but unnecessary books also.
In the dot.com glory days, we had a huge system, running several hundred transasctions per second on a geographically distributed system of clients. We made fundamental architectural changes without a hitch, switched servers live without a hitch etc, and made a zillion little changes, all live, and all without a hitch (well, other than really stupid human errors, like locking out the client upgrade system with a bad password... oops). We had zero budget, and 2.5 developers. Unit tests are the Way, and if any company that doesn't mention them in your first meeting, run like hell.
Re:It's 'Unit Tests', stupid! (Score:3, Interesting)
Reactive, Just-In-Time Development (Score:3, Insightful)
My experience tells me that the problems begin when we fall into the trap of trying to solve problems with a reactive mindset instead of a proactive mindset (proactive being favorable). We allow daunting problems and/or a need for revenue to back us into a corner time and time again and every time we are forced to hack our way out. Some of that is just old-fashioned survival, but a lot of it can be avoided with deliberate forethought, planning, discipline, and a commitment to quality and detail.
Avoiding clusterf*x has to be an institutional effort, whether the institution is a huge goverment agency or an tee-tiny, independent software shop. Everyone in the organization - operations, sales, IT - has to be on board with the policy that "if it's worth doing at all, then it's worth taking the time to do it right...the first time." I said "fundamental" earlier b/c that has to be something like lesson #5 in life's little handbook - we all heard it all too often when we were kids, we know it's true, and yet now we don't pay it any mind.
I do think the failure to heed that simple maxim usually starts in business development and snowballs by the time it gets to IT, but it really goes both ways. Everyone has to be responsible for maintaining the discipline required to produce quality.
What happened with this system is everyone involved got themselves all in a panic like a drowners who not only won't let you save them, but pull you under too. It's understandable given Sept 11, but "undertandable" and "right" are two different things. Legislators threw money at a situation they didn't try to understand. Deal-makers when after that money, promising to solve problems they knew they didn't understand. Developers enabled deal-makers by claiming to understand. No one took the time to do it right the first time.
P.S. It doesn't have to be this way.
It's not just one problem to solve (Score:3, Interesting)
1. Requirements change over time, but inadequate funding available to really redo the design
2. The software was developed and designed for a 4 year life cycle and unfortunately we are in our 5th/10th/20th year of maintenance
3. Old software is compared to brand new commercial apps and their features, which didn't exist when the software was originally written. (Think of comparing a telnet, ASCII based application to a GUI app. And yes, ASCII based systems do still exist and are used for real work)
4. Use of cheap, inexerienced people in order to make a higher profit margin
5. Unable to charge a per usage/license/maintenance fee, only able to charge for the development costs up front, so potential profit avenues are reduced.
6. No integration planning on the government's side due to lack of expertise, politics or not realizing that different projects should be coordianted among contractors.
7. Eye Candy tools sold to higher ups without adequate technical input into the chosen packages.
8. "We've never done it that way before" uttered by company management, customers, developers, etc, which leads to long involved discussions and training basically as to why new Method A might be better than old Method B.
Now, the really, really hard part, what to do about it? I dunno, if I knew that, I'd be getting paid a lot more money then I am now and I have my underlings read
Myself, I am working on educating people in our company about other ways and means to make money from software and trying to insidiously
You guys have only a vauge idea (Score:3, Interesting)
I just live here....
FUD conspiracy theory (Score:3, Interesting)
The result is that every once in a while a random student is arrested, delayed and/or deported because of a glitch in the database. As was mentioned in the article, students weren't too worried before SEVIS came into service, but now, the government is acting like a rampaging despot -- but they get to blame it on an opaque, mallfunctioning computer system. What more could they ask for?
Now, if the government wants to deport a student for (say) calling Bush a moron, all they have to do is induce a glitch in SEVIS and then have the student stopped for (say) speeding. One quick look in sevis, and they're in jail or on their way out of the country.
This whole randomness aspect is what has the foreign student body nervous, and it's almost impossible to pin down. It seems like a pretty good opposition supression system.
. . . But as a corrolary to Occam's razor says: "Never blame on belligerence what can be explained by simple stupidity." -- and we're talking about the government, after all.
DOI Websites Yanked *Again* for Security Flaws (Score:4, Interesting)
Remember in late 2001 when the US Department of Interior was ordered by the court [foxnews.com] to take more than 100 of their web servers offline due to abysmal security? Hired white hats were easily able to gain access to the US Indian Trust database and found no security measures or even audit trails in place. Worried that this could be contributing to the agency's continuing mismanagement and loss of allegedly billions [monitor.net] of dollars belonging to Native Americans, Judge Royce C. Lamberth ordered the DOI to "immediately shut down Internet access from any computer, server and system in the department that has access to individual Indian trust data."
The defense counsel noted that the fact that they took down over 100 mostly unrelated servers "...just shows you how inept they are. They don't even understand how these systems relate to each other so they just pull the plug on the entire system."
And now last month they were ordered to disconnect their servers again [sfgate.com] after refusing to let a court-appointed special master test the security measures they've supposedly put into place since then.
Sounds like an endemic problem for government agencies, at least at the federal level.
XML-enabled Kafka (Score:3, Interesting)
SEVIS, like most web-oriented applications, consists of a browser form based interface, application logic, and a database. There's also an XML-based batch update capability. It's all reasonably modern and buzzword compliant.
Looking at the manual [bcis.gov], the main problem you would expect to see is rejected transactions. Basically, the system won't accept transactions which indicate something that violates immigration policies. That makes it inherently obnoxious.
But on top of this, apparently the database back end has problems too, since data is reportedly "bleeding" from one school to another. That's not something that an end user (a Designated School Official) can cause; school officials don't have broad enough access rights.
Any major error in the data for an individual tends to prevent transactions which could correct the error. Designated School Officials can only enter transactions which are consistent with the existing data. Anything else requires approval from what used to be the INS, and is now part of Homeland Security. So that creates a huge load on an organization that's always been behind on paperwork.
All entry points to the US can access some of this data, and it's used to decide who gets in, who gets turned away, and who gets sent to a detention center.
The whole system is very Russian - great power over individuals, exercised ineptly.
"You want a vision of the future, Winston? Imagine a boot stepping on a face for eternity" - Orwell.
Too much job security? (Score:3, Insightful)
Do job really badly -> lose job
Here in the UK, government IT projects go vastly over budget and fall apart time after time, yet the same companies get hired again and again to do the contracting.
Re:Uh oh... (Score:4, Informative)
-Xerox'd copies of any forms you've filled out related to all this. Carry these with you on the plane in carry-on luggage.
-Ask your advisor for a hard-copy listing of all the data that will be entered into the SEVIS system related to you.
-Consider having a letter written and signed by the advisor that entered your data into SEVIS, indicating what he/she has done. Also consider getting that advisor's office and home phone numbers in the event something goes wrong outside the 9-5 timeframe. Again, carry-on it.
-Call your advisor from Australia and ask that he/she check to ensure that your data is in the system before you leave.
There are probably a bunch of other things you can do, but the point I'm getting at is that you should try to cover all your bases and double check everything. Yeah, it's unfortunate you have to do this s---, but it might be the only way to prevent yourself from too much hassle.
Re:Nobody read this (Score:3, Informative)
Re:Nobody read this (Score:3, Informative)
So, apparently, SEVIS is inflated from the original (1996) prototype.
Actually, I don't see your "mid-late 80's" quote in the article. I do see "late 1990's" mentioned just under the Born Too Soon header, going on to the previous poster's quote and the above reference to EDS' pilot project.