Building a Test Automation System from Scratch?

Borg#9 asks: "I am a Software Test Engineer for a small startup. I came to this company from Microsoft (hence my UserID). When I first started writing test cases, I discovered to my horror and dismay that the tool QA used for testcases was Word. WORD! I threw my hands up in the air and pleaded for a test case database. Finally, due to various reasons (the need to make money probably being one of them), QA has finally been directed to look into automation, test case databases, harnesses, which scripting language(s) to use, the whole nine yards! Since I'm the only person in QA who has experience with automation (and a successful QA structure, apparently), I've been assigned the task of researching this. We are quite literally starting from scratch. If there are Open Source programs that will work, I will push for those. If there is a 'silver bullet' commercial test suite, etc. that will work best for us, I'll push for that. Please give me experiences, recommendations, horror stories, etc. The software that we will be automating will be NT services (with a DOS debug mode). A large part of the validation will be the comparison of various text files."

What did you use at Microsoft?

[Anonymous Coward]

What did you use at Microsoft? You don't actually have to tell us, if it's a company secret or something. Just think back to the good old days and whatever tool you used when you worked for Bill, and buy that for your new company.

Unless, of course, Microsoft used a custom-made in-house tool. But I doubt that could be the case; if they had such a tool, they would probably package it up and try to market it as "Microsoft Visual QA++ 2000" or something.

xUnit

[Anonymous Coward]

A good piece to use in your overall solution is the xUnit testing framework from http://www.xprogramming.com/software.htm [xprogramming.com]

test cases written so use python

[johnjones]

personally I think what matters is the structure of the test cases and the attitude of the engineers writeing them

write your test cases /tool in python and pull the information from a DB (python can use free ones and oracle/sybase/DB2)

python is nice because it has regexpression built in(nice for compareing text files)

can be compiled to java byte code (run on any java VM)

and is READABLE & CLEAR + STANDARD

its not going to fade away or a company take it in a bizare direction

use the money you spent on other tools to get the customiseation you want in your test env(such as web reporting & emails bugging engineers when they break something) i.e good consultants

take your time and SPEC it right worry about this and the inferstucture not actual tools

regards

john jones

Re:What did you use at Microsoft?

[Borg#9]

Microsoft uses internal tools, for the most part. I used VisualTest for somethings and internal tools or VB/VBA. M$ has groups of people that do nothing other than write test tools. A co-worker and myself also wrote a test harness in VB that used Visual Test.

I wish I had an answer.

[mikehoskins]

Where I work, we use a mixture of hand-written Perl, manual testing, Segue's Silk Performer, Segue's Silk Test, Mercury, and Rational.

Each only does what it does best, unfortunately. :-)

I'm wanting to know the answer to this one, too.

Have you checked out SourceForge projects, Freshmeat, or HotScripts?

Also, remember that Perl's CPAN has modules for most scenarios and that expect (especially autoexpect and some hand-tweaking) can automate the command line....

Again, I want to see this very project come of age, too.

A good testing tool/suite should do specific things well as well as general ones. It should have good test data and should do sanity, regression, stress, and ad-hoc tests. It should be good at automation and reporting results. It should be able to automate almost anything and should allow for randomization of speed, load, timing, and data. It should also allow heavy customization, including you being able to insert your own source code or binaries to be executed. Today, it should be able to do heavy testing of Internet apps, especially webbed ones.

I know of no general-purpose OpenSource apps/projects like this, but would like to see one. If they're like me, when coders test, they run through it by hand or write a quick app that is specific to the program they're writing, which is not necessarily a good thing....

Some tools/projects, other than what's already listed:
http://freshmeat.net/search/?q=testing&filter=836
http://oss.sgi.com/projects/ltp/
http://www.protomagic.com/rjobd/
http://sourceforge.net/projects/va-ctcs

Somebody needs to get a project together, badly.

tinderbox

[Proud Geek]

The Mozilla project uses tinderbox [mozilla.org]. It comes in version 1 and 2 flavours, and is nowhere near mature (version 1 is more mature, I believe), but it is an open source framework that already exists. It is a tool for automating testing and reporting results. For running test cases you need something else, like DejaGNU [sourceforge.net].

You might also find that a lot of projects are using lighterweight scripts to automate testing. Since the software available isn't mature, a quick custom solution written in a scripting language (shell or Perl are good choices) might be a good solution.