Building a Test Automation System from Scratch? 6
Borg#9 asks: "I am a Software Test Engineer for a small startup. I came to this company from Microsoft (hence my UserID). When I first started writing test cases, I discovered to my horror and dismay that the tool QA used for testcases was Word. WORD! I threw my hands up in the air and pleaded for a test case database. Finally, due to various reasons (the need to make money probably being one of them), QA has finally been directed to look into automation, test case databases, harnesses, which scripting language(s) to use, the whole nine yards! Since I'm the only person in QA who has experience with automation (and a successful QA structure, apparently), I've been assigned the task of researching this. We are quite literally starting from scratch. If there are Open Source programs that will work, I will push for those. If there is a 'silver bullet' commercial test suite, etc. that will work best for us, I'll push for that. Please give me experiences, recommendations, horror stories, etc. The software that we will be automating will be NT services (with a DOS debug mode). A large part of the validation will be the comparison of various text files."
What did you use at Microsoft? (Score:1)
Unless, of course, Microsoft used a custom-made in-house tool. But I doubt that could be the case; if they had such a tool, they would probably package it up and try to market it as "Microsoft Visual QA++ 2000" or something.
xUnit (Score:3)
test cases written so use python (Score:1)
write your test cases
python is nice because it has regexpression built in(nice for compareing text files)
can be compiled to java byte code (run on any java VM)
and is READABLE & CLEAR + STANDARD
its not going to fade away or a company take it in a bizare direction
use the money you spent on other tools to get the customiseation you want in your test env(such as web reporting & emails bugging engineers when they break something) i.e good consultants
take your time and SPEC it right worry about this and the inferstucture not actual tools
regards
john jones
Re:What did you use at Microsoft? (Score:1)
I wish I had an answer. (Score:1)
Each only does what it does best, unfortunately. :-)
I'm wanting to know the answer to this one, too.
Have you checked out SourceForge projects, Freshmeat, or HotScripts?
Also, remember that Perl's CPAN has modules for most scenarios and that expect (especially autoexpect and some hand-tweaking) can automate the command line....
Again, I want to see this very project come of age, too.
A good testing tool/suite should do specific things well as well as general ones. It should have good test data and should do sanity, regression, stress, and ad-hoc tests. It should be good at automation and reporting results. It should be able to automate almost anything and should allow for randomization of speed, load, timing, and data. It should also allow heavy customization, including you being able to insert your own source code or binaries to be executed. Today, it should be able to do heavy testing of Internet apps, especially webbed ones.
I know of no general-purpose OpenSource apps/projects like this, but would like to see one. If they're like me, when coders test, they run through it by hand or write a quick app that is specific to the program they're writing, which is not necessarily a good thing....
Some tools/projects, other than what's already listed:
http://freshmeat.net/search/?q=testing&filter=836
http://oss.sgi.com/projects/ltp/
http://www.protomagic.com/rjobd/
http://sourceforge.net/projects/va-ctcs
Somebody needs to get a project together, badly.
tinderbox (Score:1)
You might also find that a lot of projects are using lighterweight scripts to automate testing. Since the software available isn't mature, a quick custom solution written in a scripting language (shell or Perl are good choices) might be a good solution.