Automatic Functional Testing for Mac and Linux? 202
"For example, a vendor has a database interface, and wants to know how many times we can access a series of records, what the results show (are the right, blank, time out?), and how long it took for the search to take place during various times of the day, and so on. So we emulate one of their data entry people by logging onto the system, putting in some search parameter from a script, and then record what the text (or error message) was when it came back. Yes, we could connect directly to the SQL database without their front end, but it's the combination of their front end and the database we usually test. We have found many examples where were can connect raw, for instance, but the front end displays a timeout. We also test things like 'What happens if the user enters a colon or asterisk?' or 'What if the text entered in the box is over 256k?' Then we tell the vendor the results, and they hopefully fix them based on the data we gave them."
AppleScript (Score:3, Informative)
testing tools (Score:5, Informative)
www.opensta.org
There is a commercial company that provides support at www.opensta.com
Hmm. (Score:5, Informative)
Don't be scared by the name "script"...there are visual tools for ascript and the language itself is so simple my mom could do it.
XRunner (Score:5, Informative)
Re:In case it gets slashdotted (Score:5, Informative)
We're working on one... (Score:3, Informative)
The name of the project will be HASTE, look for it in a month or two.
XTest and Expect ? (Score:4, Informative)
Expect [nist.gov] allows us to do on Linux
(and a vareity of other operating systems) ?
Check out Test Quest Pro (Score:3, Informative)
It's fully scriptable using a c-ish syntax, and comes with a nice IDE and runtime environment to get the work done. It seems like it may be a nice match for what you want to do.
Here [testquest.com] is more info.
Re:AppleScript (Score:1, Informative)
Re:AppleScript (Score:2, Informative)
Re:We're working on one... (Score:2, Informative)
Re:AppleScript (Score:2, Informative)
F-Script (Score:3, Informative)
For Mac OS X have a look at F-Script. [fscript.org] You can inspect a running program and call any method in that program and any method on any object in the Objective-C runtime. This should allow you to generate events or enter text in text fields, etc.
Of course this will only work for programs developed using Cocoa. Can't help you with Carbon apps. You might be able to write something using the Apple Event manager.
Last time I had to do this for Linux (and Gtk+) I had to roll my own.
Re:XRunner (Score:1, Informative)
Re:XTest and Expect ? (Score:2, Informative)
Applescript, QuicKeys for Mac OS X and OSA (Score:5, Informative)
However, AppleScript is not the only scripting environment for the Mac. Underlying all of the native scripting languages on the Mac is something called the Open Scripting Architecture [apple.com]. This allows any OSA component (i.e. scripting language) to talk to any other OSA component.
One OSA component is QuicKeys [quickeys.com], a great product that actually does simulate mouse clicks, keystrokes, etc., and is completely scriptable. It integrates completely with AppleScript and via the power of the OSA, with shell scripts and Perl as well.
On top of this, Apple provides a powerful GUI building tool for AppleScript called AppleScript Studio [apple.com], which is free. Plus, AppleScripts can talk across a network to do RPC. Imagine, you can have an AppleScript Studio front end that drives scripts on a set of machines over the network (so that you can assess performance under load). The individual scripts rely on AppleScript to drive the logic while using QuicKeys sequences to drive the GUI.
The only issue here is how easily you can assess results. There's no general way to check what the GUI is showing other than a pair of Mark I mod 0 eyeballs, since grabbing data out of the app being tested (even simulating a copy-paste) only talks to the underlying data structures and does not address GUI bugs. A possible way is to arrange the windows in a known manner, turn off the menu bar clock, take a screen shot, and use a graphical comparison tool to XOR the bitmap with a known good screen shot to highlight differences, but this won't work if you're working with variable or randomly generated data.
Disclaimer: I work for Apple, but these are my opinions only and do not represent any sort of official endorsement.
--Paul
Re:QARun, QADirector? (Score:3, Informative)
QADirector is not a testing language in itself. It is a harness from which to run the script generated by script writers. It used to be a Unix app, but when the switch was made to Wintel there were a few missteps. If you are looking for a test harness that is comprehensive (for Windows anyways) in its command-line options, look into QAD, otherwise I would go with Test Director by Mercury or maybe Segue's new product (I believe its called SilkPlan or something similar).
QARun on the other hand is a pain in the ass. If you are looking for automation, don't use it. It's designed for straight-up "record and playback" style testing. If you wish to deploy these tests to client machines (something which some, but not all people want to do), do not use QARun. You can look into XRunner (it is the predecessor of WinRunner) by Mercury if it is still being sold/supported for *nix testing. I honestly don't know about Mac, our product doesn't run on it.
If you want a serious evaluation document, I highly suggest heading over to http://www.qaforums.com [qaforums.com] and head to the Automation section. Post a query about the Mac and you'll probably receive a response from one of the moderators with a white paper that was written comparing all of the available tools.
DISCLAIMER: My group uses QARun, QAD, Segue Silktest, and WinRunner, so as not to be accused of bias.
Re:AppleScript (Score:3, Informative)
Someone mentioned server load testing earlier... if that's what you're looking for, remember that Mac OS X is basically a pretty window manager on top of a flavor of FreeBSD -- you can have cron automate shell scripts and perl.
Re:XTest and Expect ? (Score:4, Informative)
(FWIW, my job very recently involved buying or writing an automated test scaffold for graphical apps on Linux that would work across architectures and windowing systems -- it turned out that the latter was necessary, as the available commercial products just weren't/aren't cross-platform capable, and android has some serious faults which I'll expound on if asked. The product partially exists, having been put on hold for other internal development... perhaps when it's finished we'll release it under an open license).
Re:Automatic Functional Testing for Mac and Linux? (Score:1, Informative)
The problem is that tools like VisualTest and WinRunner can find specific objects (buttons, dropdown boxes etc... and treat them as objects. When you use such a tool with an X Server, all the tool can see is a mouse click at a specific location. This makes your automated playback screen size and window location dependent. This is not a reliable way to handle this.
WinRunner (Score:2, Informative)
Mercury Interactive [mercuryinteractive.com]
Hope that helps
Joe
Pounder and Abbot (Score:1, Informative)
http://sourceforge.net/projects/pounder
http:/
Re:testing tools (Score:3, Informative)
IANATester.
Re:testing tools (Score:1, Informative)
Also, it is designed to give perforance stats on web-based apps, not checking for failures like the person asked for (although I guess you could look at the number of 404 errors and then track them through the audit log for each test).
Also another major drawback is that it seems to hook into several components of NT/2000/XP to run the tests so it won't work on any other OS's, although you can test whatever type of web server and OS you want.
Well thats my 2p on the subject...
* This Christmas give your child a gift they wont forget, Mental Blocks! *