Automatic Functional Testing for Mac and Linux? 202
"For example, a vendor has a database interface, and wants to know how many times we can access a series of records, what the results show (are the right, blank, time out?), and how long it took for the search to take place during various times of the day, and so on. So we emulate one of their data entry people by logging onto the system, putting in some search parameter from a script, and then record what the text (or error message) was when it came back. Yes, we could connect directly to the SQL database without their front end, but it's the combination of their front end and the database we usually test. We have found many examples where were can connect raw, for instance, but the front end displays a timeout. We also test things like 'What happens if the user enters a colon or asterisk?' or 'What if the text entered in the box is over 256k?' Then we tell the vendor the results, and they hopefully fix them based on the data we gave them."
AppleScript (Score:3, Informative)
Re:AppleScript (Score:1, Informative)
Re:AppleScript (Score:2)
I think this guys best bet is to develop using cocoa and to give a timed public beta to the mac community.
However, I couldn't hurt to check out Apple's developer connection page as well. There is a -lot- of good info there. Ask some developers at Apple. Moreover, try to ask some developers from the OmniGroup ( http://www.omnigroup.com ). They do a lot of consulting, they are friendly, and they have been developing with NeXT, Rhapsody, and OS X for years.
Re:AppleScript (Score:2)
If you can read and write English, you can become a pretty decent AppleScripter in a few hours. Pick up a decent book on it, and you'll be making your Mac do backflips in a few days.
I've used AppleScript to write a CGI that duplicates the functionality of Outlook Web Access with Microsoft Entourage.
I also make extensive use of it in my web-based control system for my home automation rig-- hell, AppleScript is the BACKBONE of my home automation rig, gluing together various apps.
In about 20 minutes one day, I wrote a script to strip out the stupid, extraneous shit from "CNN Breaking News" e-mails, create a new message with nothing but the actual news item in it, and send it along to my cell phone.
One of my clients (a huge, multi-national food company whose products are mm mm good) uses two AppleScript CGIs that I wrote for searching job archives they have on their internal web site.
Sure, sometimes I've had to write a small 'test' script on the side to help myself understand how to extract a particular data item from a record, but it's a relatively painless process.
Whenever you have trouble with a scriptable app not made by Apple, blame that app's vendor for not doing a good enough job at making it scriptable, DON'T blame AppleScript.
~Philly
Re:AppleScript (Score:2)
What he needs is a load test and functional test product similar to Rational's, but cross-platform. I'd suggest looking into Mercury Interactive's software, though I'm not sure just how cross-platform it is.
Re:AppleScript (Score:2, Informative)
Re:AppleScript (Score:3, Informative)
Someone mentioned server load testing earlier... if that's what you're looking for, remember that Mac OS X is basically a pretty window manager on top of a flavor of FreeBSD -- you can have cron automate shell scripts and perl.
Re:AppleScript (Score:1)
Re:AppleScript (Score:1)
there is one called player (as in player piano) that seems to work pretty well...
Re:AppleScript (Score:2, Informative)
Timing metrics... (Score:1)
OTOH, Real life timing is really important for scalability testing, i.e. the domain of load testing tools, which mimic front end client behaviour in real time via back-end automation. These are definitely far more reliable, but also exceedingly more expensive.
Re:AppleScript (Score:1)
No, you can't. How does AppleScript "simulate mouse clicks, text entry, menu pulldowns?"
Answer: it doesn't. It calls the application directly, skipping the UI. Which is not so good if you want to do user interface testing.
testing tools (Score:5, Informative)
www.opensta.org
There is a commercial company that provides support at www.opensta.com
Re:testing tools (Score:1)
Re:testing tools (Score:1)
Re:testing tools (Score:3, Informative)
IANATester.
For what it's worth... (Score:1)
Re:For what it's worth... (Score:1)
Hmm. (Score:5, Informative)
Don't be scared by the name "script"...there are visual tools for ascript and the language itself is so simple my mom could do it.
Re:Hmm. (Score:1)
Re:Hmm. (Score:1)
Re:Hmm. (Score:1)
Re:Hmm. (Score:1)
Seems plausible to me. Maybe I'm weird.
Slashdot math is at it again (Score:2, Funny)
Let's see here... 1993-2002 is roughly 20 years?
Has CowboyNeal been teaching you "new math"?
Re:Slashdot math is at it again (Score:3, Funny)
It is if you're using base 5 - due to a tragic accident involving penut-butter, weasels, and a large bowling-ball; and thus only have one hand.
Re:Slashdot math is at it again (Score:1)
Re:Slashdot math is at it again (Score:1)
Re:Slashdot math is at it again (Score:1)
Re:Slashdot math is at it again (Score:1)
Re:Slashdot math is at it again"Think Different" (Score:2)
"Think Different"
--toq
AppleScript doesn't do what he wants (Score:2)
Re:AppleScript doesn't do what he wants (Score:2)
Actually I believe it does have calls to do UI-level actions - mouse clicks for instance.
XRunner (Score:5, Informative)
Re:XRunner (Score:1, Informative)
Re:XRunner (Score:1)
But I agree, WinRunner blows. So do many of the competing products, (ahem, QARun) which all seem to lack serious QA prior to release. This is what you get when you've have to fight over a few thousand customers who are used to paying ridiculous amounts of money for broken enterprise software.
Couldn't you use PERL? to do a lot? (Score:1, Insightful)
Re:Couldn't you use PERL? to do a lot? (Score:2)
<g> IIRC, it's called Visual Test because it used to be Microsoft Visual Test and was bundled with Visual Studio. Then they sold it (back?) to Rational.
Java... sort of (Score:4, Insightful)
Obviously this would work on Linux, OS X, and Windows as well.
Hope this helps.
Robot and BeanShell (Score:2, Interesting)
Re:Java... sort of (Score:1, Funny)
to quote the documentation: I've used it to make remote control apps that work in windows and linux: take a screenshot of desktop A, send it to B. Send mouse events from B to A. Freak out user on machine A.
Quite simple, and easy to do in Mac OS as well!
Re:Java... sort of (Score:2)
At work here I suggested as a product that was in beta, on the basis that it looked interesting and "why not give it a shot" and the other recommendations were so expensive. And the guys in testing got a trial copy and have been very happy with it, though I haven't used it myself. It is qftest [www.qfs.de] from Quality Software First. We build our java stuff for Win2K , Linux and Solaris. And this seems to do the job well. And its dirt cheap compared to anything from Rational or Mercury where you've gotta shell out big time.
Legos! (Score:5, Funny)
All you need is a cat (Score:2, Funny)
Re:All you need is a cat (Score:2)
Re:Legos! (Score:1)
one lego
two lego
three lego
one million lego
Re:Legos! (Score:1)
Actually, LEGO is an adjective, so has no plural.
one LEGO brick
two LEGO bricks
Re:Legos! (Score:1)
Funny thing is, I remember thinking, why not just use a microprocessor hooked up to a keyboard controller, and plug the thing in thru the external keyboard port??? (yes, most laptops back then had 'em)
We're working on one... (Score:3, Informative)
The name of the project will be HASTE, look for it in a month or two.
Re:We're working on one... (Score:2, Informative)
XTest and Expect ? (Score:4, Informative)
Expect [nist.gov] allows us to do on Linux
(and a vareity of other operating systems) ?
Re:XTest and Expect ? (Score:2, Informative)
Re:XTest and Expect ? (Score:4, Informative)
(FWIW, my job very recently involved buying or writing an automated test scaffold for graphical apps on Linux that would work across architectures and windowing systems -- it turned out that the latter was necessary, as the available commercial products just weren't/aren't cross-platform capable, and android has some serious faults which I'll expound on if asked. The product partially exists, having been put on hold for other internal development... perhaps when it's finished we'll release it under an open license).
Automatic Functional Testing for Mac and Linux? (Score:2, Insightful)
Re:Automatic Functional Testing for Mac and Linux? (Score:1, Informative)
The problem is that tools like VisualTest and WinRunner can find specific objects (buttons, dropdown boxes etc... and treat them as objects. When you use such a tool with an X Server, all the tool can see is a mouse click at a specific location. This makes your automated playback screen size and window location dependent. This is not a reliable way to handle this.
a little offtopic, but.... (Score:1, Flamebait)
It's called freshmeat and google. (Score:1)
Okay there's an offchance that another slashdotter might be working on an esotric project like he is and happens to have a viable solution. Highly unlikely. Or perhpas slashdot just wants to bring this to our attention. Don't know why though? Or mabey IT'S SHAMLESS ADVERTISEMENT PLUGGING yet again =P
Go ahead and give me your worst repliers.
Re:It's called freshmeat and google. (Score:1)
Re:It's called freshmeat and google. (Score:1)
Actually, I usually scan these posts, they often spin out some very interesting viewpoints and experiences, especially when the topic is a development or testing tool, I probably encountered ant on
My theory is that these are usually people who have not been trolling sourceforge or building linux boxes fgor years, are in fact probably coming out from the dark side. Have some compassion for them!
Rational preVue-X (Score:1)
xnee (Score:3, Interesting)
you could always port it to OS X ... :-)
Check out Test Quest Pro (Score:3, Informative)
It's fully scriptable using a c-ish syntax, and comes with a nice IDE and runtime environment to get the work done. It seems like it may be a nice match for what you want to do.
Here [testquest.com] is more info.
I've got two words for you... (Score:5, Funny)
Re:I've got two words for you... (Score:1)
Homer: Stop that! You're a helper monkey! This isn't helping!
Re:I've got two words for you... (Score:2)
You might also look into pigeons [google.com].
Expect (Score:2)
Here's a website that talks about the language: http://expect.nist.gov/. I bet there's other sites so check google. O'Reiley also has a book about it. I still think I'd use Applescript under MacOS, but under Linux I'd give expect a try. I played with it a little under Solaris and it looked pretty useful. I was able to automate the setup for US Robotics modem racks and Cabletron ethernet hubs using expect. Never did get that stuff totally debugged and my script would go crazy, but that's not the language's fault.
Re:Expect (Score:2)
Depends (Score:1)
Re:Depends (Score:1)
(Disclaimer: QA is what I do for a living; I've got "joke immunity")
For Mac OS X... (Score:2)
F-Script (Score:3, Informative)
For Mac OS X have a look at F-Script. [fscript.org] You can inspect a running program and call any method in that program and any method on any object in the Objective-C runtime. This should allow you to generate events or enter text in text fields, etc.
Of course this will only work for programs developed using Cocoa. Can't help you with Carbon apps. You might be able to write something using the Apple Event manager.
Last time I had to do this for Linux (and Gtk+) I had to roll my own.
Use Google PigeonRank Technology (TM) ! (Score:1)
Google [google.com]
xmacro (Score:1)
Xmacro [sourceforge.net]
Funny (Score:3, Interesting)
BTW, don't use xse, which is a wrapper for XSendEvent, if you roll your own. There are so many places where it just doesn't work, or doesn't work right, because the XSendEvent function just doesn't work reliably. I lost a lot of sleep over that before I gave up and just wrote xte.
QARun, QADirector? (Score:2)
Re:QARun, QADirector? (Score:3, Informative)
QADirector is not a testing language in itself. It is a harness from which to run the script generated by script writers. It used to be a Unix app, but when the switch was made to Wintel there were a few missteps. If you are looking for a test harness that is comprehensive (for Windows anyways) in its command-line options, look into QAD, otherwise I would go with Test Director by Mercury or maybe Segue's new product (I believe its called SilkPlan or something similar).
QARun on the other hand is a pain in the ass. If you are looking for automation, don't use it. It's designed for straight-up "record and playback" style testing. If you wish to deploy these tests to client machines (something which some, but not all people want to do), do not use QARun. You can look into XRunner (it is the predecessor of WinRunner) by Mercury if it is still being sold/supported for *nix testing. I honestly don't know about Mac, our product doesn't run on it.
If you want a serious evaluation document, I highly suggest heading over to http://www.qaforums.com [qaforums.com] and head to the Automation section. Post a query about the Mac and you'll probably receive a response from one of the moderators with a white paper that was written comparing all of the available tools.
DISCLAIMER: My group uses QARun, QAD, Segue Silktest, and WinRunner, so as not to be accused of bias.
QuicKeys X (Score:1)
I'll do it! (Score:1)
Finally, I've discovered a use for this strange box on my desk.
Rational Software and Linux don't mix (Score:2, Interesting)
I'll tell you, it's a major hassle to develop a product for Linux while being forced to debug it on Solaris because that's the only place our debug tools will run. The first company to make decent Linux tools that are similar to Rational's will have my business. And Rational won't.
Re:Rational Software and Linux don't mix (Score:2)
I'm not impressed with their testing software - we paid for Rational Robot (£20,000 - something like 6 months software budget for a small company like ours) and a programmer has been trying to work out how to use it for another 6 months. It still can't test reliably. For that time/money we could have just had the programmer actually doing the damned testing themselves.
Applescript, QuicKeys for Mac OS X and OSA (Score:5, Informative)
However, AppleScript is not the only scripting environment for the Mac. Underlying all of the native scripting languages on the Mac is something called the Open Scripting Architecture [apple.com]. This allows any OSA component (i.e. scripting language) to talk to any other OSA component.
One OSA component is QuicKeys [quickeys.com], a great product that actually does simulate mouse clicks, keystrokes, etc., and is completely scriptable. It integrates completely with AppleScript and via the power of the OSA, with shell scripts and Perl as well.
On top of this, Apple provides a powerful GUI building tool for AppleScript called AppleScript Studio [apple.com], which is free. Plus, AppleScripts can talk across a network to do RPC. Imagine, you can have an AppleScript Studio front end that drives scripts on a set of machines over the network (so that you can assess performance under load). The individual scripts rely on AppleScript to drive the logic while using QuicKeys sequences to drive the GUI.
The only issue here is how easily you can assess results. There's no general way to check what the GUI is showing other than a pair of Mark I mod 0 eyeballs, since grabbing data out of the app being tested (even simulating a copy-paste) only talks to the underlying data structures and does not address GUI bugs. A possible way is to arrange the windows in a known manner, turn off the menu bar clock, take a screen shot, and use a graphical comparison tool to XOR the bitmap with a known good screen shot to highlight differences, but this won't work if you're working with variable or randomly generated data.
Disclaimer: I work for Apple, but these are my opinions only and do not represent any sort of official endorsement.
--Paul
Cross Platform Automated Testing (Score:1)
If you want a cross-platform automated testing tool that is:
1. Localizable
2. Non-Intrusive (tests the environment on which the user runs).
Then check out TestQuest [testquest.com].
While a bit pricey - consider it a capital expense - this system is fantastic. It was originally developed for testing medical devices and systems, where the FDA requires that what you test is what you ship exactly .
TestQuest hardware (Score:1)
They may have other newer systems, or a software solution for now, but for what we do it works great. We only test on PC hardware, but they have a variety of modules and interfaces for touchpads, VGA, svideo, Mac and Sparc hardware as well.
Good topic of much debate (Score:1)
QA Forums [qaforums.com].
Scroll on down until you get to the Testing Tools forum.
Java's Robot class would work (Score:1)
Java solution (Score:1)
Check out java.awt.Robot [sun.com].
~Kevin
OS projects, wine (Score:1)
A lot of open source projects could benifit from automated testing. The wine project [winehq.com] has spent nine years working on translating the windows API to X and reproducing the windows GUI.
Many people [google.com] think wine is a vital tool for greater use of linux. They rely on people downloading new versions and testing windows programs using whatever versions of X, system libraries and windows DLL's they happen to have and then posting bug reports in a newsgroup. [ms-windows.wine] There are still lots of quirks with relativly simple windows programs not redrawing or displaying dialogs correctly.
I believe that wine would progress much faster if the wine developers could make a change then click test and later, with no more effort get a report saying "3 tests which failed previously now pass, 1 test which passed on the last version now fails".
I will be very impressed if somone comes up with a free automated testing tool that allows tests to be set up easily and is powerfull enough to provide human readable results "the bitmap is drawn 1 pixel to the left", "text as expected but wrong font" "dialog appeared behind other window" etc.
DejaGnu and friends (Score:2)
There is also LTP, the Linux Test Project, which may be of use to you as well. (Do a Google search for "linux test" and you'll find it quick.)
If you're not a TCL programmer, you're probably not going to like DejaGnu very much. If this is the case, good luck with the GUI tool search; you can expect to spend some serious money for decent test tools. (The pun is simply there. If you look at it too long or pay too much attention to it, it may or may not vanish. Just chalk it up to some crazy cat and don't worry about it.)
Best wishes,
Android (Score:2)
Android runs on top of these and can be downloaded free for Linux and Unix
See the July 2001 issue of "Dr. Dobb's Journal"
for an overview of Android and how to download.
Plays back keyboard and mouse events for most UNIX implementations. (maybe even Mac OS X)
It is a little bit roll-your-own
Expect to spend some time writing perl to support your android scripts.
Goodbye Karma (Obligatory Off-Topic Amiga Post) (Score:1)
*Sniffle, wipes tear from eye* Ah, the good old days.
WinRunner (Score:2, Informative)
Mercury Interactive [mercuryinteractive.com]
Hope that helps
Joe
Re:WinRunner (Score:1)
Don't use WinRunner..
We use it at work (actually, I just got back from spending the entire day working with the software). It works perfectly until you try to "replay" something you have "recorded". Then all hell breaks loose.
They use a kind of "property map" which they use to locate GUI objects. Problem is, the GUI map keeps changing on you because the default properties they pick vary from load to load (especially with the Java add-in, IMHO). Also, it can be very tricky (i.e. almost impossible) to find the right set of properties that will uniquely identify your UI objects, and that won't change from load to load, so that you can successfully regress your software.
Blechhk! Add to that, it costs an arm and a leg.
If I had to make the decision all over again, I would say skip the UI testing, and do unit testing on your classes instead. Then hire some monkeys (verification types) to sanitize the UI part on a semi-regular basis.
Rational suite (Score:1)
If you want to get job done right, I would recommend eXtreeme Programming, over most other ideologies.
2c,
p.
ATK for GTK+ apps. (Score:2)
Gerd? (Score:1)
Re:In case it gets slashdotted (Score:5, Informative)
Re:Automatic Mouse Movement (Score:1)
Re:Automatic Mouse Movement (Score:1)
Re:Here's a good automatic functional test (Score:2, Insightful)
Yeah, right. That paraphases exactly why we've used VHS for 20+ years instead of Beta. NOT! It is arguable that the Apple was a superior computing platform to the IBM PC of its day. The IBM-PC had the full corporate power and marketing strength of, well, IBM behind it while the Apple had the comparatively limited resources of two guys named Steve.
For the uninformed... Back in the days when the VCR was just being released on the world there were two competing formats, JVC's VHS format (which we all know) and Sony's Beta (a.k.a BetaMax). The latter was far superior but, alas, was first out marketed (a bit of history [urbanlegends.com]), then out litigated (Read case story [hrrc.org]). To date the majority of video production houses still use Beta because of it's superior image and audio quality.
VHS Beta (Score:1)
1) They're made by Sony 2) They can use the same size cassette shell 3) They have Beta in their name
They are all professional formats and are not the same as BetaMax, BetaMax II and EDBeta of old....
Re:Here is a possible solution (Score:2)
You could roll your own based on VNC by creating your own VNC client that generates it's own scriptable mouse and keyboard events.
Then just run the standard server on the tested system.
-Peter