Open Source Batch Management? 37
Asgard asks: "My employer is currently running a commercial batch management platform. Unfortunately the licensing model makes it unfeasible to run it in the development / testing environments, leading to poor usage of the tool and unexpected failures in production. I'm looking for an equivalent Open Source tool and am wondering how others have approached the problem. Does Slashdot have any suggestions?" Imagine a system like cron, but with job dependencies. Are there any batch systems out there like this?
"The tools I've found through web searches mostly treat 'batch management' from the cluster perspective -- a user submits an ad-hoc job and the tool figures out where and when to run it based on load and architecture requirements. Instead I am looking for something that manages daily schedules of jobs based on their dependencies with other jobs and external events, such as files arriving or time.
An example might be that every day jobs a, b, c, and d must run. Job a must not run before 9pm and requires file X to be present. Jobs b and c depend on a completing successfully. Job d must run after 2am and after b and c have completed successfully. If job c fails then an operator must fix the issue and rerun it, after which the tool will move on to job d. "
I think this question was asked before... (Score:1)
Please, some manners... (Score:1, Informative)
For a second there ... (Score:4, Funny)
I thought it said Open Source Bath Management.
Maybe I speak for myself, but some things are better off left proprietary.
Re:For a second there ... (Score:2)
You know, I saw "Open Source Bitch Management.
I was unclear as to its function. i-Pimp 2.0 or something.
Would you be doing adhoc runs? (Score:2)
Systems like this are handy (Score:4, Interesting)
A few years ago I was working in change control, and updates to software stored on network shares across the company were handled using a decrepit old VB app that generated linear xcopy scripts that updated each server (of which there were about 160 spread across the US) one by one. Most of the servers were on slow links, so distributing a 10MB file could take twelve hours or more.
I hadn't learned to code properly at that time, but we used an enterprise batch scheduler called Control-M* that worked like the original post describes. What I did was wrote a batch script that read a config file and then executed a single robocopy command targeted at the server in the Control-M job definition.
I had a whole array of these jobs, one for every target server, and they all depended on another job that would run at - for example - 11PM. So when that time rolled around, all of the dependent jobs could run. As-is, that would have overloaded the WAN and source server bandwidth. So I assigned what Control-M called a "resource" to all of the jobs. It was just an integer counter that I capped at 16. So at any given time, there were 16 "threads" of robocopy running. It ended up being between 20 and 30 times more efficient than the crappy xcopy scripts.
Anyway, they're really handy, and if there isn't an OSS project like this, it would be a great idea.
* This is not an endorsement of Control-M. In my new(er) job, I'm working as an engineer, and I discovered that the encryption system that it uses for storing account passwords in the registry is so poor that I was able to write a universal decoder for it using only vbscript and Excel. There are certainly other downsides to the app as well, although one cool thing is it runs on just about any platform - Unix, AS/400, OS/390, Windows, etc.
Re:Systems like this are handy (Score:2, Funny)
And you just adimtted to the world that you violated the DMCA. Pity, I was starting to like you...
Re:Systems like this are handy (Score:2)
It was for security testing purposes. I submitted my findings to the vendor, who didn't act on them to my knowledge.
OSS Systems like this are handy (Score:4, Informative)
A search of google or any of the OSS sites should turn up several more.
a system like cron, but with job dependencies (Score:4, Interesting)
Build systems aren't just for running compilers.
cron + make + caffeine (Score:3, Informative)
It works great for me. Just have to do a caffeine check before making major changes (and remember to stop the cron job plus test in a sandbox).
Some handy tips:
For example: (Score:5, Informative)
make -j $NCPUs -C
all: tasks/1 tasks/2 tasks/3
tasks/1:
foo bar baz
frob fritz
touch tasks/1
tasks/2: tasks/2.1 tasks/2.2 some_make_test(tasks/2.3)
bar baz qyzzy and touch tasks/2
etc. etc. etc.
Clarifications (Score:4, Informative)
2) For a batch automation system, you'll need to use make -k, and if you need to, put targets in
3) If you have a dependancy chain of targets and you don't want to have to clean up explicitly (or you want your job to run entirely in phases), you can label intermediate targets with
4) Make sure to fully outline dependancies. If you need to somehow prevent two things from running in parallel, you need to create an artificial barrier with the script itself unfortunately. The easiest way to do this would be perl and IPC::SysV, I should think. You might know of some other shell tricks or opening a device that blocks like a FIFO... but it sucks that gnu make doesn't have it. (However HP-UX and SCO's make have a
Re:a system like cron, but with job dependencies (Score:2)
Re:a system like cron, but with job dependencies (Score:1, Flamebait)
You attack my suggestion as being wasteful or suboptimal, but at the time I posted there were no solutions.
I hope it feels good to have pointed out the inadequacies of my attempt to provide some initial direction, especially given the fact that at the time I write this, 22 hours after th
Re:a system like cron, but with job dependencies (Score:2)
Re:a system like cron, but with job dependencies (Score:1, Flamebait)
Congratulations, you're special. Just like every other flamebaiting troll.
That's not true... (Score:1)
make can told to proceed as far as possible with missing results. If you keep running it every so often, it will eventually get all it's dependancies as soon as possible and produce the "final" result. (These results are intermediate files that just checkpoint progress... unless you are using a custom make test)
What's interesting is that you can ask make to treat "dependancies" as either a all-present or a do-in-order type of thing (or both). Even cooler
Condor (Score:2)
DOS (Score:1, Funny)
Workflow systems... (Score:1)
Get a real vendor... (Score:2)
We use an enterprise scheduling system called AutoSys which is suppose to be the industry st
Cluster solutions work in single-machine mode too (Score:2, Insightful)
management, but for clusters of machines.
There's nothing that says you can't have
a cluster of 1 machine though.
Use Gridengine (Score:1, Interesting)
It handles batch jobs, dependancies etc etc.
TORQUE Resource Manager (Score:5, Informative)
Sounds familiar (Score:2)
e.g. run Job A when job X has successfully completed and file P has been updated, run job B if job X hasn't run by 3am, run job C if job Y fails.
I've got most of the rough design done, the main problem is specifiying date/time information - I would like to say "every 2n
Re:Sounds familiar (Score:1, Funny)
What, the project manager couldn't squeeze "devs must hit themselves in the balls with a hammer each hour" into the requirements list?
Re:Sounds familiar (Score:2)
Actually there's no project manager for this yet, just me. I'm, er, doing a feasibility study at present, of course sometimes the only way to determine if something is possible is to do it...
Suggestions (Score:2, Informative)
Apache Ant (Score:2)
Apache's Ant [apache.org] may be worth a look. It handles dependancies very well. It may not be so great with timing of jobs (cron + ant?) or handling jobs running in parallel (ant plus a custom 'run task in the background'?).
--
Linux Server + Persistence => Solution [rimuhosting.com]
Roll your own... (Score:2)
Really. It doesn't sound like it would be too difficult to write this yourself with some good Unix scripting (Perl, bash, etc.)
You said it's to serve as a test system for a commercial application. I assume you already have a "schedule" in mind, so maybe you could simplify things a bit by writing a system that only runs your specific schedule, rather than writing something more general. I don't know if that would provide a valid test case for your purpose.
Fcron? (Score:1)
http://fcron.free.fr/ [fcron.free.fr]
We need Open Source AutoSys replacement (Score:1)
I've been looking (waiting?) for an open source equivalent. What we really need is something like Condor and Globus, ala the NSF Cluster Toolkit, with a cron interface (cluster-centric solutions have great features like redirection of STDOUT and STDERR, but don't have the ability to schedule a job for later execution.) Java W