Programming Environment For "Event Correlation"? 25
sireenmalik asks: "Of late I have become interested in this field of research namely Event Correlation on a Distributed Network System. The more I read about it, the more ignorant I feel. There is so much to it: distributed network systems, databasing, artifical intelligence (neural networks, baysian belief networks, rule based,etc.), software engineering, computer science, telecommmunication....etc. If I were to really attack it from a programming point of view, can somebody tell me what tools and languages should I use? I suppose it will be a realtime environment. Academicians support ADA but I can't figure how the artificial intelligence part will be done. If I use PROLOG/LISP I get into HEAP management business which really is a dragon for realtime systems. C/C++ .. Java....? To add the list I also know about the diverse implementations using JIRO (from SUN), ECDL (from HP), RAPIDE (from Stanford.edu), JAVA Management API, ELAVA, GEM Language, MODEL Language, IF/PROLOG......and the list goes on and on and on! It's interesting as well as confusing (I can't help but agree here). Let's talk about it. Maybe something useful happens here?"
Personally (Score:1)
And then for serious stuff, I'd use something more structured and formal. Personally, I like Java; it's well known, pretty well supported, good at handling errors, has a reasonable amount of network stuff built in or available, and from the beginning had threading in mind. But I would certainly use some sort of OO language for this, as this kind of work strikes me as well-suited for an OO approach.
Oh, and do your work incrementally! Avoid grand plans; code in an exploratory fashion. If you're doing OO work, the book Refactoring [isbn.nu] is a great one. And if you're doing incremental design in Java, you will soon grow addicted to unit testing; I have grown to love JUnit [junit.org], a Java unit testing framework.
But if this is mainly for your own use, use whatever tools you are most familiar with. I find that tackling a new tool and a new problem domain at the same time is generally too much to allow quality work. New problem? Old tool. New tool? Old problem first!
realtime collection, offline analysis. (Score:3)
Theres several advantages to this approach:
- you don't have to have such a fast machine
- the data collection software can be *simple*
- you don't alter the data collection software when you alter your analysis
- you have the raw data to hand for applying more analysis if you need to do a second pass.
For real-time processing I would look at using an offline analysis to generate state machines for recognizing events. And I would get these machines to *generate* events into the stream as well. That way you can build your analysis hierarchically by recognizing subpatterns and building patterns from them.
In any case, from a practical standpoint 'real-time' processing would not spot some of the most interesting things - such as an event pattern recurring close to a regular period of minutes,hours,days,weeks... - eg network failures due to load and due to incorrect scheduled jobs have a differnt appearance - both occur regularly but the schedule failure would have a more precisely regular period. Unless you plan to accumulate state over long periods of time and watch for such things I reckon you'll miss a lot of important recurrences.
"Event Correlation" (Score:2)
DARPA archives should help you a lot.
[glad I'm out of THAT arena]
/(o\ I'm not a medievalist - I just play one on weekends!
SQUID DECLARE ANAL-SWATTER BLOOD RIVER (Score:1)
jizzum powered lifeboat. tidal wave, a flood of rectal
mucus, jolly laughing bear wipes pickle off his face.
burping brown bubbles, shit smeared nasal hole. getaway
spinning phallus suction cup popping. rusty metal dustpan
scraping dead paint, airborn virus lead paint wall-licking
little boy. "mommy watch me lick the wall" said little
fredrick. he banged his head against the floor, bloody
snot catfood. his mom wiped up the mess with vinegar and
toilet paper, remembering to clean the ears with a fork.
green fog filled the cavern, red rocks flecked with
gold each a blinking nose. hot stuffy air infecting
ashy lungs with speared micromen. toxic waste released
into bloodstream. wild orgy of man fuck apocalypse.
hyperdimensional spear fuck rotten scalp meat blast off
display. the little men jumped out of fleshy gold noses
and fell, hundreds dropping like gassed jews. falling
falling ass spread wide, smack landing on sharp clean
spears. misty yellow and brown. others gag on the
smoke, puking out innards. warpole ass ram fiery phallus
anal splinter. newly birthed micro fetus budding and
popping off eggs. splash of bloody and shit. upturned
noses fill with smog and thick juicy flesh. hyper
vibrate rumbling cavern walls, scalding lava flashing
red, green, blue, steaming. dead fetii hung on a
clothesline to keep them out of the horrid storm. they
cry for more as string is run in rectum and out the
heart, pulsating throbbing yellow. screams of micro
baby fish, countered by puke and shit. crisscrossed
spiders lair net of jewels each shiny anus reflecting
shit splatter. explosion of smog and lights. still the
micro men jump to their anal death, smack on the shiny
spear. glare of spearheads and shiny fish ass. rising
tide, mixed blue green. the walls worn away by waves
of hot jizzum spear smacked erections. brown spots shot
in red walls. melting walls. strings pulled tighter
baby fish sliding back and forth indian burned asshole
heart skinned. slosh slosh slosh. the sky opens up,
golden light pierces the cave, melting rotten bloody
flesh, smog rushes out. golden noses spurt bloody
fireworks. dry baked cracking walls crumble. fault
lines rumble, arise the demons. a swirl of black smoke
and laughter. arms and legs, half-rectums, dead fish,
nose pieces all whip against the walls hyper smack.
flesh dangles from hooks and smacks dead bodies.
equilibrium.
Re:realtime collection, offline analysis. (Score:2)
Like, say, trying to correlate events from a network monitor and deciding if a massive DOS / hack attack is hitting you.
Re:realtime collection, offline analysis. (Score:1)
Re:Crapflooding is fun (Score:1)
Re:realtime collection, offline analysis. (Score:1)
Anyway, the idea was to reduce the amount of damage a network outage could cause. For example if the route hosting your path to your servers failed it is better to get a page or notification simply for that and not for the 20 server that are no longer responding. It also allowed for an automatic escallation method such that if the server did not respond to a ping the system would wait for a minute to see if it was just a packet drop vs. a real outage.
The programming was pretty simple. Pick your events and then tell the system how they correlate. Then if it reaches level X have it generate a new event for the base system to handle. It was actually pretty simple and was more drag and drop than real programming. The programming came when you wanted the system to check something that was not one of the normal services. For example, making sure that the LDAP service was still up and authenticating. As long as you can program a way to communicate with the correlation and collection engine (HP OpenView as the base--the other products ran on top of that) you would be fine. The stuff I wrote generally included a mix of SNMP traps via Perl. Although I did not write an expanded MIB to acutally handle this, I simple repurposed some existing ones and then tweaked the sent messages and parsed it with the engine. For what we were doing speed was better than perfection (at least according to my manager at the time).
We also kept data stores of other SNMP type things (performance type data) which were then analyzed by a nightly SAS run. But that is slightly off topic.
-Joel
Floating shit (Score:1)
Why not Ruby then (Score:1)
Other than that, I agree.
Shit my pants (Score:1)
Re:Shit my pants (Score:1)
Crapflooding is fun (Score:1)
Farts and shits (Score:1)
Re:Floating shit (Score:1)
Re:First poast, biatches!! (Score:1)
*throws a childlike fit*
Re:realtime collection, offline analysis. (Score:1)
Re:"Event Correlation" (Score:1)
First poast, biatches!! (Score:1)
Re:Crapdot (Score:1)
*shits on your face a little*
Is that better?
Re:Crapflooding is fun (Score:1)
Re:Why not Ruby then (Score:1)
Crapdot (Score:1)
Re:Shit my pants (Score:1)
Re:Crapdot (Score:1)