Information for Managers - Understanding pthreads? 69
dnotj asks: "The boss (who is very technically astute) says: NO to using pthreads in any of our production applications. He wants us to do things the old fashion way (fork(), exec(), shared memory, etc). His reason for this is that he doesn't understand pthreads (by his own admission). Hence, he is limiting us to using methods and techniques that he understands. He is reasonable and would see our side (the developers) if presented with enough understanding in a satisfactory format. What I'm looking for are document technically detailed yet directed more towards management. Not something on the level of 'pthreads for Dummies', but more along the lines of 'pthreads for Managers'. Any suggestions? URLs or Books are fine."
Why ? (Score:1)
Secondly, if he wants to know it, and already understands fork/exec approache, it should be very easy for you developers to exlain what threads are, and the various synchronization primitives provided. Thats basically all he needs to know, the pthreads api in detail shouldn't concern him unless he wants to program.
What he might be concerned about is portability, remember pthreads are not fully supported on many platforms, and also there are many surprises on other platforms.
this [llnl.gov] chapter should be all he needs to know + portability issues.
Re:Why ? (Score:2)
Linux is for people who hate Windows, BSD is for people who love UNIX
But separate processes ARE better. (Score:3, Informative)
Re:But separate processes ARE better. (Score:2)
Forking has a context switch to deal with. That's a fact. Now if I was doing some heavy mathematical computation that required new processes every second for a long period of time, threads would win out.
If i need to split off the process once per day, forking is fine.
As for protection, it's a mutex lock. Learn to use them properly, no deadlocks or ill use of resources, and you are fine. It's extra work for using threads, but isn't it worth it if you save thousands of context switches on fork()ing per second if you need to do parallel processing to that scale?
Oi? (Score:1)
Yes, switching between different processes is usually slower than between threads in the same process. If this is your concern, you probably better off having exactly one thread/process per CPU.
Mutex lock is an excellent protection against deadlocks (you need to use it with shared memory anyway), but it's less useful against plain old fashioned core dumps.
Re:Oi? (Score:2)
Re:But separate processes ARE better. (Score:1, Interesting)
Thread creation must have a context switch too... and even more, threads on linux are in fact processes. Process creation on modern Unixes is very cheap. (Windows is a totally different story - process creation is way more expensive than thread creation - that's why they push the concept of threads so agressively). AFAIK Threads are not really a big win in most cases. Sometimes, an user level thread implementation may be faster than a kernel thread implementation. (Read about KSE on FreeBSD and you'll know why). Threads are a win only if you're running on a SMP box, and even then, you must be careful how to use them(coarse locking vs fine grained locking).
As for programming concepts, threads may sometimes be a more elegant solution than using a state machine(threads vs socket multiplexing with select() for example).
>>If i need to split off the process once per day, forking is fine.
Bullshit.
>> It's extra work for using threads, but isn't it worth it if you save thousands of context switches on fork()ing per second if you need to do parallel processing to that scale?
As I said earlier, even threads have conext switches(depending on the implementation, a thread context switch might be cheaper(LWP on Solaris) than a process context switch, but it's a context switch in either case). Only when you're using some sort of user-level threads (ex. GNU/pth) you're context-switch free, but you cannot run on SMP.
People mistankely think that using threads will increase performance, but it's not true for many cases.
Re:But separate processes ARE better. (Score:3, Interesting)
Some context switches are cheaper than others
This is the only time I'll respond to an AC since "bullshit" is apparently an argument now-a-days.
Re:But separate processes ARE better. (Score:1)
But why bother with threads? Use one process per CPU (if needed) and accept computational tasks using either shared memory or message-passing.
Event-queues! There's the ticket! Single-process event queues..! Ha ha! Be a masochist with me and join the event-queue fanclub!
Re:But separate processes ARE better. (Score:2, Funny)
Only in Sex Education and Operating Systems I.
Re:But separate processes ARE better. (Score:4, Informative)
taken from http://basil.cs.uwp.edu/Cs370/notes/Threads.doc [uwp.edu]:
Advantages (Thread vs. Process):
-Much quicker to create a thread than a process.
-Much quicker to switch between threads than to switch between processes.
-Threads share data easily
Disadvantages (Thread vs. Process):
-No security between threads:
---One thread can stomp on another thread's data.
-For threads which are supported by user thread package instead of the kernel:
---If one thread blocks, all threads in task block.
Processes are lightweight in linux (Score:2)
Processes get you a number of advantages above.
Re:But separate processes ARE better. (Score:3, Informative)
> Advantages (Thread vs. Process):
> - Much quicker to create a thread than a process.
"Much" is the nit I am picking here. The main difference between a process and a thread is that threads share the VM. Modern operating systems do Copy-On-Write (COW) for process creation. Hence, the diff between threads and processes is the creation of Page Table Entries (PTEs) in the process example (but that isn't as dramatically slow as copying the pages). Given that VM issues are the biggest performance issue in proccess creation versus thread creation, with COW you should only see minor to moderate speed hit for process creation versus thread creation.
> -Much quicker to switch between threads than to switch between processes.
That is again an implementation issue, and not true in the general case. Solaris makes context switching between processes slow compared to context switching between threads. Don't think of Solaris as a good example of a performance operating system (they have other things just not performance). An old benchmark comparison I saw, (circa 1998), it was lmbench running on Linux and Solaris 2.5.1 on the same UltraSparce hardware. The bottom line is that Linux's process context switching was faster than Solaris' thread (LWP) context switching.
> -Threads share data easily
Urr! I know you meant that it is easy because you can just pass pointers versus using some data sharing API like SysV shared memory. But I think threads are difficult to share memory _effectively_. Either you lock shared resources, or you get clever with thread safe algorithms/data structures. And that stuff gets real HARD real FAST.
I just realized that you were including user space threading packages in your definition of threads. Yeah they can be pretty fast.
If you want to expand your mind beyond this simpletons game of debating threads. Check out event driven systems, and co-routines. There are different trade offs for sure, but they are easier to debug than threads.
Re:But separate processes ARE better. (Score:1)
However, when you say...
But I think threads are difficult to share memory _effectively_. Either you lock shared resources, or you get clever with thread safe algorithms/data structures. And that stuff gets real HARD real FAST.
you have to realize that sharing data structures amongst processes are often times going to require using similar algorithms ( like your typical muliple writer/multiple reader ones) but with semaphores. The API might be different between using a pthread mutex and a semaphore, but where you place those p() and v() calls are essentially the same.
Re:But separate processes ARE better. (Score:2)
Your point it correct. I really wanted to make the point about debugging code using shared resources, not writing the code. I meant "effective" in the sense of limiting opportunity for bugs, and quickly killing the ones that show up.
It is easier to debug processes than threads (re: availability and maturity of tools). It is conceptually easier to focus on debugging known limitied shared resources.
A point I didn't originally make, was that context switching is most important when you are switching alot (duh!). First, you don't want to be switching alot that is called "load" and you are thrashing your caches whether or not it is threads or proceses. However, Context switching does naturally occur when you are message passing between two processes/threads (another reason micro-kernels are inevitably slow). I think message passing is really good, but the rate and size of messages can really blow performance with context switching, cache thrashing, and marshalling/unmarshalling. I still like it.
Re:But separate processes ARE better. (Score:3)
> "Much" is the nit I am picking here. The main
> difference between a process and a thread is that
> threads share the VM. Modern operating systems do
> Copy-On-Write (COW) for process creation.
Modem UNIX-LIKE operating systems, yes. But the all-processes-descend-from-init model is not the only one in existance. In fact, Win32 doesn't have a native fork() call at all. And it does have at least one native pthreads library (http://sources.redhat.com/pthreads-win32/).
So, under Win32, thread creation is *much* faster than process creation, for a number of reasons (and COW is only part of that).
--Rob
(Cygwin pthreads maintainer)
Re:But separate processes ARE better. (Score:2)
I am aware of the init-fork model versus VMS/Win32 model of spawning processes without any inheritance.
I would love to know what makes spawn() or CreateProcess() (I think those are the respective calls) so much slower in the thread versus process? I realize creating the task struct and inititializing its parts is greater than thread init, but is it the same order of magnitude of slowness as on old non-COW operating systems?
Re:But separate processes ARE better. (Score:2)
On WIN32, creating a process involves:
* Allocating a new thread and process kernel structs (~ Same overhead as COW fork())
* Map the process to be loaded into memory (COW fork() doesn't do this, but exec() does)
* Recursively resolve all unresolved symbols. - which includes looking through the oftimes long PATH variable, as win32 doesn't have a separate LD_PATH. (COW fork() doesn't do this, but exec() does)
* Attach all the loaded dll's (COW fork doesn't do this. exec() has something similar, IIRC).
* Finally, the WinMain function gets to run (fork() doesn't start at main again, but exec() does).
So, the fork + exec model is roughly the same overhead as CreateProcess. The fork(), serve and exit() model is much lower on unix-like os's that support COW fork() because theres less to do. And any OS that has no kernel fork() call is way behind to start with.
Hope that helps. I've probably missed something critical and will now die a flaming death. Oh well
Re:But separate processes ARE better. (Score:1)
Right... (Score:1, Informative)
http://www.google.com/search?q=pthreads [google.com] shows, for starters :
Re:Right... (Score:2)
Pthreads stands for POSIX threads.
Threads are different "paths of execution" through a program that can be run in parallel. They have both advantages and disadvantages, without getting into a lot of technical details, they are generally regarded as "lighter" in terms of resource consumption and easier to code for.
The general idea is that when launching a new thread you launch a function in your program, so you can launch the music function in a game and then the graphics engine function and the keyboard reading function, etc, having them run simultaneously.
Re:Right... (Score:2)
what how why tutorial (Score:1)
For you managers out there... (Score:2)
There are a lot of managers out there who distance themselves... so be happy they are learning the technology vs a 2 minute shpeel on why they should use something.
Whats the Royal diff btw threads and processes (Score:1)
I've done some reading on these and Ive come up with:
(1) Threads are in the same process, so you would see just 1 in ps ax.
(2) Threads are new. Not many apps use it and older linuxen dont either.
(3) Everything is shared btw threads.
(4) Threads are a part of POSIX standard
I know nothing else. I couldnt find the why and how of it, and what condition will warrant the use of threads but not processes. Can someone give a clear and convincing reason to use threads over processes??(despite the overhead of learning it, and some incompatibilities across ports)
Re:Whats the Royal diff btw threads and processes (Score:1)
On Linux , as far as the kernel is concerned, a thread is a process,
threads are shown as a process in ps. This will change much with the new 2.6 kernel(and newer ps/top utils)
>Threads are new. Not many apps use it and older linuxen dont either.
There are wast number of application that uses it(though not as common as forking/nonblocking programming yet)
Most desktops. Mozilla,OpeoOffice, Java, named, etc.
>(3) Everything is shared btw threads.
Not the stack.
Re:Whats the Royal diff btw threads and processes (Score:2, Informative)
(1) Depends on what platform and how the threads are implemented. In some cases, you do see multiple entries in ps for a process running threads. Typically you see this in network deamons, but often that's because they use fork().
(2) They've been around a very long time. Dijkstra (the guy who developed the shortest path algorithm for graphs) also developed a lot of stuff for threading in the 1960s. Though, fork() and exec() have probably been around longer.
(3) Correct. Threads share memory. The trick is to make sure you eliminate race conditions so that one thread doesnt blow away what another thread is working on while it's working on it. There's two major ways of doing this: Semaphores and Monitors. pthreads use semaphores because they're a little easier to work with in C, whereas monitors generally need support from the programming language (eg, Java implements monitors, but can add your own semaphores to Java code if you wish).
(4) POSIX has a Threading specification. iirc, pthreads in one such implementation.
Uses: Within an OS Kernel, in GUIs, server apps, less overhead than using fork() et al. Useful in massively parallel computing applications where a lot of data gets shared. Like everything else in programming, it really depends on what's being developed.
This may seem harsh ... (Score:1)
"I'm sorry boss, i love you and all, but i'm the engineer here, and decisions like these are none of your business. If you're sure it would be done better without pthreads, then fine, code up a testcase, and we'll see."
Re:This may seem harsh ... (Score:2)
"I understand you're lack of support for pthreads comes from an admitted lack of understanding of them. But having used them, I can tell you the only differences are A, B, C. If you're sure it would be done better without pthreads, then ok, I'LL code up a testcase, and we'll see how readable the code is, and how well it runs."
Keep in mind, because someone has the title 'Manager' doesn't mean they don't know a damn thing (I'm a 'Manager', but I'm also the only IT person. For me, it includes everything from Network Admin, to PBX Admin, to Programmer, to Invoice authorizer). Rather, it means they can explain themselves without coming across as a total ass.
Keep that in mind. (I always have a problem explaining things in an easy-to-understand way without making my opponant look like a moron.)
Communications Needed (Score:3, Insightful)
By all accounts, if your perception is that:
- boss is technically astute
- boss hates pthreads cuz he doesn't understand `em
Then there's inherent contradictions.You need to learn how to talk to your boss more, listen more, and, after listening, patiently explain with about 2 viewgraphs of bulleted items, the key features of threads and processes, with an even-handed listing of their respective pros and cons.
Then let him make a decision. Tell him your opinion is that pthreads is a better choice for this project, but you'll go with whatever he decides. In turn, express your appreciation for him holding up whatever decision and supporting you whichever way you go.
It wouldn't hurt your case if you explained that you've programmed in pthreads before, are familiar with the pitfalls and have encountered them previously, and think they are outweighed by the advantages. Tell him that only if it's true, though:)
Re:Communications Needed (Score:2)
Most likely, the bosses concern is that it might be more difficult to hire people who understand pthreads than people who understand forking. This is a very valid concern. That's one of the reasons for the popularity of Solaris, Windows, Oracle, Motif, Java etc - there's a huge pool of people who know how to develop and maintain applications on them. For a given application, Smalltalk might be the best language on purely technical grounds and for the next project it might be LISP, but most sane organizations will do both projects in C because everyone knows C.
Re:Communications Needed (Score:1)
Wow, I sure wish I knew the same everyone that you do
Re:Communications Needed (Score:2)
Nah.
Maybe the boss really knows what he's doing. (Score:2)
"Do not oppose those with their backs to the wall."
"Leave an escape route for a surrounded army."
Maybe the boss really knows that pthreads aren't the way to solve the particular problems they have. I'm not that technically astute so I wouldn't use threads for a production app. But from the little I know, if I'm the boss I'll require someone _very_very_ good at pthreads to implement a production app.
So what if this guy doesn't seem good enough to the boss?
So he gives him a way to prove he is good enough (explain why pthreads is better to the boss), AND also a way to save face if he isn't.
Better than risking:
a) Misjudging this guy capabilities and thus create resentment and making him less useful.
OR
b) Correctly judging his current capabilities but in the process obliterating this guy's ego/confidence and thus potentially making him less useful to you.
The tradeoff is a little loss in respect at technical matters by claiming to not understand pthreads (do recall that the subordinate still does believe that the boss is very technically astute).
There might be a better way to handle such a scenario. But I'm going to sleep
Re:Maybe the boss really knows what he's doing. (Score:2)
I need to sleep
Re:Communications Needed (Score:1)
Event queues dammit! select(), poll(), or signal-driven event queues are the way to go for anyone who wants to do something Really Impressive.
For once, a manager is right... (Score:3, Insightful)
Book Recoomendations: (Score:1)
Better: "Pthreads for Managers"
Best: "Pthreads for the Mentally Retarded"
Maybe the magager is right. (Score:2)
I don't just need to understand a technology, I need a deep undertanding of its strengths and weaknesses before I can start including it in our programs in intelligent ways. In practice, it means that I experiment with new things before I'll let the team use them.
Re:Maybe the magager is right. (Score:2)
This sounds to me like a misunderstanding... (Score:3, Interesting)
I applaud your boss for his decision. There are many disadvantages of using threads, that I'm more than sure your boss knows about. See e.g. this discussion [kuro5hin.org] for more detail.
Now, granted, some applications will also benefit enough from threads to outweigh those disadvantages. It is rare, but it happens.
What I'm looking for are document technically detailed yet directed more towards management. Not something on the level of 'pthreads for Dummies', but more along the lines of 'pthreads for Managers'. Any suggestions? URLs or Books are fine."
If you have a boss that already knows enough technical stuff to micromanage your project in this way, I'm sure he is capable of googling out a pthread introduction himself. What you should be focusing on is explaining exactly why your projects needs threads, and why fork()/exec() won't cut it. This may also result in you accepting his decision, but that is as it should be.
AC summary (Score:1)
State machines are for macho programmer (Score:3, Informative)
Computer work with machine code. High level languages are for people who cannot program in machine code.
OK, Alan Cox gets my het off because DOES program in Machine code, and I have too, but the thing is, threads are easier to maintain than a state machine because usually the two things are separate while in a state machine the bloody states have a complicated interaction.
Obviously the model where you fork 10000 threads off to anything is not viable. But for many apps threads make it easier to maintain the thing and keep separate things separate. A UI which has a separate thread for the UI and a thread for the program for instance is a VASTLY better model to program in than the stupid Event based crap that you see everywhere. See Java Swing vs. Java AWT for an example. And definitely see BEOS for an example.
Thread reduce latency because a single process with a state machine has to wait in blocking IO while a thread can go on. See Netscape that hangs in all windows because a TCP connect blocks in pone window for a good and extremely irreitating example. And Mozilla does the same. For that matter, IE too but in IE you can switch it off!
In general, if things are logically different things that run on their own (The UI vs the program that does something in the background) they belong in different threads. If you do not believe me, install BeOS one day ons a 200 Mhz machine and see how much more snappy it than Linux on your multi Ghz machine because the UI things (in their own thread) repond immediately even then the background thing that they control do not. State machines are simply to difficult to handle than seprate threads but because everryon uses them all our programs have godawful latency in the User Interface.
Also, one important thing. ALL computer programs that have an active GUI have two processors in it. The computer AND the user's brain. The part that the user controls is the GUI. That belongs in another thread.
And one last thing. Thread spawning on Linux is very, very good. Mucho better than under most Unixes AFAIK and therefore the "too much overhead" is crap. On Linux at least. And most program will get along just fine with 2-10 threads.
Your examples (Score:2)
Your point that 2.5 different threaded browsers have that "one window blocks others" problem, seems to be more an example against using threads to me.
I think in most situations on Linux/FreeBSD it is best not to use threads. Use fork - default unshared memory, and then just explicitly share memory if necessary. Safer and easier to debug than "share everything".
And if one browser instance dies, it doesn't take the other 30 with it.
Should your manager be a manager? (Score:1)
Book reference (Score:1)
Re:Book reference (Score:1)
Think this through.... (Score:5, Insightful)
Laugh, but it seems to be a common attitude that a manager is never right, and never will be.
Your manager is not there to code, he's there to keep all of you developers on task. Apparently, for some reason he needs to understand your code, in order to do his job to his satisfaction.
Actually, he may not NEED to understand pthreads at all. But I believe his approach is pretty smart in this case. Why should he allow you to adopt a new method, when you cannot explain it to him?
I wouldn't let you use pthreads either, if my paycheck depended on you getting your job done.
What's the difference? (Score:1)
The C10K problem (Score:2)
may no be what you'r looking for, but gives a lot of insight into different ways of dealing with this (prosesses, thread etc) problem. from Dan Kegel The C10K problem [kegel.com]
technically astute? (Score:4, Insightful)
except the stack. That's pretty darn simple, and
probably all he needs to know.
However, he's probably right. Threads are horribly
abused in most applications. They create bloat
and spaghetti code opportunities that lead to
user dissatisfaction and unreasonable maintenance
costs.
Threads should be used if you need SMP scaling
for a I/O intensive application, or shared memory
decomposition of a problem space dominates the
structure of the code. Otherwise, small
is beautiful, and asynchronous I/O is far more
efficient and elegant (which means maintainable).
The O'Reilly pthreads book is good. (Score:1)
It introduces the classic models with code examples and good diagrams and explanations (boss/workers, consumer/producer, peer workers etc) but it doesn't go much into real program or algorithm design with multiple threads of execution however, which is of course another can of worms (but you're already dealing with that sort of problem since you've said he likes you to use fork()s).
YMMV, I have a fairly old edition (1994 or something)...
This says more about your mgr. than anything else (Score:1)
DDB (who's had to deal with entirely too many managers whose concept of "appropriate" or "proven" tech == "what I know & am comfortable with")
What you need to prove... (Score:3, Insightful)
1) cost savings short term
2) cost savings long term
Anything else is just typical geek elitism. "Hey, let's use foobar technology because it's, well, COOL!"
If you are developing on Linux the new threads implementation shall kick ass anyway, so it should become sort of moot.
some disadvantages... (Score:4, Informative)
Threads can be a lot harder to debug.
I don't think gdb really understands threads very well. Other application support is spotty as well. Actually, Solaris has pretty good thread-aware debugging and has good threading tools like deadlock tools.
People who write threaded applications also seem to think it makes life easier. They say - I'll just have a read thread and a write thread and a processing thread.
So people end up using select() anyway. And then they still need interprocess communication to kick the read thread out of the select sometimes.
I think you have to be a better programmer to use threads - you have to be aware of the normal issues plus all the threading issues. Avoid cancelling threads. You have to be very clear how you do signal handling. You have to lock all your data. Don't use fork+exec AND threads. You have to keep track of multiple threads, especially during error conditions.
And you have to be a much better debugger to debug someone else's threaded application.
Not that there aren't good applications for threads, just be aware of the complications.
Threads are *way* overrated (Score:4, Insightful)
Just a short rant on my experience and conclusions regarding threads. Me, having been formed in the "new age" school of programming (trained on the Commodore Amiga), always used threads. Used to design all my software with threads in mind. But not anymore. Since my jump to Unix, some five years ago, I've experienced a rather deep change of perception on this issue.
IMO, threaded code is much, much harder to implement correctly, and to debug, than single-threaded code. I used to blame the crappy thread support of many Unix libraries and tools. But now that gdb supports threads, along with pretty much everything else (except gprof, but the workaround is simple), I've found that it still is a PIA to debug all but the simplest threaded apps.
After years of hard work trying to determine what library calls are not reentrant, how are asynchronous signals delivered in each platform you support (God! Never, ever, ever mix threads and signals when writing portable code, not if you can avoid it), and which objects should be synchronized to get the best compromise between sync overhead and avoiding trashing the heap, you just fall in love with a design in which no locking is necessary at all. No ambiguities in library calls, no undocumented whatever_r() calls, no weird signal handling. A design in which a single core dump can tell you EXACTLY what was your program doing and why did it crash. Really, the simplicity and elegance of Unix programming without threads is just beautiful. The best adjective I can come up with is "liberating".
Of course, you do have to switch to a state-machine kind of mentality when designing your applications. It is not easy (was not easy for me, at least). But it can be done very cleanly and elegantly (for an example, look for the "State" pattern, the one in which you derive classes for representing your state machine states). If portability is not an issue, it can also be done very efficiently, if you use IO signals instead of select().
Now, having said all that, I'll now point out that I still use threads, because my code usually has to run on WIN32 too, and programming without threads there is hell (not that programming with them is much better). But I avoid threads as much as I can. I still worry about writing reentrant functions, because I find that to be a very good practice, even if no threads ae involved. But threads are used only when no other portable solution cuts it (like when checking asynchronusly for activity on non-socket descriptors, on WIN32).
A very good, brief pthreads tutorial (Score:1)
It gets right to the point and includes simple C code examples showing how to create and manipulate threads, mutex's, and condition variables. It's still a draft, but you might find it usefull.
Here is the introduction to the paper:
"This document is intended to be a short but useful tutorial on how to use POSIX threads (pthreads). In this document I do not attempt to give a full description of all pthread features. Instead I hope to give you enough information to use pthreads in a basic, yet effective way. Please refer to a text on pthreads for the more esoteric details of the standard.
"In addition to talking about the pthread interface itself, I also spend time in this document discussion some issues regarding thread programming in general. While not specific to pthreads, it is a must that you understand these issues if you are to use pthreads--or any thread library--effectively."
Your impression of your boss may be incorrect (Score:3, Insightful)
Not trying to flamebait here. But if your boss is really technically astute, he should be able to do the research, assess the pros and cons, and make a decision.
He should be able to pick up a book, google for some benchmarks, and talk to some people either face-to-face or on Usenet and then make a decision. A manager should be able to make this decision based on the facts over the weekend.
A very good manager would be able to make the decision in an afternoon. The real difference between good management and bad management is the rate at which you can make well-informed decision.