I was surprised to learn about the concept of barrier. Until now, I thought of multi-threading as giving work to some threads to execute and then just wait for them to finish (with .join()).

I guess my question is, how should we think about using a barrier. And what is its overhead, I would guess that it is not as lightweight as a simple lock.

@nassostrez I think the main difference between .join() and the barriers is that .join() waits for the threads to fully complete their task, whereas barriers allow for intra-task synchronization.

You could imagine we have some function where it would be really useful for all the threads to synchronize halfway through the function. Instead of splitting the function into two pieces f1() and f2(), creating a bunch of threads for f1() and then joining them all, and then creating a bunch of threads for f2(), we could just insert a barrier in the middle of the original function and launch threads to take care of the original function. I would guess that there's less overhead in this method of communication than having to join everything and respawn all the threads.

Maybe a more concrete example of where a barrier could be helpful is suppose I have a function with a std::atomic counter variable which each thread increments based on some condition, and then there some computation in the second half of the function which uses the value of counter, expecting counter to have been (conditionally) incremented by all the threads from the first part of the function. Then you would want a barrier, since with just the atomic counter increments (or you can think about it as just a locked region for the increment), some threads could proceed on to the second half of the function (which assumes that the counter state has been finalized) before all threads have gotten a chance to update the counter. With a barrier after the counter increment, we make sure that no threads continue onto the second half of the function until they have all done their counter increment.

Thanks @ardenma for the detailed explanation. It's super helpful!