How does the programmer deal with the case when the cost of synchronization is variable/unknown (i.e. deploying to many different device/CPU/memory setups)?

I think that would probably require testing utilization in real time to see how much overhead synchronization creates, and then adjusting the task size to achieve the optimal workload balance among workers