So in what situations are shared "single" caches used and in what situations are they not?

The first and second paragraphs of the introduction of this paper (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.114.3941&rep=rep1&type=pdf) was helpful in understanding the distinction above @hamood.

Overall, it seems like shared caches are best in situations when data is infrequently changed but heavily read. Additionally, they may be better in situations where cache storage is expensive since shared caches have higher utilization. In situations where data is frequently written and processors are doing different work (less temporal/spatial locality to exploit b/w processors) I can imagine a private cache being more useful. This would also be the case in situations where we want to minimize absolute cache hit latency per processor.

Wonder if there is any research/ data that can serve as a good indicator of when a shared cache may be useful. Also wonder if we can still have some shared caches, while also utilize other cache coherence protocols

@hamood, there are typically several (2-4) levels of cache in any computer. Some of the lower level caches (L3) are shared, while the higher level caches (L1, L2) are private for each core. As you might expect, the size and latency increase as you go lower in the cache system (L1 to 2 to 3 to main memory). To answer your question, shared caches are used all the time in most systems along with private separate caches for each core.

@hamood I think one of the main drawbacks of using a single cache is that 1) the size of the cache makes it disadvantageous to use it over memory after some point, and 2) there is a lot of synchronization that needs to be performed in the single cache case to ensure that there is memory consistency.