What's happening when P1 reads from P3's cache? Is P3 first doing a WB to memory and then P1 reading from memory or is P1 really reading directly from P3's cache? If processors can just read directly from other processors' caches, why don't we always just do that since that would mean there's only one copy of every variable?

@jchao01 My guess is that P1 does read directly from P3's cache in that P3 broadcasts the data on the bus and P1 listens for it. We don't do this every time, however, because this would take up a lot of utilization on the bus and we wouldn't get locality benefits due to the transfer time.

@jchao01. In the MSI/MESI protocols we showed in class, the short answer is yes, the flush writes data to memory and the cache requiring the data reads it from memory.

Most modern systems are based on 5-stage coherence protocols. See examples like MOESI https://en.wikipedia.org/wiki/Cache_coherency_protocols_(examples)#MOESI_protocol or MESIF https://en.wikipedia.org/wiki/Cache_coherency_protocols_(examples)#MERSI_(IBM)_/_MESIF_(Intel)_protocol.

In short, the 5th stage of these protocols identifies which of the caches containing a line should provide the line to a new cache wanting the line. Therefore, there is an on-chip cache to cache transfer instead of a load of that data from memory.

In the lecture following this we talk a bit more about the specifics of what happens when issuing a bus instruction and the procedure of performing loads and stores. I suppose my question is how do processors communicate acknowledgement of messages (how fast is this channel, what is its BW etc.)

I find Step 4 to be quite confusing. This step says processor 1 gets data from processor 3's cache. However, what actually happens is that P3 gives up the cache line, writes the data to memory, goes into the Shared state, and then P1 reads x from memory (which was most recently updated by P3).

When one processor is doing a write, do all the other processors go to invalid statement and the processor doing the write goes to modified state? Is that correct?

@jasonalouda, Yes, I believe that is correct. A processor which wishes to write issues a BusRdX, the other processors acknowledge this and move from shared to invalid states (S -> I). The first processor is now in the modified state (M) and can write, knowing it has exclusive access.