Assuming the matrices are square, it still seems like it would be better to implement (2) as it only requires the transposition of one matrix.

What are some of the optimization techniques for sparse matrix multiplications? Is it very different from optimizations for dense matrices?

@xiyan In general, sparse matrix are treated differently from dense matrices. For instance, the data structure may be very different. It doesn't make sense to store a bunch of zeros. http://www.mathcs.emory.edu/~cheung/Courses/561/Syllabus/3-C/sparse.html

Here's a paper on optimizing sparse matrix multiplication on GPU. https://www.ideals.illinois.edu/bitstream/handle/2142/42667/spmm_tr.pdf

What is the reason for looping over SIMD_WIDTH? It seems like the SIMD instruction is serialized.