Just to make sure, N is the the number of pixels in the image right?

I believe N is the dimensionality of the discrete representation of the SPD and response functions. There shouldn't be a concept of 'image' here as this is purely about color.