Just for a fun analogy?

mesh simplification ~ high dimensional to low dimensional encoding sub division ~ low dimensional to high dimensional decoding

How do we choose how many triangles to use? Is there a notion of diminishing returns for adding more triangles in terms of adding more detail and how do people optimize?

There definitely is diminishing returns for adding more triangles.