Back to Lecture Thumbnails
It's me!
awu
How expensive is it to create and maintain these lineages?
noelma
Spark's fault tolerance can be attributed to its use of RDDs? As professor Kunle put it, a lineage of RDDs let's us recreate a sequence of transformations to recover some intermediate data. Where are these RDDs stored/saved relative to nodes?
jkuro
@awu I think it depends on the implementation of the RDDs. RDDs are just an abstraction
A bar cat
I still do not completely understand what an RDD is. To check my understanding, an RDD is its own data structure that cannot be changed?
Please log in to leave a comment.
Copyright 2021 Stanford University
Lineage is sequence of transformations which convert data in the persistent memory into the required result by creating multiple RDDs on its way.