Why is it that the master node might be replicated. If not replicated and the node holding this data turns off, don't we lose the most important piece of data that allows us to navigate the file system?

I would think of 2 things.

Firstly, as it was mentioned in the lecture, the probability of failure at this node is very low. Secondly, it could be the case that asynchronous replication of the contents of the master node occur on many of the chunks. In that sense, the master node data is scattered across the system and if failure occurs, it could be recovered.

When I look up HDFS, it stated that there are generally 2 or 3 sets of the master node or NameNode. So while the probability of failure is fairly low with the implementation. It seems that the system still favors redundancy and recoverability.

I had the same question about the suggestion that the master node may not be replicated — in this case low probability of large failure doesn’t seem preferable to high probability of small failure… (in contrast to the case of a map-reduce job where a small failure is amplified because other workers have to wait around idle for a failed node )

I read this blog to learn more about the HDFS NameNode: https://data-flair.training/blogs/hadoop-hdfs-namenode-high-availability/. It mentions that, before Hadoop 2.0.0, the NameNode was a single point of failure in an HDFS cluster. Each cluster had a single NameNode. Hadoop 2.0 overcomes this by using two NameNodes, the active NameNode and standby NameNode.

From an abstraction POV, the way the master node handles chunks of data is similar to how Linux handles file data block locations using an inode tree? The difference seems to be that in a DFS, location indexes are needed to find the replicated files whereas in an inode the location data is for retrieving blocks of data of the same file. Can we think of the location abstraction this way?