Efficient File Accessing Techniques on Hadoop Distributed File Systems

• 作者: Hongzhi Wang Siyao Cheng Wei Qu
• 来源期刊: 国际计算机前沿大会会议论文集
• 学科: 社会科学
• 关键词: HDFS HADOOP INDEX Small FILES
• 年，卷（期）: 2016,（1）
• 页码范围: 88-90
• 页数: 3页
• 分类号: C5

摘要

Hadoop framework emerged at the right moment when traditional tools were powerless in terms of handling big data. Hadoop Distributed File System (HDFS) which serves as a highly fault-tolerance distributed file system in Hadoop, can improve the throughput of data access effectively. It is very suitable for the application of handling large amounts of datasets. However, Hadoop has the disadvantage that the memory usage rate in NameNode is so high when processing large amounts of small files that it has become the limit of the whole system. In this paper, we propose an approach to optimize the performance of HDFS with small files. The basic idea is to merge small files into a large one whose size is suitable for a block. Furthermore, indexes are built to meet the requirements for fast access to all files in HDFS. Preliminary experiment results show that our approach achieves better performance.