文档介绍：Dissertation Submitted toShanghai Jiao Tong University for the Degreeof Master DESIGNAND IMPLEMENTATION OF MASSIVE FILESYSTEM WITH HIGH SCALABILITY Candidate: Longze Li Student ID: 1110369053 Supervisor: Associate Prof. Xiaoyong Li Academic Degree Applied for: Master ofEngineering Speciality: Computer Technology Affiliation: School of Information Security Engineering Date of Defence: Jan, 2014 Degree-Conferring-Institution:Shanghai Jiao Tong University 万方数据万方数据万方数据上海交通大学硕士学位论文第I页高扩展性的海量存储文件系统设计与实现摘要随着大数据时代的到来,数据规模增长迅速,数据的应用模式也越来越多样,这自然给存储系统的发展带来了新的机遇和挑战。采用纵向扩展模式来对存储系统进行扩容的传统方法,已经越来越无法满足具体应用在存储规模、并发性能与成本等方面的要求;而通过良好的分布式调度策略来集成大量商品机器的横向扩展模式却能取得奇效。碧海文件系统正是采用横向扩展模式的大规模分布式文件系统, 它能良好的作为Hadoop之类的批处理框架的底层存储堆栈,然而当它应用于通用存储环境时,却存在着扩展性、可用性和并发性的不足, 而其中扩展性不足的问题更是阻碍了碧海文件系统的发展。这些不足大都来源于元数据节点的单点架构,因而本课题尝试着使用分布式元数据节点的方式来解决上述不足。具体的工作内容为:对目录树进行子树分割,从而将元数据布局在不同的节点之上;设计了监控节点,监控节点能够仲裁元数据节点的状态,收集并分发子树信息,以及增减元数据节点集群的规模;优化了数据节点与元数据节点的通信模型,避免了重命名操作造成文件数据的移动;规范了分布式元数据节点的事务处理方式,使元数据信息可以高可靠地在不同节点之间移动。万方数据上海交通大学硕士学位论文第II页测试表明,通过以上的改进,碧海文件系统的扩展性有了较为显著的提升,并且较好的支持监控协议和分布式事务。关键字:元数据管理;子树分割;可扩展性;事务处理;分布式文件系统万方数据上海交通大学硕士学位论文第III页 DESIGNAND IMPLEMENTATION OF MASSIVE FILE SYSTEM WITH HIGH SCALABILITY ABSTRACT With the BIG DATA approaching, the scale ofdata has been exploding andmeanwhile applicationsof data takeson an air of prosperity, which naturally brings the storage system’s developmentwith new opportunities and challenges. The obsoletemethod—scale-up, can’t cope with the new applications’requirements:huge volume, high concurrency and low costs; rather,the scale-out method which aggregates commodity computers petentdistributed schedulingpolicies works well. BlueOcean file system is a scale-out massive distributed file system, which performs well as a storage stack for the batch processing system such asHadoop. However,there is a hard nut to crack when BlueOcean acts as a traditional file system. Shortage in scalability, availabilityand performance i