原作者：
来源IBM Builds Biggest Data Drive Ever - Technology Review
译者QJJ
IBM Builds Biggest Data Drive Ever
IBM开建迄今为止最大数据驱动
The system could enable detailed simulations of real-world phenomena—or store 24 billion MP3s.
该系统能够详细模拟现实世界现象--或存储240亿首MP3歌曲。
Thursday, August 25, 2011 By Tom Simonite
2011年8月25日星期四，汤姆 西蒙尼特提供
A data repository almost 10 times bigger than any made before is being built by researchers at IBM's Almaden, California, research lab. The 120 petabyte "drive"—that's 120 million gigabytes—is made up of 200,000 conventional hard disk drives working together. The giant data container is expected to store around one trillion files and should provide the space needed to allow more powerful simulations of complex systems, like those used to model weather and climate.
加利福尼亚IBM阿尔马登研究实验室的研究人员正在建造一个数据仓库，他比现有的数据仓库几乎大十倍。这个120拍字节的“驱动”--也是是12,000万千兆字节--是由一同工作的200,000个普通硬盘组成。这个巨大的数据集装箱有望存储大约1万亿个文档，允许更加强大而复杂系统模拟，比如那些用于模拟天气和气候，能够为它们提供需要的空间。
A 120 petabyte drive could hold 24 billion typical five-megabyte MP3 files or comfortably swallow 60 copies of the biggest backup of the Web, the 150 billion pages that make up the Internet Archive's WayBack Machine.
120拍字节驱动能够存放240亿首标准的五兆字节MP3文件或者轻松地吞下万维网最大备份的60份拷贝，一个备份由1500亿个网页组成的互联网档案馆WayBack Machine.
The data storage group at IBM Almaden is developing the record-breaking storage system for an unnamed client that needs a new supercomputer for detailed simulations of real-world phenomena. However, the new technologies developed to build such a large repository could enable similar systems for more conventional commercial computing, says Bruce Hillsberg, director of storage research at IBM and leader of the project.
IBM阿尔马登研究实验室的数据存储组正在为不知名的客户开发破纪录的存储系统，该客户需要为一台超级计算机用于详细模拟现实世界现象。然而，开发建造如此大的数据仓库使用的新技术也能为更多普通商业计算机建立类似系统，布鲁斯 Hillsberg说，IBM存储研究主任，这个项目的带头人。
"This 120 petabyte system is on the lunatic fringe now, but in a few years it may be that all cloud computing systems are like it," Hillsberg says. Just keeping track of the names, types, and other attributes of the files stored in the system will consume around two petabytes of its capacity.
“现在120拍字节系统属于极少数，但以后几年也许所有云计算系统与他类似，”Hillsberg说。存储在系统中光光保存姓名，类型和其他文件属性的记录就会占大约2拍字节容量。
Steve Conway, a vice president of research with the analyst firm IDC who specializes in high-performance computing (HPC), says IBM's repository is significantly bigger than previous storage systems. "A 120-petabye storage array would easily be the largest I've encountered," he says. The largest arrays available today are about 15 petabytes in size. Supercomputing problems that could benefit from more data storage include weather forecasts, seismic processing in the petroleum industry, and molecular studies of genomes or proteins, says Conway.
史蒂夫 康韦，IDC公司研究分析副总经理，专门研究高性能计算（HPC）,他说IBM的数据仓库明显比以前的存储系统大。“120拍字节存储阵列毫无疑问是我遇到过最大的，”他说。如今在用最大的存储阵列大约15拍字节。更大的数据存储有利于超级计算难题，包括天气预报，石油工业的震波图分析和基因组或蛋白质的分子研究，康韦说。
IBM's engineers developed a series of new hardware and software techniques to enable such a large hike in data-storage capacity. Finding a way to efficiently combine the thousands of hard drives that the system is built from was one challenge. As in most data centers, the drives sit in horizontal drawers stacked inside tall racks. Yet IBM's researchers had to make those significantly wider than usual to fit more disks into a smaller area. The disks must be cooled with circulating water rather than standard fans.
IBM工程师开发了一系列新的软件和硬件技术，使数据存储容量得到很大提升。难题之一是找到解决办法有效地联合系统建造中用到的上千个硬盘驱动器。与大部分数据中心相同，硬盘驱动器被堆叠安放在高大的机架的水平抽屉里。但是IBM的研究人员必须使他比普通的更宽，在更小的区域装下更多硬盘。硬盘必须用循环水冷却而不是标准风扇。
The inevitable failures that occur regularly in such a large collection of disks present another major challenge, says Hillsberg. IBM uses the standard tactic of storing multiple copies of data on different disks, but it employs new refinements that allow a supercomputer to keep working at almost full speed even when a drive breaks down.
另一个主要的难题是，如此多的硬盘不可避免地定期发生故障，Hillsberg说。IBM使用的标准策略是在不同的硬盘存放多个数据拷贝，然而他们使用新的改良方法，即使当驱动器发生故障，容许超级计算机保持近乎全速工作状态。
When a lone disk dies, the system pulls data from other drives and writes it to the disk's replacement slowly, so the supercomputer can continue working. If more failures occur among nearby drives, the rebuilding process speeds up to avoid the possibility that yet another failure occurs and wipes out some data permanently. Hillsberg says that the result is a system that should not lose any data for a million years without making any compromises on performance.
一个单独的硬盘停止时，系统会从其他驱动器取得数据并且缓慢地向替换硬盘写入数据，因此超级计算机能够继续工作。如果相邻驱动器出现更多故障，加速重建过程避免可能另一故障发生使得部分数据永久毁坏。Hillsberg说，结果是不需要对性能作任何妥协，系统一百万年内不会丢失任何数据。
The new system also benefits from a file system known as GPFS that was developed at IBM Almaden to enable supercomputers faster data access. It spreads individual files across multiple disks so that many parts of a file can be read or written at the same time. GPFS also enables a large system to keep track of its many files without laboriously scanning through every one. Last month a team from IBM used GPFS to index 10 billion files in 43 minutes, effortlessly breaking the previous record of one billion files scanned in three hours.
新系统得益于一种称为GPFS的文件系统，是IBM阿尔马登研究实验室开发的能够让超级计算机数据存取更快。他把多个独立文件摆放到多个硬盘，这样就可以同时对文件的多个部分进行读或写。GPFS也能够让一个大的系统记录很多文件而不必费力地扫描每个文件。上个月，IBM一个工作组用GPFS在43分钟内索引100亿个文件，毫不费力地打破了原先三个小时内扫描10亿文件的纪录。
Software improvements like those being developed for GPFS and disk recovery are crucial to enabling such giant data drives, says Hillsberg, because in order to be practical, they must become not only bigger, but also faster. Hard disks are not becoming faster or more reliable in proportion to the demands for more storage, so software must make up the difference.
要使这么巨大的数据驱动实现，那些正在对GPFS的开发和硬盘恢复方面的软件改进至关重要，Hillsberg说，为了实用性，所以他们必一定不能那么大，但速度要更快。要求更大的存储量相对应硬盘会变得不再更快更可靠，因此软件必须弥补其中的差别。
IDC's Conway agrees that faster access to larger data storage systems is becoming crucial to supercomputing—even though supercomputers are most often publicly compared on their processor speeds, as is the case with the global TOP500 list used to determine international bragging rights. Big drives are becoming important because simulations are getting larger and many problems are tackled using so-called iterative methods, where a simulation is run thousands of times and the results compared, says Conway. "Checkpointing," a technique in which a supercomputer saves snapshots of its work in case the job doesn't complete successfully, is also common. "These trends have produced a data explosion in the HPC community," says Conway.
IDC公司康威承认对于超级计算来说快速存取大型数据仓库越来越重要--虽然超级计算更多在计算速度方面进行比较，全球TOP500通常就是这样确定国际（bragging rights）。因为模拟量变得更大，很多问题使用称为迭代的方法解决，一项模拟要运行上千次并比较结果，康韦说。“检验点，”一项用于保存万一任务没有成功完成时的工作快照，也经常被用到。“高性能计算机群体出现了数据爆炸的趋势”。康韦说。
相关文章：
