MIT creates real-time flash storage system that’s fast enough for universe simulations

If you build a new PC in 2014, chances are you’ll end up installing three types of storage inside: RAM, SSDs, and hard drives. RAM is the fastest, but wont retain its data without power, SSDs beat out hard drives on speed but is price-limited on storage size, and hard drives give you terabytes of space at very affordable prices, but they are slow.
That transition to using faster, and currently more expensive flash memory has been going on for a number of years now and is set to continue, but it goes well beyond the consumer market. Enterprise computing is also experimenting heavily with flash memory, with the latest company to embrace it being IBM who put 12TB of flash memory inside a server. The reason: the huge performance gains being realized by doing so.
Now the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) is using flash storage to revolutionize the so-called “Big Data” industry. We’re talking systems that have to cope with massive amounts of data, like the University of Washington’s universe simulation, which requires gigabytes of data be passed around a network in order to function.



Doing such work using existing systems requires thousands of hard drives all linked together and accessed over an Ethernet network. It works, but it’s relatively slow. A hard drive typically has an access time measured in milliseconds, where as flash memory cuts that to microseconds.
Even if you switch out the hard drives for flash storage, you are still limited by the Ethernet network that connects them all together. You get a performance increase for sure, but it’s still slow, and by no means real-time.
A team at CSAIL believes it has come up with a solution to this problem and in the process created a real-time data storage system. It’s called the Blue Database Machine, or BlueDM, and the key to its speed is a combination of flash storage, programmable FPGA chips, and a serial network connecting the nodes together.
What the team did was to create a node consisting of flash memory connected to an FPGA. By doing that the node not only has very fast storage, it gains the ability to do some local processing on the data stored there using the FPGA. That’s much faster than sending the data to a central computer to be processed.
Nodes can also be linked together to form large networks, and by using a high performance serial network, communication between the nodes is measured in nanoseconds. The end result is a network of storage and processing power that allows real-time access to data while remaining highly scalable.
CSAIL has been working out the kinks in BlueDM using a 4-node network built using 5-year-old hardware. A new 16-node prototype should demonstrate the true potential of the system, though, which is expected to have 32 terabytes of storage and nodes operating at 3 gigabytes per second.
The BlueDM is prototype is expected to make an appearance at FPGA 2014 symposium being held in California on February 26-28.