CopyTool Created by Compute Canada Team Being Presented at LUG2016
They’ll be the toast of America’s craft brewery capital, but they’ll be talking (peta)bytes, rather than beer.
Two Compute Canada team members are presenting their unique innovation at LUG2016 (April 5 to 7 in Portland, Oregon.) The Lustre User Group (LUG) conference is the industry’s preeminent destination for discussions and seminars on the Lustre parallel file system and other open-source file system technologies. Frédérick Lefebvre, team leader, and Simon Guilbault, system administrator, both at the Université Laval site of Calcul Quebec, Compute Canada’s regional partner, will present a talk titled “Developing an open-source object storage copytool for HSM on Lustre” the morning of April 7.
Lefebvre, who also presented at LUG 2015 in Denver, Colorado, said he was “happy to have the opportunity to present our work.”
The two will present their HSM (hierarchical storage management) adapter known as CopyTool that enables HSM with Lustre and object storage. The adapter allows automatic policy-driven transfer of data from fast parallel file systems to slower — and, more to the point, cheaper — object storage.
“The interesting part about our copy tool is that it is open-source, so it can be used by everyone in the community,” Lefebvre said. “Alternatively, you’d need a storage vendor to provide a version for their specific object storage technology. The copytool uses the industry standard S3 API to interface with most compatible object storage.”
The adapter, Lefebvre said, is a key enabler to facilitate the use of Compute Canada’s future object storage infrastructure. Compute Canada is in the process of acquiring tens of petabytes of object storage.
“Our project would allow a site to extend its Lustre-based parallel storage onto the central object storage in a cost-effective way, using open source tools,” Lefebvre said and added that another benefit will be to offer researching the ability of leveraging existing object store infrastructure when purchasing new parallel filesystems with dedicated grants from CFI. As accessible, secure data storage is a growing concern for researchers, this innovation will allow Compute Canada to help them optimize their funds and storage by enabling them to extend on its central object storage platform without forcing them to adhere to a specific vendor.