What Is The Storage Tiering Filter Driver
The CloudTier Storage Tiering SDK is a Hierarchical Storage Management (HSM) file system filter driver. It is a data storage technique that automatically moves data between high-cost and low-cost storage media. The CloudTier Storage Tiering SDK provides you a simple and low-cost solution to integrate your on-premise storage to cloud seamlessly.
The cloud tiering provides a native cloud storage tier, it allows you to free up on-premise storage capacity transparently, by moving out cooler data to the cloud storage, thereby reducing capital and operational expenditures.
Understand the file system filter driver
A file system filter driver intercepts requests targeted at a file system or another file system filter driver. By intercepting the request before it reaches its intended target, the filter driver can extend or replace functionality provided by the original target of the request. File system filtering services are available through the filter manager in Windows. The Filter Manager provides a framework for developing File Systems and File System Filter Drivers without having to manage all the complexities of file I/O.
How does the storage tiering work
The storage tiering is a comprehensive technology to access the remote storage transparently. The CloudTier storage SDK integrates with the local file system. When an application accesses a stub file, the CloudTier Filter Driver will bring back the remote storage’s data back to the application or rehydrate the stub file as a normal physical file based on the recall policy.
A stub file looks and acts like a regular file, it has the same file attributes with the original physical file (file size, creation time, last write time, last access time), it also keeps the original file’s security. A stub file is a file with sparse file and reparse point attributes, your customized tag data can be embedded to the stub file. A stub file doesn’t take the storage space for the file data, it only keeps the meta data of the file.
To implement the storage tiering solution, first you need to transfer the files to the remote storage, it can be your private or public cloud server. After the files were transferred to the server, you can replace them with the stub file based on your ILM (Information Lifecycle Manager) policy, after the file was replaced with the stub file, the file data space in your premise storage was freed up.
What can you do with the storage tiering SDK
The CloudTier Storage Tiering can be widely used in telecommunications, government, oil, medical and other industries. When users and applications access the stub files in the local storage, it is completely transparent, the system will automatically restore the data back to the stub file from the remote storage server. The network attached storage is scalable, tiered storage products provide users with an infinite online data space.
- Healthcare Data Archiving:Healthcare providers are increasingly required to retain patient medical record data for multi-year periods. Storage requirements for patient data can quickly escalate when the records include high resolution images and ultrasound content.
- Business Policy Mandated Data Archiving:Companies can generate PB’s of data in the course of day to day operations and to meet legal compliance requirements. Archive Storage can work with Smart Archiving offerings from ISV’s to create a low cost, content archiving solution.
- Digital Media Content Retention:Creators can generate PB’s worth of video and picture content that is used in the development of original digital media. Archive Storage gives creators a low-cost storage repository for original source content. File-level tiering makes it easy to shift from cold to hot storage should the need arise to use that content for another project.
- Security/Public Safety data retention: As the number and sophistication of threats to personal and business safety continue to increase, so does the demand for video surveillance. Public and now private sector companies generate TB’s of surveillance footage daily in the course of protecting their citizens and assets. Archive Storage is a low cost option for storing that data.
How to use the CloudTier Storage Tiering SDK
To use the CloudTier Storage Tiering SDK is very simple, we have the C++/C# demo source code, it demonstrates how to use the SDK. The first step is generate the stub file with the API “CreateStubFileEx” as below:
/// <summary>
/// Create stub file.
/// </summary>
/// <param name="fileName">the stub file name to be created</param>
/// <param name="fileSize">if it is 0 and the file exist,it will use the current file size.</param>
/// <param name="fileAttributes">if it is 0 and the file exist, it will use the current file attributes.</param>
/// <param name="tagDataLength">the length of the reparse point tag data</param>
/// <param name="tagData">the reparse point tag data</param>
/// <param name="creationTime">set the creation time of the stub file if it is not 0</param>
/// <param name="lastWriteTime">set the last write time of the stub file if it is not 0</param>
/// <param name="lastAccessTime">set the last access time of the stub file if it is not 0</param>
/// <param name="overwriteIfExist">overwrite the existing file if it is true and the file exist. </param>
/// <param name="fileHandle">the return file handle of the stub file</param>
/// <returns>return true if the stub file was created successfully.</returns>
[DllImport("FilterAPI.dll", SetLastError = true)]
public static extern bool CreateStubFileEx(
[MarshalAs(UnmanagedType.LPWStr)]string fileName,
long fileSize,
uint fileAttributes,
uint tagDataLength,
IntPtr tagData,
long creationTime,
long lastWriteTime,
long lastAccessTime,
bool overwriteIfExist,
ref IntPtr fileHandle);After the stub file was generated, you need to handle the read request of the stub file, you just need to register the callback function for the file system filter driver. When the stub file was accessed, the callback function will be invoked, the callback function will retrieve the data from the remote server and send back to the filter driver.
The following example will generate some stub files. To handle the read request of the stub file, we need to register the callback function for the file system filter driver. When the stub file was accessed, the callback function will be invoked, the callback function will retrieve the data from the remote server and send back to the filter driver.
public static Boolean ProcessRequest(MessageSendData messageSend, ref MessageReplyData messageReply)
{
Boolean ret = false;
try
{
//here the data buffer is the reparse point tag data, in our test,
//we assume the reparse point tag data is the cache file name of the stub file.
string cacheFileName = Encoding.Unicode.GetString(messageSend.DataBuffer);
cacheFileName = cacheFileName.Substring(0, (int)messageSend.DataBufferLength / 2);
if (messageSend.MessageType == (uint)MessageType.MESSAGE_TYPE_RESTORE_FILE_TO_CACHE)
{
//for the first write request, the filter driver needs to restore the whole file first,
//here we need to download the whole cache file and return the cache file name to the filter driver,
//the filter driver will replace the stub file data with the cache file data.
//for memory mapping file open( for example open file with notepad in local computer,
//it also needs to download the whole cache file and return the cache file name to the filter driver,
//the filter driver will read the cache file data, but it won't restore the stub file.
ret = DownloadCacheFile(messageSend,cacheFileName, ref messageReply);
}
else if (messageSend.MessageType == (uint)MessageType.MESSAGE_TYPE_RESTORE_BLOCK_OR_FILE)
{
//for this request, the user is trying to read block of data, you can either return the whole cache file
//or you can just restore the block of data as the request need, you also can rehydrate the file at this point.
//if the whole cache file was restored, you better to return the cache file instead of block data.
if (GlobalConfig.RehydrateFileOnFirstRead || GlobalConfig.ReturnCacheFileName)
{
ret = DownloadCacheFile(messageSend, cacheFileName, ref messageReply);
}
else
{
ret = GetRequestedBlockData(cacheFileName, messageSend.Offset, messageSend.Length, ref messageReply);
}
}
else
{
messageReply.ReturnStatus = (uint)NTSTATUS.STATUS_UNSUCCESSFUL;
}
messageReply.MessageId = messageSend.MessageId;
messageReply.MessageType = messageSend.MessageType;
EventLevel eventLevel = EventLevel.Information;
if (messageReply.ReturnStatus != (uint)NTSTATUS.STATUS_SUCCESS)
{
eventLevel = EventLevel.Error;
}
ret = true;
}
catch (Exception ex)
{
EventManager.WriteMessage(181, "ProcessRequest", EventLevel.Error, "Process request exception:" + ex.Message);
return false;
}
return ret;
}