What is the CloudTier Storage Tiering SDK
The CloudTier Storage Tiering SDK is a Hierarchical Storage Management (HSM) file system filter driver. It is a data storage technique that automatically moves data between high-cost and low-cost storage media. The CloudTier Storage Tiering SDK provides you a simple and low-cost solution to integrate your on-premise storage to cloud seamlessly.
The cloud tiering provides a native cloud storage tier, it allows you to free up on-premise storage capacity transparently, by moving out cooler data to the cloud storage, thereby reducing capital and operational expenditures.
CloudTier Storage Tiering Demo
The CloudTier Storage Tiering Demo is a C# demo project, it demonstrates how to use the CloudTier Storage Tiering SDK. The CloudTier SDK includes the kernel mode filter driver “cloudtier.sys”, a user mode library FilterAPI.dll which exports the APIs from the filter driver, and communication between the filter driver and the user mode application.
The C# CloudTier demo project includes project “FilterControl”, “CommObjects” and “CloudTierDemo. You must reference the project “FilterControl” which is wrapper project for FilterAPI, it controls the communication between the filter driver and the application.
When you run the “CloudTierDemo.exe”, it will generate the test source files in folder “TestSourceFolder” and the associated test stub files in folder “TestStubFolder”, when you read the test stub file “TestStubFolder\testFile.1.txt”, it will get the file data from the source file “TestSourceFolder\testFile.1.txt”.
You also can create your own test stub file based on your own custom source files if you have. You can specify your test source file folder first, then you can choose a folder to create the test stub files based on the source file as below:
There is a few settings which related to the stub file reading.
- Exclude process Ids, you can exclude the process to read the stub file. The excluded processes can’t read the stub files.
- The number of the threads to handle the stub file reading.
- The connection timeout which the filter driver waits for the response from the user mode application.
- The return data type: a. Return block data on read, this return method allows you only return the read data back to the filter driver, i.e. if the application only read 10 bytes, you just need to return a block of data includes the read data. b. Return cache file on read, this return method allows you to return the whole cache file with file data to the filter driver. c. Rehydrate file on read, this return method allows you to rehydrate the stub file after it was read.
How to use the CloudTier Storage Tiering SDK
To use the CloudTier Storage Tiering SDK is very simple. The first step is generate the stub file with the API “CreateStubFileEx” as below:
/// <summary>
/// Create stub file.
/// </summary>
/// <param name="fileName">the stub file name to be created</param>
/// <param name="fileSize">if it is 0 and the file exist,it will use the current file size.</param>
/// <param name="fileAttributes">if it is 0 and the file exist, it will use the current file attributes.</param>
/// <param name="tagDataLength">the length of the reparse point tag data</param>
/// <param name="tagData">the reparse point tag data</param>
/// <param name="creationTime">set the creation time of the stub file if it is not 0</param>
/// <param name="lastWriteTime">set the last write time of the stub file if it is not 0</param>
/// <param name="lastAccessTime">set the last access time of the stub file if it is not 0</param>
/// <param name="overwriteIfExist">overwrite the existing file if it is true and the file exist. </param>
/// <param name="fileHandle">the return file handle of the stub file</param>
/// <returns>return true if the stub file was created successfully.</returns>
[DllImport("FilterAPI.dll", SetLastError = true)]
public static extern bool CreateStubFileEx(
[MarshalAs(UnmanagedType.LPWStr)]string fileName,
long fileSize,
uint fileAttributes,
uint tagDataLength,
IntPtr tagData,
long creationTime,
long lastWriteTime,
long lastAccessTime,
bool overwriteIfExist,
ref IntPtr fileHandle);After the stub file was generated, you need to handle the read request of the stub file, you just need to register the callback function for the file system filter driver. When the stub file was accessed, the callback function will be invoked, the callback function will retrieve the data from the remote server and send back to the filter driver. In your application you need to start the filter driver service and register the callback function as below:
if(!filterControl.StartFilter(numberOfThreads,connectionTimeOut, licenseKey, ref lastError))
{
return false;
}
if (!filterControl.SendConfigSettingsToFilter(ref lastError))
{
return false;
}
filterControl.OnFilterRequest += OnFilterRequestHandler;The OnFilterRequestHandler callback function will handle the stub file read request. The parameters passing to the callback function includes the user name, process name, file name, file size, file attributes, file time, read offset and read length.
To handle the read request in the callback function, you can return the block data or the whole file data from your remote server based on the read request type.
static void OnFilterRequestHandler(object sender, FilterRequestEventArgs e)
{
Boolean ret = true;
try
{
//here the data buffer is the reparse point tag data, in our test, we added the test source file name of the stub file as the tag data.
string cacheFileName = Encoding.Unicode.GetString(e.TagData);
//remove the extra data of the file name.
cacheFileName = cacheFileName.Substring(0, e.TagDataLength / 2);
if (e.MessageType == FilterAPI.MessageType.MESSAGE_TYPE_RESTORE_FILE_TO_CACHE)
{
//for the write request, the filter driver needs to restore the whole file first,
//here we need to download the whole cache file and return the cache file name to the filter driver,
//the filter driver will replace the stub file data with the cache file data.
//for memory mapped file open( for example open file with notepad in local computer )
//it also needs to download the whole cache file and return the cache file name to the filter driver,
//the filter driver will read the cache file data, but it won't restore the stub file.
e.ReturnCacheFileName = cacheFileName;
//if you want to rehydrate the stub file, please return with REHYDRATE_FILE_VIA_CACHE_FILE
if (GlobalConfig.RehydrateFileOnFirstRead)
{
e.FilterStatus = FilterAPI.FilterStatus.REHYDRATE_FILE_VIA_CACHE_FILE;
}
else
{
e.FilterStatus = FilterAPI.FilterStatus.CACHE_FILE_WAS_RETURNED;
}
e.ReturnStatus = (uint)FilterAPI.NTSTATUS.STATUS_SUCCESS;
}
else if (e.MessageType == FilterAPI.MessageType.MESSAGE_TYPE_RESTORE_BLOCK_OR_FILE)
{
e.ReturnCacheFileName = cacheFileName;
//for this request, the user is trying to read block of data, you can either return the whole cache file
//or you can just restore the block of data as the request need, you also can rehydrate the file at this point.
//if you want to rehydrate the stub file, please return with REHYDRATE_FILE_VIA_CACHE_FILE
if (GlobalConfig.RehydrateFileOnFirstRead)
{
e.FilterStatus = FilterAPI.FilterStatus.REHYDRATE_FILE_VIA_CACHE_FILE;
}
else if (GlobalConfig.ReturnCacheFileName)
{
e.FilterStatus = FilterAPI.FilterStatus.CACHE_FILE_WAS_RETURNED;
}
else
{
//we return the block the data back to the filter driver.
FileStream fs = new FileStream(cacheFileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
fs.Position = e.ReadOffset;
int returnReadLength = fs.Read(e.ReturnBuffer, 0, (int)e.ReadLength);
e.ReturnBufferLength = (uint)returnReadLength;
e.FilterStatus = FilterAPI.FilterStatus.BLOCK_DATA_WAS_RETURNED;
fs.Close();
}
e.ReturnStatus = FilterAPI.NTSTATUS.STATUS_SUCCESS;
}
else
{
EventManager.WriteMessage(158, "ProcessRequest", EventLevel.Error, "File " + e.FileName + " messageType:" + e.MessageType + " unknow.");
e.ReturnStatus = FilterAPI.NTSTATUS.STATUS_UNSUCCESSFUL;
ret = false;
}
}
catch (Exception ex)
{
EventManager.WriteMessage(181, "ProcessRequest", EventLevel.Error, "Process request exception:" + ex.Message);
}
}