December 2016

Volume 31 Number 13

[Universal Windows Platform]

File System Monitoring in Universal Windows Platform Apps

By Adam Wilson | December 2016

The file system of a Windows device changes rapidly. Shared libraries, such as the camera roll, are one of the few places where all processes on the device can simultaneously interact with the same data. Building a Universal Windows Platform (UWP) app that provides a great experience with a user’s photos means you’re going to have to wade deep into this chaos.

In the Window 10 Anniversary Update, Microsoft added new features to make managing this chaos easier. The system is now able to provide a list of all the changes that are happening in a library, from a picture being taken all the way up to entire folders being deleted. This is a huge help if you’re looking to build a cloud backup provider, track files being moved off the device or even just display the most recent photos.

Showing the most recent photos taken by a device isn’t just for developers trying to build the next Instagram. Recently I had the pleasure of working with enterprise partners as they built inspection apps for their organizations. The apps follow a similar pattern: An inspector visits a site with a Windows tablet or phone, fills in a report with some information about the site, takes pictures of the site using the device and, finally, uploads the report to a secure server. For all of the companies, it’s critical that the correct, unmodified photos are uploaded with the reports.

Over the next few pages I’m going to walk through building an enterprise inspection app. Along the way I’ll point out some particularly challenging issues I encountered during the development process and note how you can avoid them in your app. These lessons can also apply to any other app that’s looking to track changes in the file system, such as a cloud backup service, but because inspection apps are quite common, I’ll start there and you can modify the code for whatever type of application you’re building.

Inspections: Everybody Does Them

Across enterprises of all sizes and industries, one thing is common: There are inspectors overseeing critical business processes. From massive manufacturing companies tracking the state of their equipment to retailers ensuring that displays are assembled correctly, businesses depend on making sure things are done consistently and safely across all of their sites.

The basic process I’m going to support here is quite simple:

1. An inspector creates a new report instance for the site on his tablet.
2. The Inspector takes pictures of the site for a report.
3. The pictures are uploaded to a secure server along with the report.

In step 2, however, a number of things can go wrong:

• The inspector could pick the wrong pictures to attach to the report.
• A picture could be modified to show incorrect information.
• A picture could be accidently deleted before the report is uploaded but after the inspector leaves a location.

In any of these cases, the report would be invalid and require the inspector to repeat the inspection, an added expense for the business. Luckily, with the new change-tracking APIs in Windows 10 Anniversary Update, there’s a simple way to prevent such mistakes and help users complete their tasks quickly and accurately.

The Basics: Getting Access to Pictures from the Camera

The first step is making sure the application has access to the pictures coming from the camera. In Windows, the system camera will write to the Camera Roll folder, which is a subfolder of the Pictures library. I could (and in fact did) write an entire article about how to access the Pictures library (bit.ly/2dCdj4O), but here are the basics:

• Declare your intent to access the Pictures library in your manifest (bit.ly/2dqoRJX) by adding Capability Name=“music­Library”/> in the manifest editor or checking the Pictures Library box under the Capabilities tab in the wizard.
• Get a StorageFolder representing the location where pictures from the camera are being written to using KnownFolders.CameraRoll (bit.ly/2dEiVMS).
• Get an object representing the entire pictures library using StorageLibrary.GetLibraryAsync(KnownLibraryId.Pictures) (bit.ly/2dmZ85X).

Don’t worry if the device has an SD card and a user setting for writing new pictures to either internal storage or the SD card. The KnownFolder class will abstract all that away for you and give you a virtual folder encompassing all the locations to which the camera can write a file.

Getting Notified When Something Changes

Once you have access to the files, it’s time to start tracking the changes. I recommend doing this as early as possible. In a perfect world, inspectors would always create a new report before they start taking pictures, but in reality they’ve generally taken a few pictures before they remember to create a new report.

Setting up the notifications and change tracking involves three steps:

1. Initializing the change tracker, which tells the system what libraries you’re interested in tracking. The tracking will continue even when your app isn’t running, and the app can read the list of changes at any time.
2. Register for change notifications in the background, which will activate your app’s background task whenever there’s a change in a library, whether or not it’s currently running.
3. Register for change notifications in the foreground. If your app is in the foreground, you can register for additional events whenever a file is changed under a certain scope.

Note that steps 2 and 3 could potentially overlap. I’ll cover how to set up both types of notifications in this article, but the chart in Figure 1 can help you choose which you’d like to use in your app. The general recommendation is to always use background change notifications with StorageLibraryContentChangeTrigger, and to use foreground events if you have a specific UI need, such as displaying a view of the file system to your users.

Figure 1 Types of Change Notifications

The StorageLibraryChangeTracker is a new class added in the Anniversary Update (bit.ly/2dMFlfu). It allows apps to subscribe to a list of changes that are happening in a library. The system watches all the files in the library and builds up a list of the changes that happen to them. Your app can request the list of changes and process them at its leisure.

StorageLibraryChangeTracker is very similar to the NTFS change journal, if you’ve ever worked with that, but it works on FAT-formatted drives, as well. For more information you can read through the in-depth discussion on my blog (bit.ly/2dQ6MEK).

Initializing the StorageLibraryChangeTracker is quite simple—you just get the library-specific instance of the change tracker and call Enable:

StorageLibrary picsLib =
  await StorageLibrary.GetLibraryAsync(KnownLibraryId.Pictures);
picsLib.ChangeTracker.Enable();

From this point, the change tracker will maintain a list of all the changes to the library. Now let’s make sure your app gets notified with each change.

Listening to Foreground Notifications

To listen to foreground notifications your app has to create, execute, and hold open a query over the location whose changes concern you. Creating and executing the query tells the system which locations are interesting to your app. Holding a reference to the results of that query afterwards indicates that your app wants to be notified when something changes:

StorageFolder photos = KnownFolders.CameraRoll;
// Create a query containing all the files your app will be tracking
QueryOptions option = new QueryOptions(CommonFileQuery.DefaultQuery,
  supportedExtentions);
option.FolderDepth = FolderDepth.Shallow;
// This is important because you are going to use indexer for notifications
option.IndexerOption = IndexerOption.UseIndexerWhenAvailable;
StorageFileQueryResult resultSet =
  photos.CreateFileQueryWithOptions(option);
// Indicate to the system the app is ready to change track
await resultSet.GetFilesAsync(0, 1);
// Attach an event handler for when something changes on the system
resultSet.ContentsChanged += resultSet_ContentsChanged;

As you can see, I’m using a couple of interesting optimizations that might be helpful in your app:

• CommonFileQuery.DefaultQuery makes the entire operation much faster in cases where the indexer isn’t available. If another sort order is used and the indexer isn’t available, the system must walk the entire query space before it returns the first result.
• The query is a shallow query. This works because the camera will always write to the root of the camera roll, and avoiding a deep query minimizes the number of files the system has to track for changes.
• Using the indexer isn’t mandatory, but it speeds up notifications appearing in the app. Without the indexer, notifications can take up to 30 seconds to reach your app.
• Querying for one file is the fastest way to start tracking changes in an indexed location, though you may want to query for more files in case the UI needs them.

Now anytime an item in the query changes, the event handler will be triggered, giving your app a chance to process the change.

Registering for the Background Change Trigger

Not all changes are going to happen while your app is in the foreground, and even if your app is in the foreground it may not need the granularity of foreground notifications. StorageLibraryContentsChangedTrigger is a great way to be notified when anything changes in a library. Because you register a background task using the standard process (bit.ly/2dqKt9i), I’ll go through it quickly (see Figure 2).

Figure 2 Registering a Background Task

// Check if your app has access to the background
var requestStatus = await BackgroundExecutionManager.RequestAccessAsync();
if (!(requestStatus ==
  BackgroundAccessStatus.AllowedMayUseActiveRealTimeConnectivity ||
  requestStatus == BackgroundAccessStatus.AllowedSubjectToSystemPolicy ||
  requestStatus ==
    BackgroundAccessStatus.AllowedWithAlwaysOnRealTimeConnectivity ||
  requestStatus == BackgroundAccessStatus.AlwaysAllowed))
{
  log("Failed to get access to the background");
  return;
}
// Build up the trigger to fire when something changes in the pictures library
var builder = new BackgroundTaskBuilder();
builder.Name = "Photo Change Trigger";
StorageLibrary picturesLib =
  await StorageLibrary.GetLibraryAsync(KnownLibraryId.Pictures);
var picturesTrigger = StorageLibraryContentChangedTrigger.Create(picturesLib);
// We are registering to be activated in OnBackgroundActivated instead of
// BackgroundTask.Run; either works, but I prefer the single-process model
builder.SetTrigger(picturesTrigger);
BackgroundTaskRegistration task = builder.Register();

There are a couple of important things to note in the sample in Figure 2.

You can still use the old two-process model to register your background task, but take a good look at the single-process model that was added in the Anniversary Update. It quickly won me over with how simple it is to use and how easy it is to receive background triggers while your app is in the foreground.

StorageLibraryContentChangedTrigger will fire for any changes in the Pictures library, which may include files that aren’t interesting to your app. I’ll cover how to filter those out in a later section, but it’s always important to be aware that sometimes there will be nothing to do when your app is activated.

It’s worth checking your background task whether you’re running in the background or foreground because the resource allocations are different. You’ll find more detail on the resource allocation model at bit.ly/2cNvcSr. 

Reading the Changes

Now your app is going to be given a chance to run code every time something changes in the camera roll, either in the foreground or the background. To figure out what has changed, you need to read the set of changes from StorageLibraryChangeTracker. The first step is to get a reader object, which will allow you to enumerate the changes that have happened since the last time your app checked. While you’re at it, you can also grab the first batch of changes to process:

StorageLibrary picturesLib =
  await StorageLibrary.GetLibraryAsync(KnownLibraryId.Pictures);           
StorageLibraryChangeTracker picturesTracker= picturesLib.ChangeTracker;
picturesTracker.Enable();
StorageLibraryChangeReader changeReader = picturesTracker.GetChangeReader();
IReadOnlyList<StorageLibraryChange> changes = await changeReader.ReadBatchAsync();

Once you get a set of changes, it’s time to process them. In this app, I’m going to focus only on pictures being changed and ignore all other file and folder changes. If you’re interested in other types of changes on the system, my deep dive on the change tracker has the details about all the different types of changes (bit.ly/2dQ6MEK).

I’m going to walk through the changes and pull out the ones in which I’m interested. For this application, it’s going to be any change to the contents of a .jpg file, as shown in Figure 3.

Figure 3 Checking for Changes to Files

foreach (StorageLibraryChange change in changes)
{
  if (change.ChangeType == StorageLibraryChangeType.ChangeTrackingLost)
  {
    // Change tracker is in an invalid state and must be reset
    // This should be a very rare case, but must be handled
    picturesLib.ChangeTracker.Reset();
    return;
  }
  if (change.IsOfType(StorageItemTypes.File))
  {
    await ProcessFileChange(change);
  }
  else if (change.IsOfType(StorageItemTypes.Folder))
  {
    // No-op; not interested in folders
  }
  else
  {
    if (change.ChangeType == StorageLibraryChangeType.Deleted)
    {
      UnknownItemRemoved(change.Path);
    }
  }
}
// Mark that all the changes have been seen and for the change tracker
// to never return these changes again
await changeReader.AcceptChangesAsync();

Here are a few interesting things about the code snippet in Figure 3.

The first thing I do is check for StorageLibraryChangeType.Change­TrackingLost. This should happen only after a large file system operation (where the system doesn’t have enough storage for the entire set of changes) or in the case of a critical internal failure. In either case, anything being read from the change tracker can no longer be trusted. The app must reset the change tracker for future results to be trustworthy.

UnknownItemRemoved(change.Path) will be hit any time a file or folder is deleted from a FAT partition, such as an SD card. Once an item is deleted from a FAT partition the system can’t tell whether it was a directory or a file that was deleted. I’m going to walk through some code in a bit that shows how you can figure out what happened in your app.

When all the changes have been processed, you call changeReader.AcceptChangesAsync. This tells the change tracker that your app has handled all the changes and doesn’t need to see them again. Next time you create a StorageLibraryChangeReader it will contain only changes that have happened since this point. Not calling AcceptChangesAsync will cause the internal change buffer to fill up and overflow, resulting in StorageLibraryChangeType.ChangeTrackingLost.

Handling a Change

Now that you know how to walk through the changes, the next step is to flesh out the ProcessFileChange method. One key concept to remember is that opening files (by creating StorageFile objects) is extremely expensive. To get a StorageFile object in your process, the system must make a cross-process call to check permissions, create a handle to the file, read some metadata about the file from the disk, and then marshal the handle and metadata back to your app’s process. Thus, you want to minimize the number of StorageFiles you create, especially if you don’t intend to open the file streams.

The change tracker provides your app with paths and the type of change to help you determine whether a file is of the type your app is interested in before you create the StorageFile. Let’s start by doing as much filtering as possible with this information before creating the StorageFile, as shown in Figure 4.

Figure 4 Processing Changes

private async Task ProcessFileChange(StorageLibraryChange change)
{
  // Temp variable used for instantiating StorageFiles for sorting if needed later
  StorageFile newFile = null;
  switch (change.ChangeType)
  {
    // New File in the Library
    case StorageLibraryChangeType.Created:
    case StorageLibraryChangeType.MovedIntoLibrary:
    case StorageLibraryChangeType.MovedOrRenamed:
      if (change.Path.EndsWith(".jpg", StringComparison.OrdinalIgnoreCase))
      {
        StorageFile image = (StorageFile)(await change.GetStorageItemAsync());
        AddImageToReport(image);                                               
      }                   
      break;
    // File Removed From Library
    case StorageLibraryChangeType.Deleted:
    case StorageLibraryChangeType.MovedOutOfLibrary:
      if (change.Path.EndsWith(".jpg", StringComparison.OrdinalIgnoreCase))
      {
        var args = new FileDeletedWarningEventArgs();
        args.Path = change.Path;
        FileDeletedWarningEvent(this, args);
      }
      break;
    // Modified Contents
    case StorageLibraryChangeType.ContentsChanged:
      if (change.Path.EndsWith(".jpg", StringComparison.OrdinalIgnoreCase))
      {
        newFile = (StorageFile)(await change.GetStorageItemAsync());
        var imageProps = await newFile.Properties.GetImagePropertiesAsync();
        DateTimeOffset dateTaken = imageProps.DateTaken;
        DateTimeOffset dateModified = newFile.DateCreated;
        if (DateTimeOffset.Compare(dateTaken.AddSeconds(70), dateModified) > 0)
        {
          // File was modified by the user
          log("File path: " + newFile.Path + " was modified after being taken");
        }
      }                  
      break;
    // Ignored Cases
    case StorageLibraryChangeType.EncryptionChanged:
    case StorageLibraryChangeType.ContentsReplaced:
    case StorageLibraryChangeType.IndexingStatusChanged:
    default:
      // These are safe to ignore in this application
      break;                   
  }
}

Let’s break down what’s happening here by looking at each of the case statements.

New File in the Library: In the first statement I’m looking for new files that have been added to the library. In the case of the system camera, the file is going to be created in the library and I’ll see a change of type StorageLibraryChangeType.Created. Some third-party apps actually create the image in their app data container and then move it to the library, so I’m going to treat StorageLibraryChangeType.MovedIntoLibrary as a creation, too.

I’m going to process StorageLibraryChangeType.MovedOrRenamed as a new file. This is to work around an oddity of the built-in camera on Windows phone. When the camera takes a picture, it writes out a temporary file with the ending .jpg.~tmp. Later it will finalize the picture, removing the .~tmp extension. If your app is quick enough to catch the file before the camera is done finalizing the image, it may see a rename event instead of the creation event.

Because this app is going to be interested only in pictures taken by the user, I’ll filter down to only files with the .jpg extension. You could do this by creating a StorageFile and checking its ContentType property, but I’m trying to avoid creating unneeded StorageFiles. Once I know a file is one I’m interested in, I’m going to hand the file off to another method to do some data processing.

I’m using a cast here instead of the as keyword because I’ve already asked StorageLibraryChange what type the StorageItem will be using: StorageLibraryChange.IsOfType(StorageItemTypes.File).

File Removed from Library: In the second case, the app is looking at situations where a file has been removed from the library. You’ll note that I’ve grouped two different change types together again. StorageLibraryChangeType.Deleted will be raised whenever a file is deleted permanently from disk, which is the classic case of deletion using the file system APIs. However, if the user manually deletes the file from File Explorer instead, the file will be sent to the Recycle Bin. This shows as StorageLibraryChangeType.MovedOutOfLibrary because the file is still resident on disk.

In this case, I’m going to raise a warning to the inspector that the file is gone, in case it was an accidental deletion. In more security-­conscious applications, it might make sense to store file deletions or modifications for later auditing in case of an investigation.

Modified Contents:The contents of a file being modified is an interesting case for this app. Because this app might be used for safety inspections, you don’t want to allow users to change the images before they’re uploaded to the report, though there may be valid reasons for the contents of an image to be changed after a picture is taken.

You might see a notification of the type StorageLibraryChangeType.ContentsChanged being raised about three to 60 seconds after a picture is taken with the camera. This is because it can sometimes take up to a minute to get the coordinates from the GPS system. In this app I’m not concerned about GPS information so I’m processing the file right away. In some apps it might make sense to check if the location data has been written to the file, and if it hasn’t, to wait until the location is determined.

In this case, I’m going to go for a very safe middle ground. If a file is modified more than 70 seconds after it was taken, I’m going to assume it was modified by the user and log out a compliance issue for later investigation. If it was modified within the 70-second window, I’ll assume this was done by the system when adding the GPS data and can be safely ignored.

Encryption Changed: Whether you should care about this case comes down to one question: Does your enterprise use Windows Information Protection (WIP) with encryption to protect its sensitive information? If so, then this type of change is a huge concern. It means that someone has changed the protection on the file, and the file could be sitting on the disk unprotected. Walking through all of the WIP information about how to check if the file is secure (versus just being moved to another protection level) is beyond the scope of this article. If you’re deploying WIP in your enterprise, this type of change is important for you to monitor.

For those not using WIP or encryption to protect sensitive corporate assets, I highly recommend you look into it, but for now this change is safe to ignore.

Ignored Cases: Finally, in this application it makes sense to ignore some of the things that could be happening on the file system. StorageLibraryChangeType.IndexingStatusChanged only matters in the case of desktop apps that are going to be doing repeated queries of the library and expecting the same results each time. Storage­LibraryChangeType.ContentsReplaced indicates that a hard link for the file has been changed, which isn’t interesting for this application.

Detecting Deletion of an Image on an SD Card

As you’ll recall, the system can’t determine if a removed item is a file or folder after it’s been deleted on FAT drives such as an SD card. Thus, some special sorting code is needed if you care about deletions on FAT drives. For my app, I only want to know if pictures are being deleted from the SD card, which makes the code really simple:

private void UnknownItemRemoved(StorageLibraryChange change)
{
  if (change.Path.EndsWith(".jpg", StringComparison.OrdinalIgnoreCase))
  {
    var args = new FileDeletedWarningEventArgs();
    args.Path = change.Path;
    FileDeletedWarningEvent(this, args);
  }
}

The code simply checks to see if the item used to have a .jpg extension, and if so raises the same UI event as in Figure 4.

Final Thoughts

Once the basic code in place, there are a couple of things worth considering that can help make sure your new app will run as smoothly as possible.

Experiment with your configuration! This is the most important piece of advice I can give you about creating an app that tracks file system changes. Write a quick app that logs what’s happening on your devices before you get too deep into coding. A lot of components try to get away with writing out temp files and trying to quickly delete them, such as the built-in camera I discussed earlier. I’ve even seen cases in the wild where apps were writing out incomplete files, closing the handle, then immediately reopening the handle to finish writing out the file to disk. With a change-tracking app, you’re not only going to start seeing all of these changes, but potentially get in the middle of these operations. It’s important to be a good citizen of the system, which means understanding and respecting what other apps are trying to do.

Keep in mind that because SD cards can be removed from the device while it’s powered off, and there’s no journaling on FAT-based file systems, change tracking can’t be guaranteed on an SD card across boot sessions. If your enterprise has very strict requirements about ensuring that files are not tampered with, consider using a mobile device management policy to force encrypted camera images to be written to internal storage only. This will ensure that the data is protected at rest and that all changes to the file are accounted for.

Wrapping Up

And that’s it. The inspection app is now ready to automatically add files when they’re created on the system, and to notify users when one is deleted. Although it may seem simple, enabling this experience allows users to focus on their inspections instead of picking thumbnails out of the system-picker experience. If the inspector is constantly taking pictures of similar-looking equipment, automatically including recent images in the report can greatly cut down on mistakes that cost businesses time and money. Most important, it can help your app stand out among the crowded field of enterprise apps for Windows.

Adam Wilson is a program manager on the Windows Developer Ecosystem and Platform team, working on the Windows indexer and push notifications. You can reach him at Adam.D.Wilson@microsoft.com.

Thanks to the following Microsoft technical experts who reviewed this article: Brendan Flynn, Mary Anne Noskowski and Kyle Yuan

Discuss this article in the MSDN Magazine forum