Since my last post demonstrated that I shouldn’t be allowed anywhere near a keyboard when I’ve “just had a good idea for something to write”, and this week I haven’t had any ideas for something to write, let alone good ones, I thought I’d just write a brief heads-up on something that I’ve personally wanted for ages, but only found out actually exists in the last year or so. (I have this horrible feeling that everyone else has known about this for decades, though)

That thing is FileSystemWatcher – a means for a C# application on Windows (and maybe Mono?) to tell when a file or folder changes on the local filesystem. This is really, really useful in tools code, because you can do these sorts of things:

  • Automatically flag files as needing building when they are modified, rather than having to scan everything for changes every time the user hits the build button.
  • Tell when an external application (eg Maya) saves a file so you can refresh your view of it (in a preview tool, for example).
  • Spot what files an “unknown” external EXE (like a compiler) is writing to so you can figure out what temporary folders/etc it uses.
  • Track things that need updating at the next SCC checkin, or syncing with a network storage system

Adding FileSystemWatcher to our build system has made a massive difference in usability – where once a full scan of the assets folder (several tens of thousands of files) could take minutes even just to check the timestamps, there now is no need to even look there after the initial boot. The system simply marks files as changed when it sees activity on them, and so pressing “build” can immediately get to work.

As the MSDN example shows, it really is remarkably simple to get working, too – although there are a few annoying gotchas.

  • You can’t get notifications about file changes that occur when you aren’t running. There doesn’t seem to be any good way to deal with this in Windows, other than brute-force scanning – the only other solution I’ve been able to find involves manually parsing the NTFS journals(!), which is both scary and seems prone to hitting all sorts of other problems.
  • You have to size the internal buffer yourself, and if it is too small you can lose events. Unfortunately as it comes out of non-pagable memory, you can’t make it too big, either. It also seems that some machine conditions (probably related to the callback notification thread being starved?) can cause buffer overruns even with a relatively large buffer. It doesn’t seem to happen too often, though, so in our case I’ve taken the approach of simply forcing a manual rescan on the next build if an overflow does occur.
  • Not a huge gotcha, but the callback itself needs to be as lightweight as possible, otherwise the buffer problems get much worse (for obvious reasons). Copying the filename data elsewhere and then doing heavy lifting in another thread is the way to go here.
  • The widest area you can watch is a drive, it seems, so you need to know what drive your files are on as a minimum. Iterating through the system drive list is also an option, but then UNC paths will escape you, sadly…
  • …on which note, behaviour over a network seems somewhat sporadic anyway, especially if non-Windows machines are involved (not that you can really blame it in that case).
  • You can’t watch for file read requests or other non-modifying operations. SysInternal’s Process Monitor (formerly FileMon) can do that, but it sadly lacks an easy route for integration with other tools.
  • Trying to read from a file that another process has just written is a somewhat fraught exercise at times – obviously the other task may still have the file exclusively locked (which is relatively easy to handle), but more awkwardly quite a few things do several writes in succession, and if your tool gets a read lock between those it can cause a crash. If anyone knows of a 100% reliable way to avoid this I would be very interested to hear it – the only one I am aware of involves using the Volume Shadow Copy service, which is really something of a sledgehammer solution for this particular problem. For now, I’ve resorted to waiting until all tasks which touch the file end before trying to read from it.

To get around a couple of these in our data pipeline (most notably the “you have to be running to see anything” issue), it seems like writing an external service which runs continuously and keeps a log of events of interest might be the way to go. In our case, we use file hashes (rather than timestamps) a lot for file change detection, so a system-wide hash database service might be an efficient encapsulation of this… if I do get around to this, I shall report back here if anyone else is interested in the results.

So, in summary – FileSystemWatcher is a tiny bit fiddly to use (but not much), but enables all sorts of cleverness in tools! Use this power for good!