In Part 1 we figured out that File IO was a structural bottleneck in PDB generation and therefore ripe for optimization. Here i describe a continuation of the optimization process into reduction of code compilation cost and how i achieved some good results without manually trawling code.
At this point I began looking at some of our code and playing around with a few longer duration cpp files to try and get a “feel” for what was taking so long. The compiler isn’t very useful in this regard so at this point we’re reliant on empirical data. One particular file didn’t seem to do very much (main hook for unit testing “core” systems) but was taking 9 seconds to compile so i began decimating it, stubbed out all its functions and it still took ~8 seconds to compile. Because this was a core systems test file it was including a lot of headers for toolbox like functionality: templated containers, specialized math classes, job framework. Many of those systems do not contain much in terms of state full code, they simply provide a framework for the client to include and template functionality into their systems. There’s that word again, template, we’ll come back to that in a later post.
I see Dead People
Upon examining some of the includes for this single long build cpp file i realized that the development of the unit tests system was playing a part. When it was originally written the code to perform the actual system level testing resided in this cpp file. As the system developed and further modules added the file was split into a set of hooks into other cpp files and a framework added such that effectively the unit tests file literally only needed externs to those test blocks.
I pared them down, of the 50+ includes only around 10 were required. Compile time down to <1second…
So this innocuous file had 40+ includes it didn’t need, many of them heavy with inline functions, templates, all manner of other includes. The pre-processed file for it was huge (surprisingly so). Removing the headers was laborious and time intensive: remove header, compile all targets, if they all compile & link fine then we’re good and the header can be removed permanently. This process took around 40 minutes for this single file and we had (at the time) ~7000 source files active… this would take far too long.
But wait.. we have these machines, they do these things for us when we tell them to…
Do I know Kung-Fu?
At this point it was obvious that a plugin or macro could do this work for us. What i found however was that the help for VS Macros with VS2010 was somewhat lacking, searching the internet often gets more results in Word/Excel than any other application and very little seems to exist in terms of automation beyond trivial operations such as text insertion. The basics of what i wanted to do is simple
for each file in the project
for each include in the file
if (file compiles without include)
remove include
next
next I spent a weekend playing around with various options and hunting down a lot of the automation options available with VBA / DTE. It did turn out that there is a ton of info but it is very tough to search for it. In the end i found that the most helpful method was to RECORD macros yourself and dissect them into useful snippets for your work.
Some helpers for those who wish to do this themselves.
To build the startup project use
Function BuildStartupProject_f()
Dim sb As SolutionBuild = DTE.Solution.SolutionBuild
Dim projName As String = sb.StartupProjects(0)
Dim Config As String = sb.ActiveConfiguration.Name
Dim Platform As String = sb.ActiveConfiguration.PlatformName
DTE.ExecuteCommand("View.Output")
sb.BuildProject(Config + "|" + Platform, projName, True)
BuildStartupProject_f = sb.LastBuildInfo.Equals(0)
End Function
setup a find target
DTE.ExecuteCommand("Edit.Find")
Dim findResult As EnvDTE.vsFindResult
DTE.Find.FindWhat = "^:b*\#include"
DTE.Find.PatternSyntax = vsFindPatternSyntax.vsFindPatternSyntaxRegExpr
DTE.Find.MatchCase = False
DTE.Find.Target = vsFindTarget.vsFindTargetCurrentDocument
DTE.Find.MatchWholeWord = False
DTE.Find.Backwards = False
DTE.Find.MatchInHiddenText = False
DTE.Find.Action = vsFindAction.vsFindActionFind
CType(DTE.Find, EnvDTE80.Find2).WaitForFindToComplete = True
findResult = DTE.Find.Execute() create an undo context
DTE.UndoContext.Open("IncludeReplace")
' Do stuff here '
if you need to undo use DTE.UndoContext.SetAborted()
to iterate over files in a project use
http://msdn.microsoft.com/en-us/library/aa301042(v=vs.71).aspx
Apologies for not being able to supply a working set of macros but they fall under my employers contract so i can only provide snippets that are available online and guidelines. Feel free to ask any questions over email tho and i will answer what i can within the confines of said contract.
The greatest harm can result from the best intentions
The Brute force mechanism roughly described above does not come without its issues. I found that upon running it on some of our low level code I created a problem that can only be combated by looking through every change made. The main cause of this was cpp files where the entire system was compiled out (removed) based on a global define that was found IN the first include. This is a common pattern for inspection/debug systems and created 3-4 problems on my first run. To combat this i made the overall macro algorithm build all target executables. Without these inspection systems the final exe fails to link and our tests can continue. This does however send our iteration time down into the 20mins per file area. As an automated system however this didn’t really deter and avoids the false positive.
I know you! You’re dead! We killed you
Running this set of Macros on 1 project (our lowest level systems/hardware layer) garnered pretty good results. 30% of our headers were removed in 1 swoop using a purely brute force method. It took 9 hours in total and saved a respectable 10% off of the build of said project on PC/360.
This technique ultimately rests on the ability to cut off an entire tree of headers from the cpp, as such it is at the mercy of the programmers willingness and ability to setup headers in a form that reduces code dependencies whilst providing the functionality the client requires. Next i will be looking into include hierarchy and how we can analyze the cost of a tree with a mind towards culling off branches and reducing the implicit dependencies we create.