Software Builds at EA: Sourcing Code

In my previous post I discussed the general overview of EA's build framework, the one with the imaginative name Framework. Relegated to one of the last paragraphs was a mention of the package server and how it is used for enabling sharing of code across teams.

Since that post was written we've added a new mechanism to Framework to share code by means of specifying its location via a URI in the masterconfig file.

The URI sourcer has two back-ends currently implemented, one for fetching code from Perforce, and another for fetching from NuGet.

Perforce URI's encode the server, depot path, and the specific changelist or explicit head revision specifier of the software to pull down. This last condition is meant to improve build reproducability; anyone with a particular master configuration file will have the same code from which to build, local changes notwithstanding.

Because EA is a very distributed company, with studios located across several continents, and because these packages can be large (multi-gigabyte), Framework keeps an internal mapping of central Perforce servers to studio-specific proxy and edge servers, allowing users at these studios to avoid saturating slow trans-Atlantic and trans-Pacific network connections.

As Microsoft moves to distribute more components via NuGet we were looking to add support to Framework to support interoperation. By specifying a NuGet package name, version, and a URI, Framework will fetch the code from the NuGet mirrors. It then installs the package and wraps any contained DLLs in the Framework package scaffolding. This step is not strictly necessary, we could add the package to the MSVC packages.config and be done with it, but it adds a convenience layer to permit any package in the build hierarchy to express dependencies on the NuGet library as if it was a normal Framework package.

This is but the first step in enabling better distributed development workflows. Being able to pull down source code from someone else's server is handy; being able to fork a sub-component package, make a change, share it with your team and/or send a pull request back to the maintainer, while fitting in this same explicit versioning model that Framework relies on and using backing technologies that handle repositories in the tens or hundreds of gigabytes with no problem (trololol), is going to be a lot of fun. Or awesome. I'm hoping for awesome.

Software Builds at EA: The 5000' View

A couple of days ago this tweet by John Carmack popped up. In case the link ever goes away, he says, "Dealing with all the implicit state in the filesys and environment is probably going to be key to eventually making build systems not suck."

At EA we have spent a lot of time developing build systems that don't suck. They're not perfect but as we develop applications and games targeting platforms from Android to Xbox, plus everything in between, they work incredibly well for us.

Framework

The cornerstone of our build infrastructure is a version of NAnt that was forked eons ago and has undergone essentially a complete rewrite. Although the base functionality is still there, and still used, a lot of effort has been put into establishing it as a base for the higher level components.

NAnt is just one piece of a system called Framework. In addition to NAnt, Framework includes features for managing build configurations, SDKs, and software distribution. There are a few core concepts in Framework that I think have made it much easier to manage the build morass: modules, packages, dependencies, masterconfigs, build configuration management, SDK packages, and the package server.

Module

A module is Framework's representation of an artifact producing build process. At a minimum a module will have a type (static/dynamic library, executable, C# assembly, etc.), a name, and a set of input files. Modules have a number of optional components such as custom compile options, custom preprocessor definitions, custom warning suppression, sets of resource files/include files/object files/libraries, and dependencies.

Package

Packages are a collection of modules. Every package in the Framework world has a version, even if that version is "unversioned". A package contains a description of how to build its modules, its public exports (such as public include directories, build artifacts, etc.), as well as a manifest which allows the package author to include structured metadata that can be inspected by the package server, continuous build/test systems, etc.

Dependencies

Dependency handling is one of Framework's killer features. Dependencies can be made onto other modules in the same package, onto other packages, or onto other modules in other packages. For example I can specify that my module has a dependency on EASTL and the build system will ensure that I am using the correct version, build it if it is not already built, and set up the necessary include and library paths.

Masterconfig

Versioned packages lose their utility if there is no way to enforce the versions being used. The masterconfig file is the bill of materials for the application and specifies the version of each dependency, each SDK, and each configuration component.

One of NAnt's script concepts is the property. Essentially a global variable, properties are used for tweaking the state of the build system. Some are set in build files as crude variables, others like the particular configuration to build are set on the command line. A third way to set properties is to encode them in the masterconfig. This ensures that the whole development team is using the same Android SDK level, for example, as the SDK package may provide several.

Most build system state can be controlled in the masterconfig file(*), and checking the masterconfig file into source control can be a easy and effective way of ensuring the team is building with consistent settings.

(*) Provided that the build script writers do not start using environment variables on the side :-)

Build Configuration

Framework also provides global build configurations that describe how to perform a build. These build configurations take high level build goals (ie: optimized 32-bit Windows build) and use it to determine the required compiler invocation command lines.

These build configurations are used as the base for building a project and all of its dependencies. Although a particular dependency may override its build settings for warning suppression, custom optimization settings, etc., for the most part these are slight modifications made to the global configuration.

SDK Packages

SDK packages are not handled any differently by Framework than other packages, but Framework's versioning forces consistent use of SDK versions by all developers. Many SDKs are packaged in ready-to-go forms that the build system will download automatically and use to build, skipping the hassle of having to find the right SDK package and installing it. Or installing it and its numerous components.

Package Server

Where versioned packages come into their own is as a basis for sharing technology between development groups and teams. To make this sharing easier we run a server for people to post their packages on, not entirely unlike Freecode/Freshmeat. If Framework cannot find locally a package listed in the masterconfig it will try to fetch it from the package server.

Less Make, More CMake

Owing to its NAnt heritage Framework has the ability to perform builds itself but with Visual Studio support demanded by most developers it is being used more and more as just a front end for generating project files for Visual Studio or Xcode.