Anamaria Stoica

My Mozilla Blog

Mozilla’s Build System

with 12 comments

Mozilla’s Build System is a very cool distributed system run by Buildbot. The system automatically rebuilds and tests the tree every time something has changed.

The Build Infrastructure currently has around 1,000 machines grouped into 3 pools, each made up of several Build Masters and many Slaves:

  • Build Pool (handles builds triggered by all changes, except those going to Try):
    • 4 Build Masters
    • ~300 Slaves
  • Try Build Pool (handles Try builds):
    • 1 Build Master
    • ~200 Slaves
  • Test Pool (handles all tests, including Try)
    • 7 Test Masters
    • ~400 Slaves

How it works

The hg poller looks for new changes in the repository every few minutes. The changes are picked up by the Build Scheduler Master, which creates Build Requests, one for each of the supported platforms. The Build Requests go into the Scheduler Database as pending. The Build Masters look for pending Build Requests and take them on only if there are free Slaves to assign them to.

Mozilla's Build System

As the builds complete, the Build Master updates their statuses in the Scheduler Database. Also, the Test Scheduler Master creates Test Build Requests for the corresponding tests.

Next, the Test Build Requests are picked up by the Test Masters and assigns them to free Slaves. When the tests are complete, the Test Master updates back their statuses in the Scheduler Database.

Each Build Master and Test Master controls its own set of Slaves.

Build Run Life Cycle

One push to mozilla-central, if successful, generates a total of 168 Build Requests (as of October 2010, but subject to change in the future), from which 10 are builds (one for each of the supported 10 platforms), 108 unittests and 50 talos tests. All these build requests make up a Build Run.

Each of the 10 platform builds comes with its own set of test requests. The tests are created only when the corresponding build completes, and only if successful. Which means that if there are failed builds, some of the tests won’t be created, and the Build Run won’t have 168 Build Requests, but less.

Build Run Life Cycle

Two very important measures in a Build Runs’s life cycle are the Wait Time and End to End Time.

The Wait Time measures how long Build Requests wait in the queue before starting, more specific, it measures the time difference between the timestamp of the change that generated that Build Request and the timestamp of when that Build Request is assigned to a free slave. (see Build Run Life Cycle diagram above)

The End to End Time measures how long it takes for a Build Run to complete. That is, the time difference between the timestamp of the change that triggered this Build Run and the timestamp of when the last of the generated Build Requests ends (in other words, when all builds and tests are completed). (see Build Run Life Cycle diagram above)

The normal End to End Time for mozilla-central is a little under 4 hours, but greatly varies upwards with the system load.

The Great Wall of Mac minis

The builds are done on a mix of VMs, 1U servers, xserves and Mac minis, and all the testing is done on Mac minis.

The Great Wall of Mac minis is made up of a little over 400 of the Mac minis’ boxes, and is located by the Release Engineers’ desks in the Mountain View office. 😀

12 Responses

Subscribe to comments with RSS.

  1. Two questions:
    1. is there a reason for keeping the minis in the boxes? Doesn’t that hurt their cooling?
    2. What’s a talos test?


    November 8, 2010 at 10:18 am

    • you were actually kidding with the first question:)
      i hope:)


      December 6, 2010 at 10:29 am

  2. Hmm, I used to “the Mozilla build system” referring to the system of makefiles and tools that is used to actually doing a single build from the source – but then, I don’t know of a better name for the buildbot-based system. I guess we’ll need to have a good idea for that. 😉

    Robert Kaiser

    November 8, 2010 at 11:43 am

  3. do you use the mac mini servers? as opposed to the desktop mac minis…


    November 8, 2010 at 12:42 pm

  4. @mich0u: 1. I guess the boxes are just showcased there, and the actual machines reside in a farm 🙂 2. Google showed me it’s a performance test created using the Talos framework.

    I agree that the “Build System” term makes people think about the actual build files in the source code (and this is actually what I was expecting to see in the article, given the title). “Build Infrastructure” seems like a more appropriate name. What do you think?


    November 8, 2010 at 2:17 pm

  5. Framehopper, yes, “Build Infrastructure” is probably a better term. 🙂

    Robert Kaiser

    November 14, 2010 at 9:55 am

  6. […] all builds and tests are completed). (see Build Run Life Cycle diagram below, also published in Mozilla’s Build System blog […]

  7. […] one comment One push to triggers off the Build System to generate a certain number of Build Requests (depending on the branch). All these build requests […]

  8. […] See Also: End to End Times Report, Mozilla’s Build System. […]

  9. […] 英文来源:Mozilla’s Build System 中文出处:开放博客,由灰狐翻译小组制作 […]

  10. Oh yes test all these Mac Minis and return it to Apple after finished haha.


    May 10, 2011 at 8:50 pm

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: