Anamaria Stoica

My Mozilla Blog

End to End Times Report

with 10 comments

The End to End Time measures how long it takes for a Build Run to complete. That is, the time difference between the timestamp of the change that triggered this Build Run and the timestamp of when the last of the generated Build Requests ends (in other words, when all builds and tests are completed). (see Build Run Life Cycle diagram below, also published in Mozilla’s Build System blog post)

The normal End to End Time for mozilla-central is a little under 4 hours, but greatly varies upwards with the system load.

Report Contents


As you can see from the snapshot above (snapshot of the End to End Times Report for try branch as seen on October 22, 2010, a little after 12:00 PM), the report starts with some general information, like the branch selected, number of Build Runs found in the specified timeframe (given by startime and endtime URL GET parameters) and another very important value, the Average build run duration (also called the Average End to End Time).

Build Runs Info

Right under, there’s a table which displays information on individual Build Runs (each row represents a Build Run):

1. Push’s Timestamp

Initially, the table is sorted by the ‘Least When Timestamp‘ column, which is actually the push’s change timestamp. This means that at the top should be listed the most recent pushes to the repo (colored gray if still running/pending). Note: the table is sortable by all other columns too.

2. Result: success vs. warnings vs. failure

The rows have different colors depending on the Build Run’s result (‘Results‘ column): green for success, orange for warnings, red for exception and failure and gray for no result (“-“) (if all Build Requests are currently running or pending).

3. Complete? Still Running?

The ‘Complete‘ column tells whether all Build Requests are completed or not (values: yes/no).

4. End to End Time (Duration)

A very important column is ‘Duration‘, also known as the End to End Time. The duration is computed as following:

Duration := Greatest Finish Time – Least When Timestamp

, or how long it took for all Build Requests in this Build Run to complete (or up until now, if not complete). The ‘Least When Timestamp‘ is the earliest timestamp of the Build Requests’ start times and ‘Greatest Finish Time‘, the latest timestamp of the Build Requests’ finish times.

5. Build Requests Numbers Broke Down by Status And Job Type

The number of Build Requests within a Build Run (differs per branch, for example in mozilla-central there should be 168 if everything was successful) are broke down once by status: Complete, Running, Pending, Cancelled, Interrupted and Misc, and again by job type: Builds, Unittests and Talos.

6. Rebuilds And Forcebuilds

There are also counts on how many rebuilds and forcebuilds were done.

7. Further Information, Link to Build Run Report Page

To see more about the different parameters, check out the Build Run Report. The revision links on the ‘Revision‘ column points to such reports, where you can see the exact status of individual Build Requests.

The End to End Times Report contains all the Build Runs displayed by Tinderboxpushlog, but with accurate data (which does not lie! 🙂 ). However the report was not intended as a real time monitorization tool, but rather as an analysis tool which provides a peak into how well the Build System is performing. Not so far anyways…

Average End to End Times (E2E)

Here are some E2E Averages computed per month, though E2E times tend to vary greatly from week to week or even from one Build Run to another.

Month Branch Mean Median
Aug m-c 9h 22m 4h 29m
try 10h 25m 7h 8m
Sep m-c 6h 12m 4h 8m
try 7h 6m 4h 59m
Oct m-c 6h 41m 3h 43m
try 4h 20m 3h 55m

The average is currently computed only as a simple arithmetic mean, which due to large outlier values might not the the best measurement. The median values were added to the table presented above as a comparison only, and aren’t currently calculated by the report.

As you can see from the chart, the E2E times have decreased over the past 3 months for mozilla-central and try. For try the improvement is even more visible, mostly thanks to the new Try Chooser.

Problem / E2E Report Incomplete

There is one problem that prevents the E2E Times Report from being complete, and that is the nightly builds. The Build Requests generated for the nightly builds have no revision number attached, which means there is currently no exact way of regrouping the individual Build Requests back to the Build Run. To make things more complicated, the nightly’s tests do get revision numbers, that is the revision number of the most recent commit, thus making the nightly’s tests mix up with the previous Build Run’s Build Requests! (contaminating the E2E time of that Build Run too)

To solve this issue, the following bug has been issued to Bugzilla: Bug 594496 – Generate unique id for a push in schedulerdb/statusdb.

Fun Outliers

By sorting the table by the ‘Duration’ column you can run into many surprising findings, like:

  • outrageous wait times:
    • a 500h build run, failed. Cause: 500h wait times. Revision: 19a458b7ab57

  • one can ruin for them all:
    • a 60h build run,  successful, no wait times. Cause: 1 single talos took 52h and ended successfully (all other Build Requests had normal run times). Revision: 72d2863f43c7

    • a 19h build run, exception, no wait times. Cause: 1 single talos took 17h and ended in exception (all others had normal runt times). Revision: 0e40a49c27bb

  • human rescue intervention (true, a bit late):
    • a 127h build run, no wait times. Cause: some cancelled jobs, after running for too long. Revision: 6dfa6a7c94e0

Thus, the E2E Times Report can also help detect such irregularities in due time!

See Also: Average Time per Builder Report, Build Run Report, Pushes Report, Wait Times Report


Written by Anamaria Stoica

November 15, 2010 at 6:46 am

10 Responses

Subscribe to comments with RSS.

  1. […] Also: Average Time per Builder Report, End to End Times Report, Pushes Report, Wait Times […]

  2. This is all very interesting and valuable stuff! Nice plots too!


    November 15, 2010 at 8:54 am

  3. […] my previous post End to End Times Report I started talking about E2E times, by defining what they are and then looking at some monthly E2E […]

  4. Others such as tomato paste, modified food starch, salt, yeast
    extract, caramel color, hydrolized soy protein, hydrolized yeast
    protein, beef fat, flavoring, hydrolyzed wheat gluten, dextrose are ingredients
    containing hidden MSG. I highly recommend the Crock-Pot Cook Carry Countdown Slow Cooker
    as a great slow cooker that any household would be grateful to have.
    There are no real features to speak of, as the Slow Cooker
    market is somewhat unsophisticated.

    April 16, 2013 at 11:52 am

  5. A thicker pouring will produce a more soft chewy cookie.
    Waffle Maker Cooking – Believe it or not waffle makers can make more
    than just waffles. acer ble and comes with the non stick coating on both sides.

  6. I like what you guys are usually up too. This type of clever work and exposure!
    Keep up the very good works guys I’ve added you guys to blogroll.

  7. The next thing you want to do is give your gift and or card to
    whomever is getting married. This is a recipe for a
    Belgian waffle that is truly a sweet, sumptuous and nutty concoction.

    A bonus is that it can be stored upright to save storage space.

    best waffle makers

    July 13, 2013 at 3:57 am

  8. I never realized how useful his skill was until he bought his first gas air compressor.
    Well air compressors go back a long way, and have in fact
    been used for centuries. There are also many different
    handheld mechanical tools which are used in plenty of jobs and on plenty of
    job sites regularly.


    July 18, 2013 at 9:00 am

  9. When the cooking of meat is done, the meat is usually not cooked evenly on all sides.

    Their grilling units are all quality, upmarket, top of the line,
    well constructed and designed. This is why you have to look
    for the enamel-coated ones.

  10. The company who started this campaign eventually became the
    world’s largest soap company. The different culinary preparations are usually characterized by lots of vegetables, pulses and spices. There is the Neapolitan pizza usually called the pizza Napolitano.


    August 8, 2013 at 7:47 am

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: