tinderbox

Motivation

If you get more than 10 developers together without tools, there is going to be an explosion. Tinderbox keeps this potentially explosive situation under control.

Essentially, Tinderbox is a detective tool. It allows you to see what is happening in the source tree. It shows you who checked in what (by asking Bonsai); what platforms have built successfully; what platforms are broken and exactly how they are broken (the build logs); and the state of the files that made up the build (cvsblame) so you can figure out who broke the build, so you can do the most important thing, hold them accountable for their actions.

Design/Docs

Tinderbox is comprised of a server with a bunch of clients running builds and reporting status via mail.

Server

The server recieves mail from tinderbox clients in the form of

tinderbox: tree: SeaMonkey
tinderbox: builddate: 1021331520
(...)
tinderbox: status: success
tinderbox: END
(full build log)

The server then constructs a table with time on the vertical axis and the various builds on the horizontal axis.

1.0 Server, original code

Original code is checked in at mozilla/webtools/tinderbox a bit of a mess. Many mozilla.org and Netscape-isms hard-coded here. Main mozilla.org pages still use the 1.0 code, we are in the process of installing/evaluating the 2.0 server.

2.0 Server, rewrite.

Ken Estes graciously rewrote the 1.0 server code and has started a document here. Code is checked in at mozilla/webtools/tinderbox2. Things that happened here: lots of code cleanup, cleaner interfaces, less hard-coded dependencies, protocol abstractions, things like that. Possibly most-importantly the new 2.0 code does not necessarily need bonsai. Because both servers are coexisting for a while, this new server has been named "tinderbox2" for now.

Client

The client is basically an infinite perl while loop that updates, builds, tests the build, and reports status as a mail to the server.

1.0 Client, original code

The 1.0 client sends two mails each cycle, a I'm starting a build message, and a I'm done, here's the log and build status message. Code is checked in at mozilla/tools/tinderbox. See the README and examples directories there for more info about how to set a client up. All of mozilla.org build machines currently use this client, or some form of it. Also note, the client optionally loads post-mozilla.pl as a mechanism for building a branded or embedded client instead of the SeaMonkey build.

2.0 Client.

There is a new 2.0 client, the 2.0 server can also receive build still going, partial log messages so you can get progress as a build is going instead of having to wait until the end of the client loop. 1.0 client does not currently send any partial logs like this.

Client/server protocol (log headers, described above) was made backwards compatible for the 2.0 server, so a 1.0 client can talk to either server.

Mozilla.org usage of tinderbox

Here are some Mozilla projects using Tinderbox. The complete list of mozilla tinderbox pages is at tinderbox.mozilla.org.

SeaMonkey

This is our main tinderbox page. SeaMonkey is the project name for the latest incarnation of the mozilla browser. It uses NGLayout (raptor) and the new front end code (xpfe). These tinderbox machines live both in and outside of Netscape.

SeaMonkey-Ports

This shows the status of the build on minor platforms, platforms we can't cover on the main page, etc.

The rules of development for Mozilla are as follows:

• If there are any flaming builds, you can't check in code except to fix the build.
• If you check in code, you must watch the builds until they are green on all platforms.
• You can't check in and hope it works or plan to have someone else fix it. You should be very surprised if a build breaks.
• Get help with platforms you don't have access to before checking in changes. Don't checkin if you aren't sure it will build on all platforms.
• If you break these rules too often you won't be allowed to check in anymore. We're serious.
• If you break the tree, and someone has to come after you to fix it, you owe that someone a bribe of their choice. The traditional bribe to the release team has been tequila (but absolutely no Jose Cuervo.) Cookies and donuts make adequate substitutions.

Tinderbox Server User Interface FAQ

How does this work?

When you check in code, your name will appear in the guilty column. When there are successful (green) builds in all columns in a row higher than your name, you know you are ok: you did not break the build.

The tree is broken, how do I find out what's wrong?

Clicking L in the first red box above a green box (that is, the first build to break) will show you a build log for the broken build. You can also click C in this box to see exactly what code was checked in since the previous build.

The tree is on fire, and it's not my fault, and I would like to get some work done. What do I do?

One thing you can do is check your CVS tree by date: look backward in the Tinderbox log until you see green. Note the time at which the tree built correctly. Update your tree as it was at that date (with the cvs update -D date option.) That should get you a build that works. (However, you won't be able to check in until the tree is again green.)

What do "Clbr" and "Dep" mean?

Clbr = Clobber, Dep = Depend. Mozilla.org tinderbox is hurting for horizontal space and we are just saving space here. "Clobber" builds are checked out and compiled from scratch. "Depend" builds are updated and rebuilt in the same directory, and incrementally rebuilt. (Depend builds tend to have a faster cycle time, but fail occasionally if the dependency mechanism fails.)

You build on platforms Foo, Bar, and Baz, but my favorite platform is Quux, will you build on that too?

Sure. All you have to do is donate us a machine, and we'll add it to the list! Contact build@mozilla.org for information on setting up your own external tinderbox.