[go-nuts] dl.google.com now served by Go

chenglu

Google uses Go for many internal projects, but for confidentiality reasons it's rare that we can point to a specific example. YouTube’s open source vitess project (http://code.google.com/p/vitess/) is one high-profile success story, and now I'm happy to announce another.

The service that runs dl.google.com--the source for Chrome, Earth, Android SDK, and other large Google downloads--has been rewritten in Go. In fact, if you’ve had Chrome installed during the past few months then you have almost certainly talked to this Go program. :-)

Why rewrite in Go? It all started back in April of this year, when I was running "apt-get update" at home and noticed that downloads from dl.google.comwere stalling for no apparent reason. I asked about this on our internal Google+ and was put in touch with the right people. It turned out that the existing C++ dl.google.com service was no longer actively maintained (short of being kept alive) and that it relied on some assumptions about Google's internal architecture which were no longer true.

The C++ version had grown thorny over the years (so many callbacks!), and it still used some of Google’s oldest C++ libraries, long since deprecated. It also had a few HTTP details wrong and a fair bit of duplicated code, sometimes missing pieces in some copies which were present in others. Much work was required to fix it (or rewrite it in modern C++), so we decided instead to rewrite it in Go.

The Go version is much less code, more readable, more testable, doesn’t have blocking problems, and fixes a number of HTTP correctness issues from the old version.  It also compiles quickly. We were prepared to take a hit in CPU and/or memory usage in exchange for readable code, but it turned out that the CPU was the same and the memory usage actually dropped!

The Go version also uses interfaces to abstract out the file system, so migrating from local disk (as used in the Go version while transitioning from C++) to networked file systems (like our Colossus) was trivial. For the same reason, it was easy to introduce a memory caching layer for the most popular downloads.

All this is to say: apt-get against dl.google.com is fast again.  I’m happy.  :-)