Our growth in 2008.
life at a startup
As a child, I had read about the Homebrew Computer Club, where Osborne and Apple were born - and I wanted to be part of a big movement like that ever since. When I first heard about Y Combinator I thought here was something happening right now that might be looked back on in the same way. I worked my way through the list of YC companies, narrowing them down to the ones I thought were most impressive. Justin.tv was at the top of my final list. Smart people, getting lots of attention for what they were building. I applied for a job there, and they seemed as happy to have found me as I was to have found them. I became their first technical employee in 2007.
My first day at the company was quite surreal. First thing that morning, Justin and Emmett met with me and asked me to write a scalable chat backend. All of the irc server software they had tried had failed to scale-up well enough, and they'd decided we needed to start from scratch. "We'd like it to be written using Python's Twisted library" Emmett said, "Do you know Python?" I didn't, not at all. "Well, you're a Lisp guy, you'll pick it up soon enough.”
That was about all the interaction I had with anyone that first day. And then the first week was gone, and nobody was saying much to me - everyone was intensely busy, and I felt badly about interrupting people. I figured out on my own how to write Python, how Twisted worked, and even the requirements for the chat server itself - I watched tcp dumps of the existing chat traffic to figure out exactly what the server needed to do. It was a subset of irc, but I didn't want to implement the whole irc spec - that would have taken too long, so I didn't bother to read the irc spec, I just watched the real-world traffic and used that as a spec.
At the end of my third week, I somehow had a working chat server. "Great! Let's try it in production" Emmett said. I was amazed - no code review, not even any conversation about how I'd implemented things - the level of trust was amazing, and daunting. I wasn't going to get to share the responsibility of this code with anyone else - it was all mine... and I had no idea what I was doing.
Steve Huffman was in his earn-out period at reddit at this point, and used to swing by our office for dinner. I remember the night I put my new chat server into production. It basically worked, but immediately the server's cpu was pegged at 100% and simple operations were taking seconds rather than milliseconds. Steve asked how I was doing my logging. "Just a bunch of print statements," I said. He told me to replace them with real logging calls that wouldn't take as much time. I did, and rolled out the new version. Better, but still not great. I worked really late that night, debugging the new system in production while real people were using it.
A day or two later, the chat server was running smoothly enough that I was actually pleased with my work. Over time, I grew to love the huge amount of responsibility I had at the company. Both getting to say "yes,” to things like a last-minute custom video ad integration on a Friday night to get us an extra $10k over the weekend, and saying "no" to a never-ending stream of conflicting chat feature requests.
Soon after I joined justin.tv, we were firmly in what Paul Graham calls the "trough of sorrow.” All of our early press had evaporated, it turned out nobody actually wanted to broadcast their lives, and we really had no idea what to do. The next summer though, we were out of the trough and growing like crazy. Until then, chat had run comfortably on a single server. Suddenly, that was nowhere near good enough. We were also suffering fierce attacks from spammers - I guess you know your site is popular when that happens. I have a vivid memory of working from home one weekend, moving quickly to make the chat server run transparently across several servers, and at the same time frantically adding more and more rules to keep the spammers out, all while fielding emails from annoyed users telling me again and again that chat was getting slow and spammy. By this time though, I was used to the pressure and had grown to actually enjoy it!
By the summer of 2008, chat had scaled to peaks of more than half a million concurrent users, and was stable enough that I never worried too much about it any more. I was still responsible for everything chat-related - I was the programmer, the QA, and the sys-admin for the whole system. During my interview, I had negotiated a month off the following summer for my honeymoon. When the time came around, I was so confident about my work on chat that we went to Greece with very limited internet access, even though I was the only person in the company who knew anything about the chat server other than how to restart it. The server held up perfectly while I was gone. Being responsible for everything has a way of motivating you to make sure nothing will ever go wrong!
The next couple of years I darted around the company, getting myself involved in just about every piece of technology we built. I wrote a ton of Flash and javascript, bits and pieces of our rails code, our first video ad integrations, and some of the video back-end, all the while keeping chat maintained. It was great fun to be learning so many new things all the time.
Up until the release of iOS 4 in 2010, the iPhone had been incapable of broadcasting live video properly. A few of our competitors had "live broadcasting" apps, but because they had to rely on screen-scraping and software video encoding, watching the resulting video was like viewing a slideshow. In anticipation, I taught myself Objective C, and as soon as the first iOS 4 beta came out I started playing with it and reading the documentation. The first OS beta was buggy and unfinished, and definitely not designed with live streaming in mind. I spent a few weeks sending bug reports to Apple, playing with beta after beta, and then the "Gold Master" was released. It still had a few bugs that made things hard for us, but we managed to work around them. We wanted to get an app out as soon as we could - certainly before any of the competition did. I talked it over with Justin, and asked him for a couple of our best C programmers and a designer, and started pushing everyone as hard as I felt I could. It turned out we actually had two deadlines - the company wanted to get the app out very fast, and my wife was due to give birth to our daughter. I set the end-date of the app's development a few days before our daughter was due, and kept pushing people and saying no to new features. We finished the app in time for me to be with my new family, and way ahead of the competition. In the first few days after we bought our daughter home from the hospital, I was reading about our new app on TechCrunch and sending my congratulations to the team.
These days I'm working with Flash again. We're always pushing the boundaries for video players, looking for places to improve what's become standard. I found room for improvements in long-seeks, and we now do those even better than YouTube. Flash has a problem where if a big video file is buffered and you try to seek a long distance from where you're currently playing, it can take forever - attempting to seek from the beginning to the end of a 1-hour YouTube clip causes Flash to do nothing on my macbook pro for a full 30 seconds! Our player was no better until I released a fix a couple of weeks ago. Now you can seek anywhere within multiple-hour clips and it will always be very fast. We're looking for other ways we can do better than the best with our players right now - it's a lot of fun.
Justin.tv gives people huge amounts of trust and responsibility, and the opportunity to prove themselves. We've had a few bad hires, for whom the hurdles have been too high, but I love the culture we've created over the years. We have more safety nets these days for sure - it would be irresponsible to keep working without them now that we support so much traffic. Our ops and QA teams are great at providing those safety nets, at the same time as being careful not to stifle innovation and creativity.
This might sound like a post about the good old days, but it's really not. In the ways that matter, justin.tv is the same gung-ho company it has always been. Right now, the SocialCam team is questioning every assumption we've ever made about video, and the Gaming team is gearing up to embark on a similar process. Both groups are essentially working as "startups within a startup" and the company remains a place where, if you can show that a wheel is broken, people will happily let you reinvent it.
Thanks to Emmett Shear and Tatiana Moorier for reading drafts of this.
© 2011 Bill Moorier