Zorba the Hutt (zorbathut) wrote,
Zorba the Hutt

So I've been working on the distributed network quite a lot at work. One of our build steps is rather slow and involves compressing a lot of data, and I've been working on farming that out to the network. I got it working at one point, but ended up turning it off again due to a bottleneck I hadn't considered (thirty computers reading uncompressed bitmaps uses a *lot* of hard drive bandwidth.) After going back and solving the bottleneck (by bzipping all the bitmaps :P) I had some more work that needed to be done, plus one or two glitches that needed to be fixed. Turning it on produced *another* batch of glitches (so it went off again), but I'd fixed those and felt it was ready for another real-world test . . . which was good timing, all things considered.

boss> This is *incredibly* slow. I've been running this build for half an hour now, and it's not even a quarter done.
me> You know, if I turned the distributed network back on, I bet it'd be faster to restart the build than to leave it running. I think I've got the latest batch of bugs worked out.
boss> You think so?
me> Yeah. There we go, it's enabled. Go for it.
boss> Okay, let's give it a try.

It finished in ten minutes.

Of course, with my luck it'll be turned off again by the time I get in tomorrow, but maybe I finally got the thing working entirely :)

Well . . . until the next set of improvements . . . I think I can decrease disk bandwidth by another factor of eight. *grin*
  • Post a new comment


    default userpic

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.