Matthew’s back: Improving the stability and scalability of the platform


I’m very happy to be back at the Foundation for a little stint now to get my hands dirty on some coding, something I haven’t had any opportunity to do over the last year. It’s funny how you can miss these things!

As you might have noticed, I’ve been away from working on and being part of OpenAustralia Foundation for the last 12 months, with 10 of those with me working at the Digital Transformation Office (DTO). It’s been an amazing experience working at the DTO, a crazy rollercoaster ride, and I’ve learned heaps. It was an extremely difficult decision to move on but ultimately I think the right one for me., OpenAustralia Foundation’s scraping platform, used by civic technology organisations from around the world and thousands of individuals to scrape around a million web pages per week, started as an experiment.

We didn’t really know whether it would work. Would people want to use it? Would it attract a community? Would members of the community be willing and able to support the platform financially?

We knew we needed something like to help us build and maintain but we really didn’t know if it was worth making it something that a much broader group of people could use.

So, when I started working on I built it using the tech equivalent of double-sided sticky tape, brown paper and a pair of scissors. Build it in the simplest possible way to get going and prove (or disprove) the concept as quickly as possible. This is a cornerstone of lean.

Over the last year, has definitely experienced growing pains. More and more people have been joining, running more and more scrapers. This is a good problem to have! However, when things go wrong on the platform they tend to go pretty badly wrong. All at once the job queue gets backed up, the io and cpu of the server gets pegged and we all too easily run out of disk space. It takes a lot of effort to recover from those situations. It doesn’t impact users beyond their scrapers taking a long time to start or run or in the worst situations the site running slowly, but it doesn’t help people trust the platform.

All the time that we’re firefighting problems on the server, the less time we have for developing new features.

So, with that in mind I’m spending the next month or two focusing on improving the underlying stability and scalability of the platform. Here’s the list (brought together from existing issues on Github) of what I’m planning to work on.

It’s going to be a busy month or two for, so please stay tuned for updates.

If you have stability or scalability things you particularly would like to see addressed soon then get in touch.

This entry was posted in Announcement, Development, Morph. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

One Comment

  1. Adrian Smith
    Posted July 21, 2016 at 12:01 pm | Permalink

    Good job!

    Keen to help.

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>


Subscribe without commenting