A couple of weeks ago we at the OpenAustralia Foundation – Australia’s open data, open government and civic hacking charity – asked if anyone wanted to join three of our volunteers at Google Sydney for a hackfest. The request was deliberately casual as we just wanted have a bit of fun over a weekend sharing what we where doing and working on the open source projects that OpenAustralia runs.

The hackfest started on Saturday with about a dozen volunteers coming along and listening to quick introductory talks from Matthew and myself. We reminded everyone of the different projects that the foundation runs: OpenAustralia.org, Election Leaflets and PlanningAlerts.

Most people decided to hack on PlanningAlerts, a project which allows people to get alerts about what is being built or knocked down in their area. Using an online tool called ScraperWiki you can quickly and easily contribute new planning authorities to PlanningAlerts. During the two afternoons (true hackers aren’t morning people!) our volunteers took up the challenge to write ScraperWiki scrapers for PlanningAlerts with zeal.

Seven people wrote scrapers for nineteen planning authorities like Hobart City Council, Redfern-Waterloo Authority or Townsville City Council all around Australia, including councils in two states we previously didn’t cover – Western Australia and Tasmania.

What does this mean in practical terms? An additional 1,823,124 Australians can now find out what’s happening in their local community using PlanningAlerts. This is a huge result, coming from a relatively modest effort and a small group of people.

A massive thank you to all of the volunteers that attended the hackfest, especially the following people that wrote scrapers: Sam Cavenagh, Nick Evershed, Mark Kinkade, Matthew Landauer, James Polley, Kris Splittgerber and Adam Stiskala. Let’s do this again some time.

And don’t forget to remind your friends and family to sign up for PlanningAlerts!

It’s all a bit last minute. What’s the worst that happens? Nobody turns up and the three of us spend the weekend hacking. That’s okay too.

However, it would be much more fun if you came too!

We’ve got a couple of quick talks lined up to warm you up and give you some inspiration, then a full weekend dedicated to hacking on everything OpenAustralia. Come along and learn, share and build interesting things in and around open data, open government and civic hacking.

Want to reboot OpenAustralia.org by fixing the scraper? How about some finishing touches to Election Leaflets to monitor future Australian elections? Or do you want to learn how to help tens of thousands of people get PlanningAlerts by writing a simple scraper (anyone can do it!)? Then register here:

https://oa-hackfest.eventbrite.com/

This event will be hosted at Google Sydney next weekend the 10th and 11th of September, here’s our plan for the weekend:

Saturday

11:30 – Doors open
12:00 – Introduction and housekeeping
12:15 – The Gears That Power the Tubes of OpenAustralia – Matthew Landauer
12:30 – An Introduction to Guerrilla Data Liberation with ScraperWiki – Henare Degan
12:45 – Let the hacking begin
18:00 – We’ll order pizzas with anyone interested
22:00 – Doors close

Sunday

11:30 – Doors open / Let the hacking begin
16:00 – Stop hacking, start presentations – show us what you’ve build, fixed or learnt!
18:00 – Doors close, optional beers at the pub

Google are kindly supplying the venue, power and WiFi so all you need to do is bring along your notebook, some snacks to keep you hacking through the day and pizza money if you’re a hungry hacker come Saturday evening.

Sign up to the OpenAustralia Community mailing list if you have any questions and don’t forget to tag your tweets, photos, etc. #oahack

Cheers,

Henare, Matthew and Tim
OpenAustralia Foundation volunteers

Two weeks ago, a few of us who had worked on ElectionLeaflets.org.au during the Federal election got together to talk about what we should do next with the site.

With the upcoming Victorian election it seemed clear. Let’s make ElectionLeaflets work for the Victorian election as well.

We set today as our deadline for launching it.

Well, to cut a long story short, the disappointing thing is that we haven’t made it. We won’t be launching that site today.

There are a few reasons for this.

Firstly, remember that despite ElectionLeaflets being run by a charity, don’t assume that anyone is being paid to work on this. Quite the contrary, all the work that has gone into the site, setting it up, designing it, publicising it and talking to the media about it has all been done by unpaid volunteers.

We were hoping that adding support for multiple elections to ElectionLeaflets would be relatively straightforward, as the original developers had already thought about this from the beginning. However, it turned out to not be completely trivial. To make the required changes would need a bit of work – probably a few more full days effort.

We’re pretty good at getting things done on the smell of an oily rag and very little time but in this case with only one weekend, a few short evenings, and only two developers available, we simply ran out of time.

Despite the widespread media coverage and praise of ElectionLeaflets during the Federal election campaign, nobody came forward to say “Here’s a donation to support the running of the site and to ensure that it can be around for future elections.”

If that had happened we could have sponsored a developer to focus on this full time. It wouldn’t have taken much.

Donations for the site would also help pay for the PO Box that we rent to collect posted leaflets and help pay for the hosting of the leaflet images on Amazon’s S3 service.

Secondly, beyond the original core team of a dozen or so volunteers, we haven’t had many other people approach us offering their help to make this happen.

This is where you come in – our next goal is to cover the New South Wales state election in March 2011. The last couple of weeks has reminded us all that projects like this don’t build themselves and we need people that are as passionate as we are about keeping our elections fair and honest.

You can help shape this exciting project, and feel good knowing you’ve made a difference to electioneering in Australia. You’ll also be supported by a talented and diverse group of volunteers that are keen to share their skills and experience.

So if you want to see ElectionLeaflets cover the next election you’re called to vote in, you can start by making a donation, telling your friends about ElectionLeaflets, asking us questions on the mailing list, checking the existing trouble tickets, or even downloading our open source code and diving straight in.

If you’re not sure how to get started, please let us know in the comments how we can better help you get involved.

Henare and Matthew
OpenAustralia Foundation volunteers

PS. Victorians can still send their physical leaflets to:

Riley & Ephemera Collection,
C/O Australian History & Literature Team Access and Information Division State Library of Victoria,
328 Swanston Street,
Melbourne VIC 3000 Australia

Donors should include their name and contact details if they would like an acknowledgment of their donation and they only accept hardcopy donations (not scanned copies, or photographs).

Photo by chezmichelle (flickr.com)

As you may know PlanningAlerts.org.au works by collecting development applications from many local council websites and republishes them in an easy to use way, notifying people of new applications in their area and in that way keeping you informed of things happening near you.

To do this we need to know where those applications are. So far, we know where around 60 local authorities keep their applications. There’s more than 500 local authorities that we don’t know about.

We’d like to know about a whole lot more so that PlanningAlerts can be useful to more Australians. That’s where you come in.

We’ve created a Google spreadsheet with all the local councils that we know about which we don’t yet support on PlanningAlerts.org.au and we’d like your help to fill it in.

https://spreadsheets.google.com/ccc?key=0AmvYMal8CGUsdG1tM0lEWUctR194eGN6bUh0VGFfc1E&hl=en

This is what you do. Go to the spreadsheet.

1. Pick a council that hasn’t had all its details filled in yet (“DA’s online?” column is empty). The councils are ordered with the big councils (lots of people) at the top. So, start with those near the top if possible.
2. Follow the link to the council website (“Website” column)
3. Find where they store their development applications
4. Fill in the details in “DA’s online?”, “as HTML”, “website for DA notice” columns. “DA’s online” just says whether the council publishes the development applications online. Hopefully that will be a yes. “as HTML” should be “Yes” when they publish it as a regular web page. Fill in “No” if they publish the information in PDF or Microsoft Word documents for instance. The “website for DA notice” is just a link to the page on the council website where the development applications are stored.
5. Repeat for a few councils
6. Add a comment to this blog post to let us and everyone else know you helped out
7. Give yourself a big pat on the back.

Thank you in advance!

View Larger Map

From the instructions on the API page, we can work out how to get an RSS feed for our suburb (or any other searches if you’re feeling adventurous!), just change the [state] and [suburb] parts of this URL:

http://www.planningalerts.org.au/applications.rss?state=[state]&suburb=[suburb]

• Then go to Google Maps, paste that URL into its search box, and click Search Maps
• Click the Link link to the top right of the map, and copy the Paste HTML to embed in website code
• Paste that code into your blog post, sidebar, or wherever (you can alter the code to change its size etc.)

Thanks to Matthew at mySociety for the inspiration (and the better part of this blog post!). If you need any help, please don’t hesitate to sing out in the comments.

We’d all like to think that the website we’re making is so important and so useful that people will remember the name of it, bookmark it and come back to it regularly, but chances are that just isn’t the case.

In my last post I described the early stages of building PlanningAlerts and how the website was focused purely on getting people to sign up for email alerts and how, surprise surprise, the traffic was pretty small after launch.

Since this post is about searching you probably have some ideas where this is going, but I’ll digress a little further.

There’s a lot to be learned from past experience. Looking at the statistics of visitors to OpenAustralia.org while parliament is in session we get about 75% of our traffic through search engines, and of that about 50% is coming from normal searches on Google and 25% from “paid” searches on Google (we get advertising donated by Google through their Google Grants programme) and then a small sliver from other search engines. From my understanding these numbers are fairly typical of a large site.

Another interesting thing about the OpenAustralia.org traffic is that it varies hugely between the times that parliament is in session and when it’s in recess. A large part of this is because people don’t receive email alerts when parliament is in recess. Email alerts are the way that people are reminded that there is something interesting to look at and so it’s one of the really important ways that people keep coming back to OpenAustralia.

So, basically, search is what largely brings people to OpenAustralia for the first time, but email alerts are what keep them coming back.

So, how can we apply this experience to PlanningAlerts?

The answer is surprisingly simple. PlanningAlerts has a database of a huge number of planning applications spread across the country. At the moment we kind of hide this information and only make it available to you if you sign up for an email alert. If we put that information on our website and make it searchable and browseable (and indexable by search engines) we’ll find a whole lot more people will discover PlanningAlerts.

So, how did we go about doing this?

I’ll leave the details to another post. The changes are still in progress but some of the early gains are visible now. For instance, when you get an email alert it used to be that when you wanted to find out more about the planning application it would give you a link back to the council website. Now, instead we give a link that points back to the PlanningAlerts website. Every planning application has its own page on PlanningAlerts. Here’s an example:

We’re now also actively telling search engines about those pages using sitemaps. So, every time new planning applications are found on council websites (which are usually not indexed by search engines) they’re indexed by the search engines as quickly as possible.

We’re already now seeing that the majority of searches that lead people to find PlanningAlerts are coming from people doing searches on specific street addresses. This shows that it’s working.

image by Adam Foster (CC by-nc-nd)

Back in December of last year, we launched PlanningAlerts. It was a massive rush getting the site up and running. We had only seven weeks to build the site, including the scrapers, the bits of software that get all the planning data from the local council websites. We had applied to the Government 2.0 Taskforce for funding for the project and only received notice of go-ahead towards the end of November. Now, for some reason, which to this day still doesn’t make a whole lot of sense to me, they wanted every single project associated with the Taskforce wrapped up by the end of the year. So, that’s how the short deadline happened.

So, I did whatever any self-respecting developer does. I borrowed as much code as I could from elsewhere. In this case, the UK version of PlanningAlerts which was licensed as an open-source project under the GNU Public License (GPL).

That approach, similar to the way OpenAustralia was built, worked quite well. It gave me a clear target for the data that I needed to get from the local council websites through the scrapers.

The website itself did need some changing. In the UK, postcodes are really specific. They narrow things down to just a few houses on a street. This makes postcodes perfect as a quick way of specifying a location without having to give a full long-winded street address. So, the UK PlanningAlerts site uses postcodes to locate a place you want to receive planning alerts for.

Australian postcodes aren’t as specific as the UK ones. The areas of postcodes can be quite large and strangely shaped. So, postcodes are really not that great of a solution for locating a specific smallish area in Australia.

So, that meant I had to update things so that the user would enter a street address instead of a postcode. Luckily that turned out to be fairly straightforward. If my memory serves me correctly that took a little over a day to change.

So, we launched the site at the end of December to a short burst of interest. However, this time of year, towards Christmas and New Year must be about the worst possible time you could imagine to launch a new website. Everyone has other things on their mind: People to see, parties to go to, barbies to light, presents to buy…

No worries.

Fast forward a month to the end of January. We have a couple of hundred people signed up for email alerts but almost no visibility on the wider web. Relatively few people have mentioned PlanningAlerts on blogs and the like.

Time to go back to basics. What’s the point of building a service like PlanningAlerts if only a relatively small number of people use it? How can we get the message out?

The point of PlanningAlerts is to get emails of planning applications near you. Stuff comes to you in your inbox. After all, who really checks their local council website regularly for new and interesting development applications? Well clearly some people, but definitely not the majority.

So, the focus of the site is squarely on getting people to sign up for alerts. But how do you know what you’re signing up for? Well, in its current form, you just don’t.

Definitely time to go back to basics and rethink the whole experience. What came next? I’ll pick that in my next post…

image by Adam Foster (CC by-nc-nd)

Back in December of last year, we launched PlanningAlerts. It was a massive rush getting the site up and running. We had only seven weeks to build the site, including the scrapers, the bits of software that get all the planning data from the local council websites. We had applied to the Government 2.0 Taskforce for funding for the project and only received notice of go-ahead towards the end of November. Now, for some reason, which to this day still doesn’t make a whole lot of sense to me, they wanted every single project associated with the Taskforce wrapped up by the end of the year. So, that’s how the short deadline happened.

So, I did whatever any self-respecting developer does. I borrowed as much code as I could from elsewhere. In this case, the UK version of PlanningAlerts which was licensed as an open-source project under the GNU Public License (GPL).

That approach, similar to the way OpenAustralia was built, worked quite well. It gave me a clear target for the data that I needed to get from the local council websites through the scrapers.

The website itself did need some changing. In the UK, postcodes are really specific. They narrow things down to just a few houses on a street. This makes postcodes perfect as a quick way of specifying a location without having to give a full long-winded street address. So, the UK PlanningAlerts site uses postcodes to locate a place you want to receive planning alerts for.

Australian postcodes aren’t as specific as the UK ones. The areas of postcodes can be quite large and strangely shaped. So, postcodes are really not that great of a solution for locating a specific smallish area in Australia.

So, that meant I had to update things so that the user would enter a street address instead of a postcode. Luckily that turned out to be fairly straightforward. If my memory serves me correctly that took a little over a day to change.

So, we launched the site at the end of December to a short burst of interest. However, this time of year, towards Christmas and New Year must be about the worst possible time you could imagine to launch a new website. Everyone has other things on their mind: People to see, parties to go to, barbies to light, presents to buy…

No worries.

Fast forward a month to the end of January. We have a couple of hundred people signed up for email alerts but almost no visibility on the wider web. Relatively few people have mentioned PlanningAlerts on blogs and the like.

Time to go back to basics. What’s the point of building a service like PlanningAlerts if only a relatively small number of people use it? How can we get the message out?

The point of PlanningAlerts is to get emails of planning applications near you. Stuff comes to you in your inbox. After all, who really checks their local council website regularly for new and interesting development applications? Well clearly some people, but definitely not the majority.

So, the focus of the site is squarely on getting people to sign up for alerts. But how do you know what you’re signing up for? Well, in its current form, you just don’t.

Definitely time to go back to basics and rethink the whole experience. What came next? I’ll pick that in my next post…

A) Relax
B) Have a beer
C) Set up new OpenAustralia sites

If you’re one of our volunteers, Tim ‘mithro’ Ansell or your author, Henare Degan, you answered ‘C’ – well, OK, and a bit of ‘B’ too : )

During last weeks linux.conf.au 2010, held in Wellington, New Zealand, Tim and Henare setup a new site called OpenAustralia Labs.

Labs is a place to showcase some of our more crazy experiments, or just things that aren’t quite ready for prime-time.

We have two experiments already posted to labs with more coming soon:

Register of Interests Transcriber

The Register of Interests Transcriber is Tim’s tool for turning the scanned register we opened up about a year ago into searchable passages of text. It’s been looking for a home since Tim created it a few months ago.

This is a tool we’d really like to see emerge from the labs and appear on the live OpenAustralia.org site so if you think you could help make it more polished, please check out the code and start discussing it on our Community mailing list.

Words in Parliament

This is an interesting little experiment created by Stephen Bartlett during our Melbourne hackfest. It grabs the most recent house debate and analyses the most frequently spoken words in that debate.

If you have any experiments you’d like to try out and demonstrate on OpenAustralia Labs, please don’t hesitate to discuss it on our community mailing list or create a ticket against the new Labs component on our issue tracker.

Last weekend, OpenAustralia held its second hackfest at Google’s Sydney office. If you haven’t come across OpenAustralia.org before, it’s a volunteer run website which helps you to find out what your representatives get up to in parliament with the aim of bridging the growing democratic disconnect that many people feel.

This time we went all out with a two day hackfest – 10:30am till 8pm both days. That’s over 18 hours of hacking! As well as the venue and a fabulous team of volunteers, which included Tim Ansell, James Polley and Pamela Fox, Google provided lunch and snacks for both days. Thank you to Google for supporting this community!

It was an amazing turnout by an incredibly talented group of people. Over 90 people took part, including a surprise group from CSIRO who happened to be wandering past and dropped in to see what we were up to. We were also delighted to have 30 people take part in the first ever retroactively-named OpenAustralia installfest – installing the development virtual machine onto their laptops, and walking through the first steps showing how they can make changes to the code, test locally, and then submit to our repository on github. During the course of the hackfest, we had half a dozen or so patches pushed to us – we’re still working through the process of accepting them all! We also had 40-odd updates to bugs on our tracker.

Photos by Halans. For more fabulous photos by Halans and others see Flickr

This hackfest was timed to coincide with the Gov2.0 TaskforceMashupAustralia competition, and we encouraged people to work on mashups for the competition, OpenAustralia itself, or any other civic-minded project.

At the same time hacking on MashupAustralia on the weekend was the #melbhack session, organised by Lonely Planet and the GovHack folks.

If people weren’t in the mood for hacking, either on a mashup or OpenAustralia, we had presentations and directed hacking sessions scheduled throughout both days. Plenty for all to do.

We gave out small prizes (donated by Google) to some of our favourite mashups created over the weekend. Working demos, however rough, won out over plans, however well-formed.

Our favourite mashup, the “Suburb Matchmaker“, was created over the weekend by Raul Alberto Caceres (@totocol), Daniela Fernandez (@danira_98) and Roberto Arias Alegria (@roberto8080). It was particularly impressive that this team met over lunch on the first day and decided on the spot to work together. “Suburb Matchmaker” helps you find the suburb that’s right for you by answering some simple questions.

Second place went to the “Frog Census” by Simon Swain (@psi6030). See where and when different frogs have been spotted in South Australia.

The irreverant “Bouncing Teapots” by Dan Bethell (@fxmonkeydan) and Ray Haleblian (@rhaleblian) came third. The “Bouncing Teapots” simulated members of the House of Representatives as teapots falling where the more time they spoke in parliament the more bouncy they were.

We hope that one of these or one the many other projects that were worked on over the weekend will win the MashupAustralia competition!

To follow OpenAustralia’s progress and find out when we’ll be running the next Hackfest near you please read our blog, follow us on Twitter, Facebook or join our community mailing list on Google Groups!