‘It starts out very experimentally because you’re not exactly sure how it will work’. An honest and simple reflection about how projects using data science commence, and go on to deliver great benefits and services for the public.
Meet Dr Neal Hughes, Senior Economist with the Australian Bureau of Agricultural Resource Economics and Science (ABARES) and lead of the Agricultural Data Integration Project (AgDIP). This project has been a multiyear partnership between ABARES and the Australian Bureau of Statistics (ABS) to develop a new secure database of Australian farms. The project provides a range of new data to explore the effects of seasonal climate and drought on farm outcomes, trends in Australian crop production, and water productivity in the Murray-Darling Basin.
Neal and I met to talk about the ongoing policy benefits from using the database. I also learned more about Neal’s own development as a data scientist, and how to advance data science across government. Read on. (Meredith Padgett)
What inspired AgDIP?
It started after I completed my PhD through the Sir Roland Wilson Scholarship in 2015. During that time I got to learn a lot about machine learning and data analysis methods and what they are capable of. I came back to the APS with an interest in applying them to climate and drought. It’s an area where there is a lot of policy interest, and good data. Matching this data with machine learning was the seed of an idea. ABARES had worked for a number of years on a model called ‘Farmpredict’ a farm and location - based model that uses weather predictions to estimate farm returns. We considered how we might scale it up.
Tell us more about how ABS data and tools progressed the project and next steps?
AgDIP combined annual Agricultural Census and survey data since 2000-2001, to construct the Farm Level Longitudinal Dataset (FLAD) which was then integrated with the Business Longitudinal Agricultural Data Environment (BLADE). By extracting the best insights from existing data and then integrating this with new modern data analysis methods, the project aimed to deliver more value to industry and policy makers as well as helping ABS improve its agricultural statistics methods.
It was a long collaborative process. It will be a lot easier going forward. It is the intention that we continue to build on this data set. But it’s not just about the data, a lot comes down to the relationship (between ABARES and ABS) that has been built up.
You started your career as an economist and have progressed towards data science. What do you love the most about this kind of work?
I like getting the answers. The thing that attracts me to machine learning is that it is intensely practical. To me at least some of the traditional statistics that are taught in economics degrees can be a little too theoretical with too much emphasis on elegance. Here in government it isn’t about elegance, it is about finding the answer…and making sure your answer is driven by the data and not your assumptions.
It is trying to do something that has long-term public benefits, which is hopefully what brings people into the public service in the first place.
What do you expect to see from a data scientist in the APS?
You’d expect to see coding competency, that’s the bread and butter of analysing data, running regressions and building models (and), using the standard tools people use these days, like R and Python, SAS. But then you are looking to go beyond that and have the conceptual, theoretical knowledge that lead you to think about problems and work out not just how you code something but how you build a model, which variables are likely to affect which other variables; thinking about causality, which then requires you to build up domain knowledge in a particular area.
What key advice do you have for data science public servants on big projects?
Having patience, being flexible. You have to be continually evaluating. You have a period of exploration, you learn something, and then you have an idea about what the useful next step is, and you may realise that it is a completely different step to the one you thought it was going to be.
Download the AgDIP research report.