NZ Council for Civil Liberties

Submission: Future Census

About the New Zealand Council for Civil Liberties

  1. The New Zealand Council for Civil Liberties (‘the Council’) is a voluntary, not-for-profit organisation which advocates to promote human rights and maintain civil liberties.

Introduction

  1. The Council welcomes this consultation.
  2. The Council rejects all of the proposed options in this future census consultation, on both political and technical grounds.  We further reject the case for change.
  3. The Council further is troubled by the manner in which this consultation has been conducted.  Statistics New Zealand (‘Stats NZ’) does not appear to be capable of running the consultations required under the Data and Statistics Act 2022.

The Future Census Proposal

  1. The Future Census proposal is to move from conducting a true census by surveying everyone and filling in data which the survey reveals to be missing with admin-data, to using admin-data and only conducting surveys when there is reason to suspect that data is missing.

Political Flaws

Cost

  1. In the consultation documents, and in our regular meetings with leaders in Stats NZ, the cost of the 2023 census has been provided as the primary reason for change.
  2. Independent reviewers Murray Jack and Geoff Bowlby said moving from a traditional census to administrative data would involve a large and complex work programme, and great care and caution should be taken.  It should go without saying that “great care and caution” is likely to increase, rather than decrease cost for the first census under the new model.
  3. The 2023 census cost $316 million. There were significant failings in the 2023 census, including a software failure which resulted in over 100,000 unnecessary home visits.  These failures need to be addressed regardless of the approach to the next census.
  4. We are concerned that Stats NZ seems to know the cost of everything and the value of nothing.  Former Government Statistician Len Cook said about this proposal:1

The opportunity cost from potential investment failures far exceeds the statistics budget.  

and

First, neither Statistics New Zealand nor the Treasury are likely to be aware of the majority of the uses to which official statistics are put. Second, those who use official statistics do not invest enough in influencing how statistics can be developed to provide further value. Expert users themselves may be unaware of the scale of decisions in their sector which depend on the scope, frequency and quality of population statistics.

  1. Nowhere in the consultation documents or in our meetings, has Stats NZ attempted to quantify the benefit of the census. The cost of the census changed from $128 million to $316 million between 2018 and 2023. We remain entirely in the dark as to whether the benefit grew as well. Did it grow more quickly or less quickly?  And more to the point, if the value of the census as we conduct it now exceeds the cost of the census as we conduct it now, then there is less case for change.  
  2. Stats NZ have provided neither a cost estimate nor a benefits estimate for their new model. Without those estimates cost-saving can not be a reasonable argument for change.  
  3. In summary, Stats NZ has not provided the basic information needed to make a financial argument.  The Council therefore rejects the cost of conducting a traditional census as a ground for change.

The other cases for change

  1. Stats NZ has other reasons for change.  However, it’s done a poor job of presenting them, so the Council has been forced to guess at those reasons.  This alone should prompt the reader to question both the need for change and the ability of Stats NZ to enact the necessary changes.
  2. Stats NZ has identified “the time and effort it takes people to complete surveys” as something which could be reduced.  While obviously not surveying people places a smaller immediate burden on those people than surveying them does, there are other levers available to reduce the survey burden.  Asking fewer questions is an option which is available without an admin-first model.  As is not conducting the census at all.  However, we all benefit from accurate population statistics. The cost of answering and collecting the data are already taken into account in deciding which questions to ask in the existing model.  We are unconvinced by the evidence presented by Stats NZ that the burden on people when responding is disproportionate to the goals and benefits of the census. 
  3. Once again, the Council feels the need to point out that Statistics NZ has presented a case for change without evidence. In none of the supporting materials is there a measurement of the burden to fill in the 2023 census. We also note that Stats NZ has not provided any data comparing the number of questions asked in NZ censuses to the number asked in the censuses of other countries.

Reason to suspect

  1. The consultation does not clearly define where the boundary between admin and survey sources will be used.  Nor does it acknowledge that topic as being worthy of discussion.  This is a serious failing. 
  2. The Council recommends that for each attribute for which Stats NZ wishes to take an admin-first approach in any given Census, that they should publish evidence before that Census demonstrating why they have  reason to believe that the admin-data is sufficiently accurate.  Further, public consultation should be held to assess that evidence and then the plan revised as a result of those consultations.
  3. The Council notes that due process here, while increasing the total value of the census considerably, entails non-trivial additional costs over the survey-first approach, and therefore undermines cost as a case for change.

Population Registers

  1. Some nations have centralised systems which track the major events in the lives of their residents.  These countries have population registers, which go far beyond merely issuing identity cards by requiring those identity cards to be presented when conducting any business with the government, and when conducting important business when the government is not involved.
  2. It is difficult to overstate the wide ranging effects which would be felt by every person in New Zealand if we adopted a population register. Yet the list of nations used for comparison blithely includes several countries with “high functioning population register systems”.
  3. The council considers the admin-first change in future census to be a step toward a population register form of government.  We therefore oppose it as the national conversation and subsequent referendum which we think are necessary for this change have not occurred. 

Technical Flaws

Is Now the Time for Experiments?

  1. Former NZ Government Statistician and UK National Statistician Len Cook has written extensively about the problems with shifting to an admin-data census. The Council agrees with Len Cook’s assertion that:2

The failures in the censuses of population and dwellings for 2018 and 2023 leave little room for untested initiatives at this time of continued massive disruption to historic patterns of demographic change.

  1. The Council further notes that Len Cook asserts that the rest of government has signalled its concerns about the census. There does not appear to be an appetite for experimentation:3

The public reports by the parliamentary commissioner for the environment, the New Zealand Infrastructure Commission, the Ministry of Housing and Urban Development, the Ministry for Pacific Peoples and the auditor-general highlight deep concerns about the current quality of population statistics.

Absence of Ground Truth

  1. The consultation documents themselves make a clear argument in favour of a survey-first census:4

An advantage of surveying the whole population is that statistics can be produced for any group or area – no matter how large or small. With a sample survey, the ability to produce detailed information for small populations will depend on the size of the survey and the size of the group. However, even with a very large size the sample survey may not be able to deliver the same detail as when surveying the whole population.

  1. If Stats NZ could accurately identify the information missing in admin-data, then there is a theoretical case to be made that survey-first and admin-first are equivalent.  Practically, the Council believes that this task is impossible without conducting a survey of everyone, what we used to call a census.  Stats NZ’s  guesses of the gaps will always be incorrect, and will inevitably grow worse with time.

Data Quality

  1. The working definition of data quality is that data quality is good if the data was fit for its initial use.5 Despite the ubiquitous use of this definition in academia, industry, and indeed within our government, this definition surprises many people. By that definition, data is not fit for reuse except in the unusual case that the initial use had the same quality requirements as the new use:6

Whereas the coherence and compatibility of the information from a census of population or other statistical sources are anchored in the rich global infrastructure of population statistics, the qualities of each administrative record are unique and unrelated. They lack the adaptability, collective coherence, comparability and consistency that comes from information sources designed with the infrastructure of official statistics. It is quite rare for the information contained in administrative records to be designed to meet even a few of the long-standing standards and definitions on which official statistics are based; this means that the quality of the information created by their integration not only with official statistics but with other administrative records cannot be assured.

  1. Context always provides meaning to data. The very act of counting is as much about deciding what gets counted as it is about incrementing a number:7 

We are becoming increasingly aware that things are continuous spectra, not discrete categories. Writing about the failure of contact tracing apps, activist Cory Doctorow said, “We can’t add, subtract, multiply or divide qualitative elements, so we just incinerate them, sweep up the dubious quantitative residue that remains, do math on that, and simply assert that nothing important was lost in the process.

The Council acknowledges that there are usually subtle variations in definitions of what counts.

  1. Redesigning all of our government’s administrative systems to collect data fit for the quality needs of the census will not merely cost many billions of dollars.  Giving Stats NZ design control over other agencies’ data inherently also gives control over other agencies processes. It would rob every other government agency of their independence and of the flexibility needed to redesign themselves to the changing needs of our communities.
  2. In our meetings with Stats NZ, departmental officials have acknowledged that the quality of admin-data is a real problem.  However they believe that they have it under control.  They have presented no evidence.

The Consultation Process

Statistics New Zealand was unprepared for this Consultation

  1. This document has already noted that Stats NZ is attempting to make an economic argument for the proposed changes, but have failed to provide any estimates to back up that argument.
  2. The Council further notes that Stats NZ has made a number of crucial assertions about data coverage and data quality without publishing supporting material.
  3. Len Cook has noted that the United Kingdom “published methodological studies that enable the statistical and analytical limitations of substituting administrative records for a field enumeration to be evaluated.”  In meetings with the Council, Stats NZ revealed that not only has it not published equivalent material, but it does not have evidence on which to base its evaluation.  It has commissioned a report, but it will not be ready before the consultation closes.

Failure to respond to past feedback

  1. The Council noted earlier that at least five government agencies, and the independent Auditor-General, have recently criticised the quality of census data. The department has not publicly responded to these criticisms.
  2. The most recent report on the census, Our Futures Te Pae Tāwhiti, conducted by the Royal Society after the 2013 census, recommended the opposite of the proposed change to admin-first:8

It is unlikely that the census will become redundant in New Zealand. Firstly, it is an important ‘anchor’ for periodic assessment of the accuracy of other sources, such as population totals measured by births, deaths, and migration across the border. Unlike some countries, New Zealand does not have a system of civil registration which in part substitutes for census information. Secondly, New Zealand has a centralised system of government, and we use the national census for purposes which in other countries are the tasks of regional and local governments with their own information sources. Thirdly (in the course of preparing this paper, the panel became aware of this), for several issues the census is the only source of information. It is the only source of numerical information about iwi. Only the census provides information on places of residence, places of employment and means of travel. Assessing our transport options would be very difficult without a census. Linking the location of immigrants with their occupations and evolving skills would also be more difficult. In sum, the census will survive at least until there is a centralised system of accurate population and administrative data against which to pit the evolving range of research and policy questions.

The department confirmed in its May 2024 meeting with the Council that it never responded to Te Pae Tāwhiti.

The proposed options are designed to stifle debate

  1. In our May 2024 meeting with Stats NZ, the Council was surprised to hear criticism that “many of the submissions are off topic.”
  2. The substantive change which Stats NZ is proposing is the move from Survey-first to admin-first in the census.  Stats NZ maintains that the consultation is between three options, mentioned briefly on page 24 of the consultation document and never referenced again.  Those options are not significantly different, and their differences are tangential to the primary change being made.
  3. Section 36 of the Data and Statistics Act requires a consultation about the census.  However that section leaves the details completely to the discretion of the government statistician. The Government Statistician has chosen to propose a fundamental change, but to consult only trivialities. The Council is gravely concerned that Stats NZ have intentionally constructed this consultation in order to label feedback on the substantive changes as being off topic. Therefore, this is the first of two grounds on which the Council accuses the department of undertaking this consultation in bad faith.

Outcome is predetermined

  1. Stats NZ has publicly committed to a decision about which they are supposedly consulting:9

Stats NZ would soon be going to the minister with a recommendation on the shape of the 2028 census. Sowden would recommend a census that had a smaller, more focussed survey component, and a greater reliance on admin data.

  1. At every one of our meetings, and in the consultation materials, Stats NZ has left no doubt that the next census will be conducted under the admin-first model, which is what this consultation should be about.
  2. Having predetermined the outcome of this consultation is the second ground on which the Council accuses Stats NZ of conducting this consultation in bad faith.

What is a consultation

  1. The Council believes that this process does not meet the standards required of a consultation, in that the public have neither been provided with the necessary information nor does the department appear to be interested in allowing feedback to change its decision:10

Consultation must be allowed sufficient time, and genuine effort must be made. It is to be a reality, not a charade. To consult is not merely to tell or present.

  1. The Council wishes to note that consult is the second lowest level in the IAP2 Spectrum of Public Participation, from which the Department of Internal (DIA) affairs’ guidance on consultations is derived.,  Stats NZ should aspire to higher standards. 

Trust

  1. Stats NZ has a preoccupation with trust in government.  It has correctly identified that decreasing levels of trust in government pose an additional, existential threat to their work, in addition to the threat posed to government in general.
  2. Stats NZ is making some investments in building trust, like meeting regularly with the Council and other civil society organisations, as well as with Māori organisations.
  3. However, the policy processes at Stats NZ, and its approach to its obligations under the Privacy Act have been undermining public trust. Until this is fixed, Stats NZ will have little credibility with civil society and the public.
  4. There are competing definitions of trust. However, the Council believes that there is general agreement that transparency and alignment of purpose are central to most definitions.

Transparency

  1. As we have already noted at length, Stats NZ has failed to provide the information necessary to support this consultation. The Council also notes that we have made this same criticism in all of our submissions to Stats NZ in recent years.11 For example, in our 2020 submission on progress on the 2018 Data Strategy, we said:12

The Council is troubled that Statistics New Zealand provided no data when asking for an assessment of one of its programmes. To the best of our research, no information has been published on the implementation of the Data Strategy.  That lack of publication is in itself a significant failure of the strategy.  It also suggests a series of other failures in the strategy, for example a failure to define how to measure the strategy.  Finally, this failure to provide data brings competencies into question, and reduces our trust in this programme and in Statistics New Zealand. 

  1. We further note that in response to the Council’s May 2024 accusation that there is no legal basis for Stats NZ’s policy of refusing people’s Privacy Act requests for information held about them in the Integrated Data Infrastructure, Stats NZ decided that it was in the public interest to withhold all of the legal advice it has ever received on the subject. To paraphrase, Stats NZ has decided to keep secret the information at the centre of the discussion.
  2. In short, if Stats wishes to earn the public’s trust, it needs to become open.

Alignment of Purpose

  1. The failure to make a case for change leaves everyone wondering what goals Stats NZ is working toward.
  2. Further, the way in which Stats NZ has this consultation leads the Council to question Stats NZ’s motives.  This consultation should have been an exercise which demonstrates that Stats NZ is in touch with the public.  Instead by failing to disclose the necessary information and predetermining the outcome, it has done the opposite. 

Undermining Public Trust

  1. High quality public consultations, which present all the relevant evidence for all the options being consulted on, are opportunities for building trust with the public through transparency and public participation (or active citizenship as the Public Service Act calls it). Unfortunately, as we have shown, Stats NZ’s current exercise fails this test and therefore is unlikely to build public trust but instead damage public trust and confidence in the department and the government overall.

Conclusion

  1. The department is proposing a fundamental change to a cornerstone of our system of government.  They are running a sham consultation on it, having provided no data and having admitted that they do not have evidence that its proposal will work. From the publicly reported comments made by the Government Statistician and the flawed presentation of the options, it seems likely that Stats NZ has predetermined the outcome of this exercise.
  2. The Council calls for this consultation to be scrapped, and a new consultation to be conducted with an evidence-based case for change after the methodological study has been published.
  1. https://ojs.victoria.ac.nz/pq/article/view/9484/8393 ↩︎
  2. https://ojs.victoria.ac.nz/pq/article/view/9484/8393 ↩︎
  3. https://ojs.victoria.ac.nz/pq/article/view/9484/8393 ↩︎
  4.  p24 https://www.stats.govt.nz/assets/Consultations/Modernising-our-approach-to-the-2028-Census/Modernising-our-approach-to-the-2028-Census-Discussion-document-for-public-consultation.pdf ↩︎
  5. https://web.mit.edu/tdqm/www/tdqmpub/beyondaccuracy_files/beyondaccuracy.html ↩︎
  6. https://ojs.victoria.ac.nz/pq/article/view/9484/8393 ↩︎
  7. https://www.belfercenter.org/publication/seeing-data-structure ↩︎
  8. https://www.royalsociety.org.nz/assets/documents/Our-Futures-report-web-with-references.pdf ↩︎
  9. https://newsroom.co.nz/2024/05/10/government-looks-for-a-better-way-to-count-maori/ ↩︎
  10. Wellington International Airport v Air New Zealand (1993), as quoted on p4 of https://www.dia.govt.nz/diawebsite.nsf/Files/Good-Practice-Participate/$file/GPP_Levels_of_participation.pdf ↩︎
  11. see https://nzccl.org.nz/our-feedback-on-algorithm-governance/ for one example ↩︎
  12. https://nzccl.org.nz/aotearoas-data-strategy/ ↩︎