How A Tiny Change In A Census Question Had A Huge Impact On The Jewish Population

According to Statistics Canada, there were 351,705 people who identified their ethnicity as being Jewish in the 2006 census. In the 2016 census, they reported less than half that number: 143,665. Was there an exodus of biblical proportions?

“Obviously half the Jewish community in Canada didn’t disappear…” said Shimon Koffler Fogel, chief executive at the Centre for Israel and Jewish Affairs, in an interview with the Globe and Mail.

There was, however, a tiny change in a census question which had a huge impact. Let’s learn from it.

In both the 2006 and 2016 census the question was the same: “What were the ethnic or cultural origins of this person’s ancestors?” What changed was a list of examples of ethnic origins. According to Statistics Canada “…the list of examples of ethnic origins was updated in 2016 to reflect the frequency of single responses reported in the 2011 National Household Survey. For 2016, ‘Iranian’ and ‘Mexican’ were added to the list of examples, while ‘Jewish’ and ‘Salvadorean’ were removed.”

By removing the example of “Jewish,” the number of people who wrote it in as their ethnicity plummeted. But that was just one of the factors that made the question vulnerable to a radical change in results.

The fact that the citizen was asked to write in their ethnicity, rather than select from a list, meant the examples had an even more powerful influence on what people wrote. And, to make matters worse, the question of ethnicity is vague. Is being Jewish an ethnicity, a religious identity or both? Even Statistics Canada admits “Ethnicity is a difficult concept to measure, and there is no internationally recognized classification for this concept.”

All these factors came together in a perfect storm and apparently caused half of the Jewish population of Canada to vanish in a few short years. Statistical errors can cause scandals, large and small.

There are three lessons we can learn from this very public gaffe by an internationally well-regarded statistical agency:

  1. Open-ended questions are particularly vulnerable to influence by context;
  2. Providing examples will invariably influence results;
  3. Questions about vague constructs provide results that are less concrete than they appear.

Statistics Canada should be chastened by how brittle their question is, but they are certainly not alone. The U.S. Census Bureau asks a very similar and equally vulnerable question about ancestry and ethnicity, and provides examples in just the same way.

I wonder what ethnic group will suddenly disappear from the United States when they change their examples?

From the 2016 American Community Survey by the United States Census Bureau.

To learn more about the importance of how you ask a question, check out our article Asking Good Questions Is Not Simple: 3 Lessons Learned About The Art Of Asking Questions, or contact us.

next post thumbnail