{"id":522,"date":"2022-02-14T08:00:00","date_gmt":"2022-02-14T07:00:00","guid":{"rendered":"https:\/\/evidencehub.net\/blog\/?p=522"},"modified":"2022-03-07T17:09:06","modified_gmt":"2022-03-07T16:09:06","slug":"context-is-king-how-correct-data-can-lead-to-false-conclusions","status":"publish","type":"post","link":"https:\/\/evidencehub.net\/blog\/context-is-king-how-correct-data-can-lead-to-false-conclusions\/","title":{"rendered":"Context is King: How Correct Data Can Lead to False Conclusions"},"content":{"rendered":"\n

On 12 October 2021, in a now infamous Joe Rogan Podcast<\/a> anti-vaxxer Alex Berenson said that \u201cthe vast majority of people in Britain who died in September were fully vaccinated,\u201d offering this dubious fact to support his view that COVID-19 vaccines were as dangerous as the virus itself. The statement, while formally correct and based on official data from the United Kingdom National Health Service, was highly misleading: it missed the crucial context that the vast majority of the British population was vaccinated so of course the vast majority who died was vaccinated as well. In fact, the mortality rate of the unvaccinated population \u2013 seen in the right context \u2013 was several times higher than the mortality rate of vaccinated people. In the aftermath, many scientists and fact checkers<\/a> moved to clarify this. Charts were published to unpack the issues involved and inform the public about the importance of contextual data in drawing the right conclusions. The whole story went down as an episode of misinformation and has now blown into a full-fledged crisis at Spotify Technology<\/a> s.a., the company hosting and financing the podcast.<\/p>\n\n\n\n

\"\"

Source: Our World in Data<\/a> <\/figcaption><\/figure>\n\n\n\n

Of\ncourse, vaccination is a uniquely important topic, attracting high levels of\nattention, so the reaction was prompt and effective. But the majority of public-policy\nissues do not get such scrutiny and attention, and the use of out-of-context\nstatistics is much less likely to be noticed or corrected. <\/p>\n\n\n\n

One area\nwhere similar leaps to conclusion are taking place is the ongoing debate on the\ndigital services act (DSA), an omnibus piece of European Union legislation\nmaking its way through the cumbersome EU decision-making process. Differing\nversions of the legislation are now before\nthe European Parliament<\/a> and the Council\nof the European Union<\/a> (which represents the EU\u2019s 27 member states) while\ntechnical experts negotiate a common text, which, once approved, will become\nlaw (the process is called \u201ctrilogue,\u201d because the European Commission is part\nof the negotiation, too). Both versions seek to set new, tougher requirements for\nonline platforms, including granting consumers the right to appeal algorithmic\nrankings and requiring some companies to disclose potentially sensitive data to\nnon-profit organisations and journalists. And both versions seek to expand\nEuropean Union powers in a crucial area: tougher reporting requirements and\nrules for goods being sold online. <\/p>\n\n\n\n

But\nis the problem of illegal and counterfeit goods really as large as the solution\nproposed? Officials in France \u2013 the home of luxury goods makers Herm\u00e8s\nInternational s.a., Kering, L\u2019Or\u00e9al s.a. and LVMH Mo\u00ebt Hennessy Louis Vuitton \u2013\nseem to think so. On 15 October 2021, almost at the same time as the Rogan podcast\nfiasco, the French ministry of economy, finance and recovery published Conformit\u00e9\ndes produits vendus en marketplaces<\/a><\/em>, a timely report obviously intended\nto feed the ongoing DSA debate in Brussels. It stated, boldly, that 60% of the products\nit had sampled were illegal goods that did not comply with basic health or\nother product standards \u2013 and 32% were downright dangerous. But the context and\nclose analysis of the underlying data showed a different story, one where the problem\nwas perhaps not as systematic as a skewed look at the data would imply. The\nsurvey is based on merely 129 products across 10 marketplace and no information\nis provided on how the sample was selected. Amazon.com Inc., for one, offers 75\nmillion products for sale every day. Is their online market really to be judged\nbased on a highly selective survey of 129 products?<\/p>\n\n\n\n

This\nis not a unique case. The use of \u201cout-of-context\u201d evidence has been a recurrent\nissue in the debate over product safety, as this author pointed out in Fighting\nCounterfeits or Counterfeiting Policy? A European Dilemma<\/a>, a previous post\non the Evidence Hub<\/a>, a Lisbon Council\nproject created to discuss and disseminate the evidence being used in\npolicymaking. For starters, there should be a thick blanket of caution over the\nheadline \u20ac121 billion counterfeit market estimate that the European Commission cites\nin the impact\nassessment of the digital services act.<\/a> The \u20ac121 billion figure comes from Trends\nin Trade in Counterfeit and Pirated Goods<\/a>, an European Union Intellectual\nProperty Office (EUIPO) and Organisation for Economic Co-operation and\nDevelopment (OECD) report, which took a much more careful approach to the data.\nIt noted that most counterfeit goods seized at the border are watches, clothing\nand bags, which are brand-sensitive items where the original is worth much more\nthan the cheaper knock off. But the market figure presented is calculated based\non the value of the original good and not the estimated cost of the much\ncheaper counterfeit. This, as the EUIPO puts it bluntly, \u201cmay lead to an\ninflated estimated value\u201d in the overall figure. But nuance like this often\ngets lost in the policy debate. It can lead not only to wrong conclusions; it can\nalso lead to wrong actions as well. <\/p>\n\n\n\n\n\n

And there are other examples. One recurrent policy argument is that counterfeit good are a major danger to consumers. To prove this, in a summary of the data compiled in the Counterfeit and Piracy Watch List<\/a>, a European Commission-led project in which stakeholders are invited to report marketplaces where they suspect counterfeit goods are being sold, the European Commission reported that \u201c97% of reported dangerous counterfeit goods were assessed as posing a serious risk to consumers.\u201d But the data came from a different source \u2013 the Qualitative Study on Risks Posed by Counterfeits to Consumers<\/a>, an EUIPO report. There, the EUIPO stated bluntly that the dataset used for the survey was too small to allow for statistically significant analysis and even labelled the study \u201cqualitative\u201d as a nod to the limited conclusions that should be drawn from a sample so small. But even there, the results told a different story than a superficial glance might reveal. Of  the 15,459 dangerous products then identified in the EU Rapid Alert System<\/a> \u2013 an online portal where countries can share information with other countries about dangerous goods they have identified \u2013 only 191 were counterfeit. In other words, only 1% of the total number of dangerous goods turned out to be fake \u2013 not the 97% you might have thought if you\u2019d only ever seen the original statement presented out of context.  <\/p>\n\n\n\n\n\n

Most\nof these questions would be academic if policy were not due to be made on the\nbasis of these statistics \u2013 and their out-of-context citation. To the contrary,\nrules are being drawn up \u2013 sometimes hastily with only limited impact\nassessments \u2013 that would require online platforms to take steps above and\nbeyond the commitments already made in the Product\nSafety Pledge<\/a> and the Memorandum\nof Understanding on the Sale of Counterfeit Goods on the Internet.<\/a> Among\nthe provisions in the DSA are expanded rules to require intermediaries to\nverify the accuracy of trader information and listed goods before agreeing to\nsell them (providing a \u201cbest\u201d effort towards that end) and, according to the\ndraft favoured by the European Parliament, to carry out random checks on goods\noffered for sale to confirm that they are not illegal. As Victoria de Posson,\nsenior manager for public policy at the Computer and Communications Industry Association\n(CCIA), notes: \u201cWhile the public debate has centered around a few well-known\ncompanies, the truth is that lawmakers will be imposing new\nobligations on tens of thousands of companies in Europe, most of which are\nsmall businesses.\u201d<\/a> <\/p>\n\n\n\n

Behind these efforts is an unspoken implication that e-commerce is somehow contributing to a rise in the sale of counterfeit and illegal goods. But despite the continuous rise in e-commerce over the years, the amount of counterfeit goods in EU trade is falling, according to Global Trade in Fakes: A Worrying Threat<\/a>, an OECD\/EUIPO report. This remarkable trend coincides with several important developments: border agents and shipping companies are working better together to crack down on the brand knock-off trade and better police the shipping container industry, which accounts for 81% of counterfeit goods trade;<\/a> and the onset of self-reporting procedures and marketplace-based crackdowns<\/a> has brought needed transparency and accountability to a process which had few viable controls up until recently.<\/p>\n\n\n\n\n\n

But the debate over what to do now seems to be taking place in an alternate universe, a place where the attitude one demonstrates to platform behaviour is more important than the reality that platform behaviour might be begetting. The tone has become more of a Sergio Leone western than a policy discussion, as one recent tweet suggested.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Regardless\nof where the DSA lands \u2013 and it seems unlikely at this point that the governing\nbodies of Europe will re-examine the statistical basis on which they have drawn\nsome rather wide-ranging conclusions \u2013 we should all reflect on how the\nEuropean debate on technology can be brought more firmly into the realm of\nevidence-based policymaking and why this move \u2013 so obvious on the surface \u2013 is\nso difficult to realise in practice. For starters, this discussion should serve\nas a gentle reminder of a frequently overlooked point: the problem of misinformation\nis much more complex than we perceive. The same logical fallacies we consider\nmisinformation of the worst kind can still find a place in official policymaking\n\u2013  without even raising an eyebrow.<\/p>\n\n\n\n

And\nthere are better ways to handle evidence than the intense, deeply politicised\ncherry-picking that has gone into the illegal-content proposals. \u201cHandling\ncomplex scientific issues in government is never easy \u2013 especially during a\ncrisis when uncertainty is high, stakes are huge and information is changing\nfast,\u201d writes Geoff Mulgan, professor of collective intelligence, public policy\nand social innovation, in \u201cCOVID\u2019s Lessons for\nGovernments? Don\u2019t Cherry-Pick Advice, Synthesize It,\u201d<\/a> a recent article in Nature<\/a>,<\/em>\nthe science journal. \u201cThere\u2019s a striking imbalance between the scientific\nadvice available and the capacity to make sense of it,\u201d he adds, noting \u201cthe\nworst governments rely on intuition.\u201d <\/p>\n\n\n\n

Researchers have a way of avoiding problems like the ones detailed in this post \u2013 it is called \u201cevidence synthesis,\u201d which is \u201cthe process of bringing together information from a range of sources and disciplines to inform debates and decisions on specific issues,\u201d<\/a> according to one definition. It is also a way of making sure that the facts we use for policymaking reflect real market conditions and not the hand-picked realities that special interests would like us to see. It is in everyone\u2019s interest to make sure the evidence is balanced and the standards are robust, sustained and proportionate. It makes for better policymaking \u2013 and for better lives as well.<\/p>\n\n\n\n

David Osimo is director of research at the Lisbon Council <\/p>\n\n\n\n

Download<\/a>Download<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"

On 12 October 2021, in a now infamous Joe Rogan Podcast anti-vaxxer Alex Berenson said that \u201cthe vast majority of people in Britain who died in September were fully vaccinated,\u201d offering this dubious fact to support his view that COVID-19 vaccines were as dangerous as the virus itself. The statement, while formally correct and based … <\/p>\n

Continue reading “Context is King: How Correct Data Can Lead to False Conclusions”<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23,1],"tags":[],"coauthors":[15],"class_list":["post-522","post","type-post","status-publish","format-standard","hentry","category-illegal-products","category-uncategorized","entry"],"_links":{"self":[{"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/posts\/522","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/comments?post=522"}],"version-history":[{"count":14,"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/posts\/522\/revisions"}],"predecessor-version":[{"id":601,"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/posts\/522\/revisions\/601"}],"wp:attachment":[{"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/media?parent=522"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/categories?post=522"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/tags?post=522"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/evidencehub.net\/blog\/wp-json\/wp\/v2\/coauthors?post=522"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}