Close ☰
Menu ☰

The right way to share data

Posted on: Tuesday 9th of June 2009

Here’s the full text of a talk I gave for the GCLive Expo at Earls Court on 9 June 2009. It’s delivered in a debate format with a spokesman from Home Office and another from Intellect. I’ll try to deliver it in a loving and constructive spirit, but without mincing words because this is important. I’m concerned about data sharing and the Database State. I’m pleased that there may be a constructive way forward. And I’m daunted by the scale of change of heart necessary before we can embark on that change. And that’s without starting to think about the scale of the practicalities for Whitehall and the wider public sector. It feels overwhelming! Full text below…

The right way to share data for public services

talk for GCLive, Earls Court
by William Heath
9 June 2009

The right way to share data for public services
Prevailing wisdom, and endless guidance from government, suppliers and consultants tells us the WRONG ways to share data for public services.

Start with a diagram that puts your own organisation at the centre of the universe. Gather as much personal data as possible at every stage. Keep it for ever. Buy in more data from brokers to try to try to fill the gaps. Then share it with other organisations.

Dear Friends – I call this the wrong way to share data.

Let’s understand Transformational Government as it feels and how it works on the receiving end, from the perspective of a generally law-abiding taxpayer whom we have no reason to treat as a suspect. I’m talking initially about health, education and welfare. Not border control and criminal justice.

I’m guessing you became public servants to show compassion, to show respect for people’s dignity and privacy and to behave as a good and obedient servant behaves.

This “my organisation is at the centre of everything” architecture is the behaviour of a master doomed by his own arrogance.

The attempt to achieve total data sharing right across so-called “joined-up” government is doomed. Technically it’s unfeasible, without a whole “data librarian” function of a sort that has barely been thought through let alone devised and implemented.

It’s doomed socially and politically because it’s the basis of an undesirable relationship. It’s “the Computer says “no”. It pours concrete into the living heart of our relations with the state. It creates a climate of coercion and enforced conformity, when what we need to solve our shared problems is collaboration, participation and flexibility.

This wrong way of sharing data creates a sort of toxic soup of your, my and everyone’s valuable personal details in which precious facts, nuances, changes and future intentions are lost or contaminated.

Out of this primordial slime is meant to evolve:
– personalised services
– meaningful choice
– efficiency savings on a colossal scale
– and a restoration of trust.

The ill-conceived Whitehall attempt, aided and abetted by a short-termist and morally short-sighted Intellect, to know everything about everyone, to meet every need and to predict future behaviour is a doomed God Quest. Let’s be monotheistic about this. There is only one God. It’s not Gus O’Donnell. It’s not the CIO Council, however tempting their chequebooks may be, and it certainly isn’t the Home Office.

These 277 data leaks [Sureyya referred to] aren’t PA’s fault, or the fault of some junior clerk. Despite the PM’s specific denial at the time of the HMRC disks loss, they do form part of a systemic and structural problem.

The data losses are the inevitable consequence of a deeply flawed intention behind the data sharing philosophy of Transformational Government.

You can’t implement this well. It can only be implemented badly or not at all. And that’s what’s happening.

Furthermore, to compel the gathering and sharing of sensitive personal details right across essential public services is substantially illegal under European laws on privacy and human rights. These state very clearly that you may only gather and share sensitive personal data either with the data subject’s informed and explicit consent, freely given, or in exceptional cases specifically permitted by law. Any such law must be narrowly enough drawn for its effects to be predictable by data subjects. It must be proportionate and necessary in a democratic society.

I’m a co-author of the recent Joseph Rowntree Reform Trust’s report Database State which assessed 46 major public-sector database projects against these legal criteria.

We found these databases were vulnerable to challenge under European law and likely to be found illegal.

We found these databases were almost certain to be illegal under European law.

The only UK public sector database yet to have come to court (and the verdict came as we finalised our report) was the DNA database. It was indeed found to be illegal.

I ought to make it clear that I’m still speaking here about data sharing for participative and elective services. Time is short and we must leave on one side the discussion about data sharing for border control and criminal law enforcement. For services like health, education and welfare we can illustrate the present CRM-derived Transformational Government philosophy like this

This is an organisation hell-bent on the God Quest.

It wants to know everything about its millions of customers. CRM consultants said that was the way to offer personalised services, save money and gain customers’ trust.

It spent all the money, but it’s only got a quarter the data it needs. There’s inaccuracy and duplication. Each record is costing it several pounds a year to maintain. Worse still, customers now don’t trust it at all and are refusing to co-operate. They’re opting out of the relationship as fast as they can, just like they’ve opted out of direct marketing by post, phone and online and out of the unedited electoral roll.

Individuals meanwhile have dozens of organisations they have to keep up to date with. Each has a different web form, call centre, and format for the data they demand. It takes ages and rarely brings any benefit, so you do as little as possible.

This is the wrong basis for sharing data. It’s expensive, dysfunctional, and has broken down individuals’ trust and co-operation even with organisations they want to deal with online.

It’s hugely wasteful of people’s time.

The right way to share personal data (I still mean for elective services, ie not for border control or criminal law enforcement) is under the control of the individual to the greatest possible extent.

We illustrate it like this

The individual has a personal information manager for all personal data (and the BSI released its standard for such a service only last week). This can be on their phone, laptop, or in the cloud or a mixture of all three.

They’re able to ping authentication services: credit checks, DVLA to confirm their driving licence and endorsements, a university about their degree, or NIR to confirm their unique identity.

They then selectively disclose, under the individual’s control and under standard user-friendly Ts&Cs, the structured data they choose to share with the organisations they want or need to transact. They allow organisations to subscribe to the their updates or “feeds”.

Because it’s done their way, under their control, they may strat to share future intentions like I want to buy a car or help me plan and manage a complex health comdition.

When data is shared that way, the right way, under the user’s control, the result is this:


The organisation still has a database, but can lose the burden and cost of updating admin data. Individuals do it for themselves more accurately and at no cost to the organisation. Individuals feel in control and gain the confidence to volunteer personal information of greater significance and value about their real needs, preferences and future intentions.

It’s that single change of address service we were promised for 2005. It’s better than Tellusonce which spills your data everywhere: it’s the easy way to tell whomever you want, whenever you want to.

As Doc Searls says, a willing customer is more valuable than a captive one. Same applies to students, citizens, subjects and taxpayers.

Let’s focus on that new value. Just how valuable do we think that will become? Very valuable indeed.

Bear in mind that 10 years ago Google started to elicit from people what they were searching for. After three years Google worked out (or pinched from Bill Gross of a business model and monetised the first category of volunteered personal information: the search term. Search is now worth perhaps £2bn a year in the UK alone.

But there are other categories [SLIDE – LIST OF VPI CATEGORIES] And it’s clear that the rest will quickly become more valuable than search.

In 10 years it might look like this: [CTRL SHIFT SLIDE]. The value of search may have increased 50%. But even ifit were to double, search will account for only a fifth the market value of VPI by 2020.

What will be the effect of VPI on public services? Four sorts:

– it will cut costs because people update their data themselves

– improve the quality of public services because the data is more accurate and can become far richer

– de-tox the “database state” because it can minimise central data holdings and creates a strong architecture of consent and

– the basis for co-creation which is the soundest possible basis for the creation and improvement of public services.

That’s in summary, why so much of Transformational Government is fatally flawed. It’s not just badly done or not yet done; it’s inappropriate and undesirable.

Happily the tools, techniques, legal and technical standards are rapidly emerging to solve this and provide personalised services in a more varied, empowering and respectful and responsible way.

Happily, too, the emerging user-controlled model will be legal under human rights and privacy law. There’s no disproportionate, unnecessary coerced gathering and sharing of data. Data is shared with clear informed user consent – indeed under user control.

As long as the Home Office, Intellect, and CIOs can give up their preconceptions, stop seeing themselves as the centre of the universe, overcome some hubris, go with the letter and spirit of human rights law, as long as they flick the big empathy switch, we can move public services on to the right way of sharing data and provide a rich evolving variety of personalised serivces.

The right way to do this is, as far as possible, under the control of the individual.


[thanks to Brian who pointed out I meant various European laws: the Council of Europe Convention on Human Rights and EU Charter Of Fundamental Rights, and ECtHR case law. I’ve amended “EU” to European above to preserve accurate generality]