• nl
  • en

How to measure success – and work on weaknesses (Austin PASIG, 7)

16 januari 2012 Gepubliceerd door Laat uw gedicht achter

As David Giaretta (Alliance for Permanent Access) would put it: “Measuring success in digital preservation is easy – if we have a 100 years or more.” But of course we are an impatient bunch, and funders especially want to know that their money is making a real difference. And yet, in an emerging field such as digital preservation it is not easy to develop objective and measurable quality criteria.

David Giaretta himself came forth at the conference to present a European framework project which is building a three-tiered approach to certification of trusted digital repositories for digital preservation: from the light-weight instrument Data Seal of Approval (which was developed separately by the Dutch DANS archive), on to self-assessment and ending in a fully-fledged third-party audit based on ISO standards.

The European certification framework

But the question remains: what can you measure objectively? Giaretta’s approach revolves around metadata: each digital object must contain enough representation information (information about everything you need to use the object: hardware, software and, e.g., vocabularies) to enable the designated community (the clients) to use the information (see OAIS for terminology). In principle, Giaretta said, this is testable. I have described Giaretta’s work on representation information in more depth here and here.

David "have you read my book" Giaretta

The next question that drew some comments from the audience is: who is doing the testing? How can the system ensure consistency? Giaretta pleaded the establishment of formal training and national accreditation boards. US consultant Bob “mister” Rogers added from an IT perspective that prospective auditors will need much more than knowledge of OAIS and representation information. They must know a lot about security issues and IT operations.

Bob "mister" Rogers

Need for a holistic approach

Rogers referred to a number of instruments that have been developed in the IT industry to take stock of cyber risks and assess the quality of systems (check out his slides when they become available. But, he warned, IT is only a part of the picture. “We need a holistic approach,” he stressed, that includes people and processes. The website www.datalossdb.org reveals, in Rogers’ words, “a staggering amount of data loss”. And, in case you wonder: intentional or unintentional mistakes by people are the most important cause of data loss:

slide Bob Rogers

Recurring conference theme: it’s people that make the difference

So, how do you audit people, I wonder. Remember Tom Cramer at bootcamp? It is all about the mindset, not about diplomas.

Bob Rogers gave the audience some insights into his consultancy practice during bootcamp. He stressed how important it is to talk with all of the stakeholders. Typically, records managers (and I would imagine librarians) want a thousand classifications; the IT folks will want everything to be nice and simple so it can be efficient; and the lawyers? They won’t tell you what they want – because lawyers are always vague about everything. In Rogers’ view, collaboration is the key – plus (of course!) an excellent consultant who asks the embarrassing questions.

“Ah, but does he get honest answers?” I asked Bob later. “Don’t most consultants write down what the customer wants to hear?” You can imagine what Bob’s answer was: a good consultant will be able to wriggle some change into politically correct language.

Measuring for improvement: the Digital Preservation Capability Maturity Model

At the end of the session, consultant Charles M. Dollar and his colleague Lori J. Ashley brought measuring to the next level: as a means to facilitate improvement. Interestingly, in the context of a conference about IT and digital preservation, Charles Dollar used a well-known IT model (CMMI), combined it with digital preservation standards such as OAIS (post to come) and TRAC (Trusted Repositories Audit and Certification), and built a “Digital Preservation Capability Maturity Model” (DPCCM), in other words: it measures how preservation-ready organizations are.

As a couple of Dutch organizations have been doing some work with the model in the Netherlands, it was a special pleasure for me to hear Charles and Lori talk about their approach during a 7 AM breakfast session. “Records managers,” Lori told me, “are overwhelmed by digital preservation. They simply don’t know where to start.”

Charles Dollar (left) and Lori Ashley of the Digital Preservation Capability Maturity Model

The DPCMM breaks up digital preservation into clearly defined components and scores them on a scale from 0 (nothing) to 5 (excellent):

The Dollar model (click to enlarge)

The resulting scores can then inspire prioritization and a roadmap for improvement:

“But how do you know where to begin?” I asked Lori, as that is a question that had come up in the Netherlands. Lori explained to me that that is where the consultant comes in. Through interviews he/she gets an acute sense of where something is glaringly missing and where the best opportunities for improvement can be found. A five-year improvement plan can then be drafted.

And if an organization scores zeros all around or is overwhelmed by the model itself? “Even that is alright, because many are in the same boat,” says Lori. “The model is about hope, even if it’s baby steps. The important thing is to leverage people’s passion about their organization.”


Gecategoriseerd in :Geen categorie

Geef een reactie

Het e-mailadres wordt niet gepubliceerd. Vereiste velden zijn gemarkeerd met *

Translate »