Best practice guidance on digital storage and preservation

1 Introduction

How and where digital information and records are stored will affect their viability over time. Digital information and records need to be managed to meet the requirements of the Public Records Act 2005 and the principles of the Information and Records Management Standard (16/S1). This helps to ensure that these information and records remain authentic, reliable, discoverable, accessible, usable, protected and preserved for as long they are required. This also enables public sector organisations to meet their business needs and legal requirements, particularly for information and records identified as high-risk or of archival value.

This document contains guidance for organisations considering how and where their digital information and records may be stored and provides best practice guidance for those organisations intending to use or provide digital preservation storage. Given the rapidly changing and evolving nature of digital storage and preservation, this guidance is deliberately generic and high level. Organisations should first consult with their IT specialists to gain an understanding of their organisation’s technical environment before contacting Archives New Zealand for advice. This guidance does not cover contractual, jurisdictional or funding issues that are also part of decisions on digital storage and preservation solutions. 

2 Storing digital information and records

Previously, digital information and records were stored using discrete media such as individual CDs, tapes, etc. which were then migrated periodically to address media degradation and obsolescence. It is becoming more common practice to use resilient IT storage systems [1] for the growing volume of digital information and records that need to be preserved, and more importantly, that need to be easily and quickly retrievable in a culture of online access. In this way, management of digital content can be decoupled from the mechanism of its storage, i.e. the media or technologies and the supporting IT infrastructure. With this comes the consequent benefit of allowing different preservation activities to be handled independently.

How and where organisations store digital information and records is key to ensuring they remain clear and reliable for the entire time they need to be kept.

2.1 How can digital information and records be stored?

There are several ways in which digital information and records can be stored, including:

Online: This can be locally on an organisation’s server infrastructure, or by hosted storage through the internet, for example in cloud storage.  Networked online storage is where data is stored on multiple virtual servers that are generally hosted by a third party, which may be offshore. Information and records held in online storage devices are immediately accessible to users and more likely to be identified and included in changes such as system wide migration processes, and in regular integrity checks and back-ups.

Offline: This allows information and records to be relatively accessible, for example, on removable storage media such as magnetic tapes, CDs, DVDs, memory cards, flash drives (USB sticks). Organisations need to be aware of the risks with using removable media, for example, data security, malware infections, loss and hardware failures. Removable media is often overlooked when systems are upgraded, and digital information and records migrated to new formats.

Near-line: Information and records are stored separate from and not directly accessible by the organisation’s systems but from which data can be quickly retrieved and brought online for access, for example, a local tape library or a cloud storage service.

Organisations should consult with their IT specialists about their specific digital storage requirements.

2.2 Storage approach considerations

Organisations should use the following to guide their selection of digital storage systems:

Security

If information and records have privacy and/or security requirements, consider how specific storage systems will allow these to be managed and enforced.

Access and availability

If information and records need to be accessed often and/or quickly, consider selecting storage media with fast retrieval times. Also consider what specific combinations of hardware and software are needed.

Longevity

If information and records need to be kept long term, consider selecting storage media with a proven lifespan that is appropriate for the retention period of the information and records. Longer lifespans will reduce the need to migrate or refresh the storage media or undertake other preservation activities to reduce the risk of data loss.

Viability

Consider what error detection and integrity checks are in place to monitor and ensure against inadvertent change, deterioration or loss of information and records over time and/or when storage media is refreshed, or data is migrated.

Obsolescence

Most digital storage media will only last 5 to 7 years before it will be necessary to refresh or update it.[2] Consider when storage media and their technical infrastructure are likely to become obsolete or unsupported. Select storage systems that are robust with a regular, clearly defined migration path and widespread industry support.

When selecting or designing storage systems for preservation storage, organisations should also consider the following principles from the Digital Preservation Coalition’s Digital Preservation Handbook [3]:

Redundancy and diversity

For example, make lots of copies, stored in different locations; use a combination of online storage systems and offline media; use different types to spread risk and balance data safety with easy access.

Fixity, monitoring and repair

For example, use fixity measures such as checksums to record and regularly monitor the integrity of each copy; store fixity information alongside the digital objects as well as in separate systems.

Technology and vendor watch, risk assessment and proactive migrations

For example, keep an eye on new and changed technology and the viability of vendors or classes of storage solutions; migrate storage before digital objects become at risk.

Consolidation, simplicity, documentation, provenance and audit trails

For example, minimise the proliferation of legacy media types and consolidate digital objects onto a minimum number of storage systems.

3 Digital preservation storage criteria

Preservation storage supports digital preservation which is defined by the Digital Preservation Coalition as “the series of managed activities necessary to ensure continued access to digital materials for as long as necessary” [4]. The following nine criteria or characteristics of preservation storage are based on international best practice. [5] They are intended not only to help organisations with developing requirements for digital preservation storage systems or solutions, but also to help with evaluating digital storage options and services and informing IT infrastructure design and planning. Organisations should adapt the criteria to suit their individual requirements, practices, legislation and environment.

3.1 Content integrity and authenticity

The digital preservation storage solution (the solution) provides and/or supports practices ensuring that changes to the state of stored information and records are documented and can be traced. For example, the solution:

• performs verifiable and/or auditable checks to detect changes or loss in or across copies (e.g. checksum recalculation, fixity checking, identifying missing files, etc.)

• supports independent integrity checking

3.2 Content discovery, identification and reuse

The solution provides and/or supports facilities that enable users to discover, identify, understand and reuse information and records in a persistent way over time. For example, the solution:

• maintains accessible and useable metadata to appropriate standards

• provides unique, persistent identifiers or locators for digital objects

3.3 Flexibility

The solution is adaptable, interoperable and customisable to an organisation’s needs. For example, the solution:

• can adjust storage infrastructure in response to changing requirements (e.g. legal requirements, audit results, etc.)

• uses or includes storage components that can be easily integrated with other systems and applications, i.e. plug and play (e.g. standard file access protocols and file system semantics, etc.)

3.4 Information and system security

This characteristic refers to the protection that the technical infrastructure and/or the solution provides not only for the solution itself, but also for the information and records, products, services and users. For example, the solution:

• provides role-based, access controls to ensure that the information and records cannot be easily altered or inappropriately accessed

• provides immutable logs and/or reports that show all system errors, failures and other critical system activities

• includes software that regularly runs virus checks and malware detection and provides remediation actions (e.g. quarantine, notification, etc.)

3.5 Resilience

This characteristic refers to the durability and availability of the solution. For example, the solution:

• provides documented and acceptable longevity, failure rates and technical characteristics of the storage media components

• performs error detection and correction 24/7/365

• has a high percentage of uptime, i.e. operational for a long length of time, due to techniques such as eliminating single points of failure by effective monitoring

3.6 Scalability and performance

The solution meets specified computational performance and is capable of being scaled up or down according to organisational needs. For example, the solution:

• supports the bulk exporting of content and metadata for any reason, at an acceptable rate (e.g. as part of an exit strategy)

• supports long file path or directory names and diverse character encodings

3.7 Support

The solution’s technical and organisational infrastructure is reliable, stable and appropriate. For example, the solution:

• can support additional digital preservation services (e.g. migration and transformations with auditable results)

• references relevant international and community standards (e.g. ISO 14721:2012 OAIS reference model, New Zealand Government Web Standards, etc)

• documents a planned and organised approach and commitment to long-term preservation, continued access and usability

3.8 Transparency

This characteristic refers to the visibility into the solution’s functions such as auditing, reporting, error notification, and documentation. For example, the solution:

• provides reports about content (e.g. number of objects/files/formats, average file size, types of objects, etc) as well as custom configurable and on-demand reporting of content or activity

• captures and documents audit/provenance information about all changes (e.g. integrity check failures, deletions, modifications, additions, preservation actions) and who or what performed the actions

• supports management and monitoring across multiple storage availability levels (e.g. online, near-line, offline)

• provides full, complete, current and available system design and configuration documentation

3.9 Risk management

Digital preservation requires storage solutions that can be sustained over the long-term. For preservation storage, risk management must consider that no or only acceptable amounts of data may become lost. For example, the solution:

• Includes effective mechanisms to establish bit safety and detect bit corruption or loss such as number of copies, independence between copies, and integrity checks

---

[1] A resilient IT storage system consists of storage media contained within a server that provides built in resilience to various failure modes by using inbuilt redundancy and recovery. Digital Preservation Handbook, 2nd Edition, https://www.dpconline.org/handbook, Digital Preservation Coalition © 2015.

[2] Queensland State Archives, “Select storage for digital records”, 2018. Retrieved from https://www.forgov.qld.gov.au/select-storage#-a-name-digital-a-select-storage-for-digital-records.

[3] Digital Preservation Handbook, 2nd Edition, https://www.dpconline.org/handbook, Digital Preservation Coalition © 2015.

[4] Digital Preservation Coalition, “What is Digital Preservation?”, Digital Preservation Topical Note 1, n.d. Retrieved from https://www.dpconline.org/docs/knowledge-base/1862-dp-note-1-what-is-digital-preservation/file.

[5] A. Goethals, N. McGovern, S. Schaefer, G. Truman, and E. Zierau. “An overview of the Digital Preservation Storage Criteria and Usage Guide”, Proceedings of the 16th International Conference on Preservation of Digital Objects (iPres 2019). Retrieved from https://osf.io/sjc6u/ DOI 10.17605/OSF.IO/SJC6U; ISO 16363:2012. ”Space data and information transfer systems — Audit and certification of trustworthy digital repositories”, 2012; CoreTrustSeal. “Core Trustworthy Data Repositories Extended Guidance v.1.1”, 2018. Retrieved from https://www.coretrustseal.org/why-certification/requirements/.