• LTDPRM: News and blog feeds for the preservation community – in one location, you can stay up to date on key developments and publications in the industry
• LTDPRM: Video feeds and access to training and educational materials
• LTDPRM: With launch of the SNIA’s Cloud Archive SIG, I’ve put up an important set of resources on requirements for digital archive (preservation) services in the cloud.
• ILM2.0:  Postings on buyer’s guides, classification, 
  IT and Information Governance in the Cloud, moving information into the cloud, etc.

In both domains we are building a community of world-wide expert participants.  Please register to have full access to the content and to contribute. These are public domain sites with full collaborative access.  Both sites have weekly telcons to progress the work and build relationships – check the calendars and join in.

From:  ”A New Set of Rules for e-Discovery Duties and Sanctions”   By Nick Brestoff,   Published in : EDDE Journal:    WINTER 2011 VOLUME 2 Issue 1 — A Publication of the E-Discovery and Digital Evidence Committee ABA Section of Science & Technology Law

“All ESI preserved in accordance with the Preservation Duty shall not be destroyed or materially altered until four years after the Proceeding is final. If such ESI is not then subject to any other Preservation Duty, it may be destroyed. However, if such ESI is subject to a Preservation Duty arising from any other Proceeding, that ESI shall not be destroyed or altered until one year after all such Proceedings are final.”

Juxtaposed Forces:

The impact is interesting in that it is getting harder and harder to delete information and data out of the datacenter.  On one hand we have IT’s cost-driven movement to incorporate capacity optimization in the datacenter and on the other legal and regulatory governance that extends retention periods and makes it harder to delete expired information and data – driving up costs and energy consumption.  In the preservation world, we are concerned with both, and the paradox of moving towards storage of preservation objects which are larger in size and not dedupe capable is causing capacity growth angst because you must have 3-4 copies of each object distributed for recovery, access, and business continuity.

Thoughts:

• What screams at me is “classification, classification, classification!”
• And, “deletion” as soon as possible – but without order and organization of information throughout the enterprise that is hopeless.
• How to deal with the load, the cost, the complexity – I only see one path.  It is the practice approach provided by ILM2.0 .   (for more go to www.ilm20.org)

Are there others?

According to Jeff Layton’s new article, “Data is Becoming Colder“, Jan 24, 2011  the historical data I have captured a number of times is evolving – most significantly  in file access patterns. Let’s see if we can put this in context.  Here’s my way to look at this issue.

Information State has 4 classes:  ”active, reference, inactive, expired”   – Source “Building a Terminology Bridge: Guidelines for Retention and Preservation Practices in the Datacenter,” 2009

20% of data is active
25% of data is expired
55% of data is reference or inactive

Layton’s Article states that it is only “CIFS” storage, meaning it is a subset of user docs and productivity apps.  It does not include the majority of storage consuming business apps. Factor that in.  Then I find data I have a difficult time assimilating. Let me illustrate.  (I posted some of the data below for comparison.)

a] Docs are more write oriented – a 100% shift – this is true based on the other data points that there is less read access once docs age.   Duh! Of course they are.  76% never opened by more than one person.  All that follows logically but what is the big deal.  Isn’t it more interesting that there is less collaboration in this community?  Isn’t the use case more interesting and says nothing about the world at large as it is a sample of just NetApp’s internal ops.  What about the Sharepoint/Google Docs collaboration extranets? Oh, they weren’t included…

b] This one throws me: “Files live an order of magnitude longer (10x). Fewer than 50% are deleted within a day of creation.”   …A day of creation?  We used to talk about 90days or longer. Not a day. If ~50% are deleted within one day, why is there a discussion?   And how is that 10X longer?

In the end, I just don’t get how the conclusion that “Data is getting colder” is derived.  What I would expect to see to justify that claim would be data like a comparison across the two studies (3 years apart) that showed volume % by state by study. Show me the shift in the total population statistics, normalized. What is the aging of the file system – here we need a profile of volume by age. Then a comparison of the two curves, normalized again.

Draw your own conclusions:

===========================  From the paper  ================

About three years ago there was a study from the University of California, Santa Cruz and Netapp that examined CIFS storage within Netapp itself. Part of the storage was deployed in the corporate data center where the storage were used by over 1,000 marketing, sales, and finance employees. The second part of the storage was a high-end file server deployed in the engineering data center and used by over 500 engineering employees. All together the storage amounted to about 22TB’s. During the study approximately 2.25 TB’s worth of trace data was obtained (creating data to study data).

In the study they examined the access, usage, and sharing patterns of the storage over a three month period. Using the collected data, they focused their analysis on three items:

1. Changes in file access patterns and lifetimes since previous studies
2. Properties of file I/O and file sharing
3. The relationship between file type and client access patterns

The authors divided their observations into two categories – (1) observations compared to previous studies, and (2) new observations.

Below is the list of results compared to the previous study (taken from the paper with some extra comments added).

1. Both of our workloads are more write-oriented. Read to write byte ratios have significantly decreased (from 4:1 to 2:1)
2. Read-write access patterns have increased 30-fold relative to read-only and write-only access patterns.
3. Most bytes are transferred in longer sequential runs. These runs are an order of magnitude larger (10x).
4. Most bytes transferred are from larger files. File sizes are up to an order of magnitude larger (10x).
5. Files live an order of magnitude longer (10x). Fewer than 50% are deleted within a day of creation.

The new observations reported in the study are:

1. Files are rarely re-opened. Over 66% are re-opened once and 95% fewer than five times.
2. Files re-opens are temporally related. Over 60% of re-opens occur within a minute of the first.
3. A small fraction of clients account for a large fraction of file activity. Fewer than 1% of clients account for 50% of file requests.
4. Files are infrequently shared by more than one client. Over 76% of files are never opened by more than one client.
5. File sharing is rarely concurrent and sharing is usually read-only. Only 5% of files opened by multiple clients are concurrent and 90% of sharing is read-only.
6. Most file types do not have a common access pattern.

One observation mentioned in the paper yet wasn’t listed in the two lists was the fact that overall file access was random, indicating the importance of random data performance of the storage medium.

a] Long-Term Digital Preservation Reference Model :  www.ltdprm.org

b] Information Lifecycle Management 2.0 (ILM2.0) Reference Model: www.ilm20.org

So, instead of lots of blog posts – jump over to either of these sites and participate in the reference model communities we’ve started there and contribute.

We have identified and are offering two important new audit services to our clients. Take a look and let me know what you think and if we can help you.

a] Preservation in the Cloud Assessment and Audit

b] IT and Information Governance in the Cloud Audit & Assessment

.

.

.

.

.

.

.

.

.

.

.