News from Triton Consulting

Drowning in data
As Information Technology becomes ever more prevalent in nearly every aspect of our lives, the amount of data generated and stored continues to grow at an astounding rate. According to IBM, worldwide data volumes are currently doubling every two years. IDC estimates that 45GB of data currently exists for each person on the planet: that's a mind-blowing 281 Billion Gigabytes (281 Exabytes) in total. While a mere 5% of that data will end up on Enterprise data servers, it is forecast to grow at a staggering 60% per year, resulting in 14 Exabytes of corporate data by 2011.
This article takes a look at some of the reasons behind this data explosion, and some of the possible effects if the growth is not managed. We'll also examine some of the ways in which these problems can be avoided.

Industry trends
A major trend over the last few years has seen many organisations implementing ERP and CRM solutions. This in turn has caused a dramatic increase in the amount of data we are storing about our customers, prospects, partners and suppliers.

Companies are also investing in ever more sophisticated business intelligence and analytics. In an increasingly competitive marketplace, the ability to base business decisions on solid, reliable and timely management information is becoming a key differentiator, but trend analysis can require very large amounts of historical data to be stored and managed.

The trend towards company consolidation is not a new one, but the current economic situation has inevitably resulted in a significant increase in the number of mergers and acquisitions. This is creating a huge increase in data volumes, with the associated data duplication and application retirement issues. Organisations are faced with not only managing all of their own data, both historic and current, but also this influx of additional data from other parties. Imagine the "data headache" of combining all of the ERP, CRM, Business Intelligence and Analytic systems from different organisations into one manageable enterprise system.

Legislation
Corporate compliance legislation has had a major effect on how we use, store and maintain our data. The requirements placed on organisations by HIPPA, SOX, Basel ll and others mean that many companies are having to keep hold of more data, and for longer periods. Just as importantly, that retained data rapidly transforms from a corporate asset to a liability once the legal minimum retention period has expired, making it vital that such data can be accurately identified and deleted.

It is vital that organisations adhere to this legislation in order to avoid the cost of court appearances, heavy fines and the resultant damage to the brand.

Technical trends
New capabilities within the databases used to store corporate information are another major driver of data growth. For example, DB2 now supports XML and LOBs ("large objects" such as audio, video, images, etc). The ability to store this kind of data alongside more traditional structured information can be very useful, but can also have a huge impact on the overall size of the database.

Other technical trends that are contributing to database growth include storage of data in Unicode format (which can often expand overall database size by 10%-50% depending on the data), and duplication of databases due to replication requirements and/or backup strategies.

Finally, there's the perennial problem of removing old or obsolete data once it has reached the end of its useful life. Application data archiving is often considered as an optional extra, and even if it is included in the initial project plan it is often the first item to be postponed to a later release.