What is Dark Data and How Organizations can Draw Value out of it?

 Organizations rely on different data sources to capture information for making smart business decisions. A lot of the information gathered is for compliance purposes. Many organizations have discovered that they not just lack the right policies to capture the data but also lack a robust technology infrastructure to manage and understand the data. 

Over the past couple of decades, Data has grown in volume and type, which has forced organizations to finally address the issue of dark data. This excessive amount of data has not just increased the storage cost but continue to remain unutilized.

What is Dark Data?

Contrary to what the name suggests, there is nothing dark in dark data neither it is scary. Organizations collect a vast amount of data to make logical decisions for their benefit but most of the collected data is never used for making a business decision, and this unutilized data is known as dark data.

Where does Dark Data come from and what is its type? 

Dark data could be found in log files, data archives, website log files, emails, etc. of an organization. Data is very similar to the iceberg where the visible part is the data that is being utilized whereas the data that is submerged and is invisible is the dark data.

Dark Data is usually categorized into two different types. Let’s try and understand each type with an example 

Type 1: For example, let’s take chat messages or customer emails, the content in the message can turn into dark data if the organization doesn’t extract the meaning from the message in a way that the data analysis tools can analyze it.

Type 2: The metadata which comes along with the chat message or from the customer’s emails like the time at which it was sent, sender name, receiver name, device used to send it, location, attachments (if any), etc. become dark data when the email or message gets archived.

Data for both the types reside in the databases but they are not used to derive any insights. It is stored in the database so that it can be retrieved in the future if required.

Real-Life Example

One of the restaurants of a famous food chain wanted to identify the reason behind the decreasing footfall. Any restaurant will typically try and collect feedback on the quality of food, the quantity of food, pricing, presentations, taste, ambiance, service, etc.

There is a good chance that the primary reason behind decreasing footfall in the restaurant is due to the limited or no parking facilities. Information about the limited and no parking facility was always there with the restaurant but they never used it to identify the problem. This kind of data that is available with the organizations which they never consider to look at for any query is referred to as “dark data”.

Is Dark Data available only in the unstructured data? 

Dark Data can be there in both Structured Data as well as in Unstructured Data. Often unstructured data becomes dark data as organizations don’t know how to analyze the data to get the insights. However, structured data could also be a part of dark data. When data is stored in the structured format in the database but it is not being used by the organization to obtain the insights in that case stored data becomes dark data.

Problems Associated with Dark Data

Organizations often capture much more data than they are capable of. Most of the captured data stay in the dark because most organizations do not have the required tools and capabilities to process the data efficiently. 

“According to IDC, organizations fail to analyze 90% of the unstructured data.”

Most organizations don’t have access to tools that can manage and utilize all the captured data. It’s being observed that most organizations want to capture as much data as they can but they don’t have enough resources to analyze all the captured data. Organizations are looking for tools that can look inside their data and can reveal insights that can provide them with a business advantage.

How can we leverage Dark Data?

Drawbacks of storing dark data are often more than their benefits. Lack of data security associated with dark data could even lead to cyber-attacks, non-compliance issues, etc. 

The best way to tackle dark data is by utilizing it well. It may not be easy for most of the organization to utilize all the captured data as it requires both the considerable investment of time and the money. There are some ways with the help of which organizations can reduce/use most of their dark data.

• Organizations should regularly audit their databases. They should eliminate such data points which are not useful for them this will eventually save a lot of space.
• Organizations should even try to keep their data in a structured format. 
• Even if the business decides to dump dark data even then they should keep the data encrypted and in a secure manner.
• Organizations should label their unstructured data so that it is easy for them to find in the future for analysis.
• Organizations should have their data retention and data disposal policies in place so that data can be retained and disposed of with ease. 
• Organizations should use Artificial Intelligence tools as they have capabilities to make documents discoverable through search. AI has the ability to crawl through the data to understand and classify them automatically. 

Conclusion

Dark Data represents unused opportunities that organizations are unable to utilize because of the investment and technology constraint. The investment required to deal with dark data is costly but the outcome is worth the investment made. If organizations opt to sit on the dark data and do nothing about it then it could eventually lead them to several risks like cyber-attacks. The key is to do something about the dark data rather than treating it to use fewer data.

About Girikon

About Author

Amit Srivastava

Amit carries a rich experience of more than 9 years working as a Salesforce business analyst/customer success manager. His excellent analytical, and logical reasoning abilities makes him extremely desirable for successful management of projects, client success management and operations management.