Big Data Life Cycle

GUPTA, Gagan       Posted by GUPTA, Gagan
      Published: June 28, 2021
        |  

Enjoy listening to this Blog while you are working with something else !

   

Data within organizations - measured in petabytes - grows exponentially each year. As data is generated, it moves from its raw form to a processed version, to outputs that end users need to make better decisions. The five big data lifecycle stages include data ingestion, data staging, data cleansing, data analytics and visualization, and data archiving. All Big data goes through this lifecycle. Organizations can use services from multiple vendors in each stage of the data lifecycle to quickly and cost-effectively prepare, process, analyze, and present data in order to derive more value from it. Simpler data analytics, cheaper data storage, advanced predictive tools like machine learning (ML) and data visualization are necessary to make data-driven decisions and maximize the value of data. The Big Data Lifecycle helps organizations of all sizes establish and optimize a modern data analytics practice in their organization.

Data Generation or Source

For the data life cycle to begin, data must first be generated. Otherwise, the following steps can't be initiated.

Data generation occurs regardless of whether you're aware of it, especially in our increasingly online world. Some of this data is generated by your organization, some by your customers, and some by third parties you may or may not be aware of. Every sale, purchase, hire, communication, interaction-everything generates data. Given the proper attention, this data can often lead to powerful insights that allow you to better serve your customers and become more effective in your role.

Common data sources include transaction files, large systems (e.g. CRM, ERP), user-generated data (e.g. clickstream data, log files), sensor data (e.g. from Internet-ofThings or mobile devices), and databases.

Data Ingestion

Not all of the data that's generated every day is collected or used. It's up to your data team to identify what information should be captured and the best means for doing so, and what data is unnecessary or irrelevant to the project at hand.

It's important to note that many organizations take a broad approach to data collection, capturing as much data as possible from each interaction and storing it for potential use. While drawing from this supply is certainly an option, it's always important to start by creating a plan to capture the data you know is critical to your project.

Data ingestion entails the movement of data from an external source, into another location for further analysis. Generally, the destination for data is some form of storage or a database. For example, ingestion can involve moving data from an on-premises data center or physical disks, to virtual disks in the cloud, accessed via an internet connection. Data ingestion also involves identifying the correct data sources, validating and importing data files from those sources, and sending the data to the desired destination. Data sources can include transactions, enterprise-scale systems such as Enterprise Resource Planning (ERP) systems, clickstream data, log files, device or sensor data, or disparate databases. During data ingestion, high value data sources are identified, validated, and imported while data files are stored.

Key questions to consider: What is the volume and velocity of my data?

Typical Tools used in the process: Kafka, Sqoop, Storm, Kinesis, Flume, NiFi, Gobblin etc...

Our On-Premise Corporate Classroom Training is designed for your immediate training needs
Big Data Life Cycle
Big Data Life Cycle

Data Staging

Data staging provides the opportunity to perform data housekeeping and cleansing prior to making the data available for analysis. One of the most common challenges is that data is housed in multiple systems or locations, including data warehouses, spreadsheets, databases, and text files. Cloud-based tools make it easy to stage data or create a data lake in one location, while avoiding disparate storage mechanisms. Not only is the variety expanding; its volume, in many cases, is growing exponentially. Add to that, the complexity of mandatory data security and governance, user access, and the data demands of the analytics.

Key questions to consider: Which use cases is the organization looking to address with the data?

A data lake is an increasingly popular way to store and analyze data that addresses the challenges of dealing with massive volumes of heterogeneous data that is queried by multiple users within the organization. A data lake lets you store data as-is; there is no need to convert it to a predefined schema, allowing you to store all of your data - structured or unstructured - in one centralized repository.

Some tools to integrate your Data Lake tools are : Google Bigquery, Snowflake, Teradata, RedShift etc...

Data Cleansing

Before data is analyzed, data cleansing detects, corrects, and removes inaccurate data or corrupted records or files. It also identifies opportunities to append or modify dirty data to improve the accuracy of analytical outputs. In some cases, data cleansing involves translating files, turning speech files to text, digitizing audio and image files for processing, or adding metadata tags for easier search and classification. Ultimately, data cleansing transforms data so it's optimized for code (e.g. Extract, Transform, Load (ETL)).

Key questions to consider: Which tool to use for the ETL function?

There are plenty of tools to help you with data cleasning: SAS Data Quality, OpenRefine etc

The cleansing must be far more thorough for a BI analyst, than for a data scientist - who might value under-prepared data for its nuanced flavor.

Data Analytics & Visualization

Data analytics and visualization is the stage of the lifecycle where the data preparation pays off in the form of actionable results for the organization. Where, on the one hand, analytics involves generating results from the data; visualization is about exploring data and communicating results from analysis to decision-makers. The real value of data can be extracted in this stage. Decision-makers use analytics and visualization tools to predict customer needs, improve operations, transform broken processes, and innovate to compete. The ability for mission owners and executives to rely on data reduces error-prone and costly guesswork. Some of the more commonly used methods include statistical modeling, algorithms, artificial intelligence, data mining, and machine learning.

Key questions to consider: Who are the consumers of the data at this stage within the organization?

Data visualization refers to the process of creating graphical representations of your information, typically through the use of one or more visualization tools. Visualizing data makes it easier to quickly communicate your analysis to a wider audience both inside and outside your organization. The form your visualization takes depends on the data you're working with, as well as the story you want to communicate.

While technically not a required step for all data projects, data visualization has become an increasingly important part of the data life cycle.

Tools here are aplenty: Tableau, SAS, Google Charts, PowerBI, QlikSense, R programming, Jupyter, Python etc.

Our On-Premise Corporate Classroom Training is designed for your immediate training needs

Data Archiving

After data has been collected and processed, it must be stored for future use. This is most commonly achieved through the creation of databases or datasets. These datasets may then be stored in the cloud, on servers, or using another form of physical storage.

When determining how to best store data for your organization, it's important to build in a certain level of redundancy to ensure that a copy of your data will be protected and accessible, even if the original source becomes corrupted or compromised.

For the data archiving or re-use stage, use a service that makes the process easy to manage, and thus allowing organizations to focus on the storage of data - rather than on managing tape systems and libraries. Think of numerous compliance standards, security certifications, and data encryption.

Key questions to consider: Will the organization archive data for analytics or compliance?

Toosl for data archiving are : Hadoop, HDInsight, PolyBase, Hitachi HSP etc.

Benefits for Data Lifecycle Management

Big Data management differs from traditional data management primarily due to the volume, velocity and variety characteristics of the data being processes. To address the distinct requirements for performing analysis on Big Data, a step-by-step methodology is needed to organize the activities and tasks involved with acquiring, processing, analyzing and re-purposing data. While it is certainly not the only way that data can be managed, all efforts should be made so that data is available as quickly and effortlessly as possible to decision and policy makers. The benefits may include more effective marketing, new revenue opportunities, customer personalization and improved operational efficiency. With an effective strategy, these benefits can provide competitive advantages over other market players or rivals.

Conclusion

There are numerous big data life cycles to choose from. Most communicate the same basic steps necessary to deliver a successful project but often have a distinct angle. Regardless of the life cycle you use, combine it with a collaboration process so that your team can effectively coordinate with each other and stakeholders. You can customize the process based on the structure of the data solution team and the overall enterprise ecosystem.

From a Big Data adoption and planning perspective, it is important that in addition to the lifecycle, consideration be made for issues of training, education, tooling and staffing of a data analytics team. There is no doubt that big data is changing how people live their lives in ways that they could never have imagined. Good luck, it is going to be fun !

Support our effort by subscribing to our youtube channel. Update yourself with our latest videos on Data Science.

Looking forward to see you soon, till then Keep Learning !

Our On-Premise Corporate Classroom Training is designed for your immediate training needs
Big Data Life Cycle
                         



Latest Blogs