INDTRODUCTION
There is no set number of gigabytes or terabytes or petabytes that separates big data from average-sized data.In addition, every organization is different, so the amount of data that seems challenging for a small retail store may not seem like a lot to a large financial services company.Data stores are constantly growing so what seems like a lot of data right now may seem like a perfectly normal amount in a year or two.
BIG DATA INTERESTING IN BUSINESS
. Big Data is allowing the leading firms like IBM, Amazon, to develop some of the cutting-edge technologies providing high-end services to their customers Needless to say that Big Data is taking the world by storm through its countless benefits.
BIG DATA CHALLENGES
Generating insights in a timely manner
Dealing with data growth
Integrating disparate data sources
Recruiting and retaining big data talent
Securing big data
Organizational resistance
Validating data
GENERATING INSIGHTS IN A TIMELY MANNER
”Organizations don't just want to store their big data, they want to use that big data to achieve business goals.only if they can extract insights from their big data and then act on those insights quickly.
DEALING WITH DATA GROWTH
The”Digital Universe report, IDC estimates that the amount of information stored in the world's IT systems is doubling about every two years.The most obvious challenge associated with big data is simply storing and analyzing all that information.
INTEGRATING DISPARATE DATA SOURCES
The variety associated with big data leads to challenges in data integration. Big data comes from a lot of different places enterprise applications social media streams, email system employee-created documents etc. Combining all that data and reconciling it so that it can be used to create reports can be incredibly difficult. Vendors offer a variety of ETL and data integration tools designed to make the process easier, but many enterprises say that they have not solved the data integration problem yet.
RECRUITING AND RETAINING BIG DATA TALENT
”In order to develop, manage and run those applications that generate insights, organizations need professionals with big data skills. That has driven up demand for big data experts, and big data salaries have increased dramatically as a result.
SECURING BIG DATA
Security is also a big concern for organizations with big data stores..Most organizations seem to believe that their existing”data security methods”are sufficient for their big data needs as well. After all, some big data stores can be attractive targets for hackers or Advanced Persistent Threats (APTs) In the IDG survey, less than half of those surveyed (39 percent) said that they were using additional security measure for their big data repositories or analyses.
ORGANIZATIONAL RESISTANCE
It is not only the technological aspects of big data that can be challenging ‘ people can be an issue too.In the New Vantage Partners survey, 85.5 percent of those surveyed said that their firms were committed to creating a data-driven culture, but only 37.1 percent said they had been successful with those efforts.”
VALIDATING DATA
Closely related to the idea of data integration is the idea of data validation. Often organizations are getting similar pieces of data from different system and the data in those different systems doesn't always agree. For example, the e-commerce system may show daily sales at a certain level while the Enterprise Resource Planning (ERP) system has a slightly different number. Or a hospital's Electronic Health Record (EHR) system may have one address for a patient, while a partner pharmacy has a different address on record.
REDUCE TIME TO MARKET
Introducing the new products or services involves many life cycle stages, some of which are easier to accelerate than others. For the past two decades, drug manufacturers have been using clinical trial simulations to speed learning, reduce costs, and the limit unnecessary burdens on patients participating in the trials. Armed with the power of the cloud and big data, the clinical trial simulations can be done faster in ways that benefit the manufacturer and patients.
Bristol-Myers Squibb”reduced the time it takes to run clinical trial simulations by 98 percentage by extending its internally hosted grid environment into the”Amazon Web Services(AWS) cloud. The company has also been able to optimize dosing levels, make drugs safer, and require fewer blood samples from clinical trial patients.
Because clinical trial data is highly sensitive, Bristol-Myers Squibb built a dedicated, encrypted Virtual Private Network(VPN) tunnel to the Amazon gateway and configured a virtual private cloud so the environments would be isolated from public customers.
Before moving into the cloud, scientists were using a shared internal environment, so it took 60 hours to run hundreds of jobs. Now that each scientist has a dedicated environment 2,000 jobs can be processed in 1.2 hours without causing an impact to other members of the team.
As a result of the move, Bristol-Myers Squibb was able to reduce the number of clinical trial subjects in a pediatric study from 60 to 40, while shortening the length of the study by more than a year.
Conclusion
The availability of Big Data low-cost commodity hardware and the new information management and analytic software have produced a unique moment in the history of data analysis. The convergence of these trends means that we have the capabilities required to analyze astonishing data sets quickly and the cost effectively for the first time in history. These capabilities are neither theoretical nor trivial. They represent a genuine leap forward and a clear opportunity to realize enormous gains in terms of efficiency, productivity, revenue, and profitability.
The Age of Big Data is here, and these are truly revolutionary stages if both business and technology professionals continue to work together & deliver on the promise.