David A. Steinberg On Managing Goliath: Handling Big Data

Massive data warehouses are cropping up across the internet landscape, and one of the most current trends in business is finding ways to perform ‘deep analytics’ on the mountains of data contained in the virtual storage lockers.

Once you have found a subset of information that you want to analyze, what comes next? There are some basic guidelines for performing data analysis to make sure you get the information that will help you maximize your profits and online presence.

1. Get the team onboard

Make sure the players in your organization understand the value of Big Data. Delving into this information can offer insights to help gain competitive advantage. Some believe Big Data may be the last online opportunity to do this. The more data, the more detail.

2. Choose your flavor

Decide what kind of analytics you need the most. Generally there are two different kinds: exploration and analysis, or prediction and optimization.

Exploration and analysis involves a business analyst acting as a virtual Sherlock Holmes. In this scenario, someone notices a trend, and wants to find out why it is happening. The analyst forms a theory, and culls through the data to see if it is correct or to see if alternative theories are suggested or supported.  This is a more traditional kind of data analysis and it is still useful and popular today.

Prediction and optimization flips the data upside down. Here, analysts look to see what trends are currently trending  in order to predict where the trends will go. Historical data is mined and analyzed with statistics to see patterns. These patterns are then used to predict possible future events. But, these tools are not infallible. The analyst needs to have a solid knowledge of business practices and data to really use these strategies effectively.

3. Look at your own database

Instead of downloading data to servers that cannot handle big volumes of data, many companies now manipulate the information in the database itself. This lets companies run queries against all the data, and not just smaller portions of it.

4. SQL is not the only choice

SQL requires a lot of IT management time, and gets expensive for large amounts of data. New software is available to allow business analysts to add analytic functions to databases rather than have the programmers do it. This saves time and money.

The Final Word

Essentially, know how much you want to do, and how much you can do in order to decide what you need to do.  Many smaller analyses can easily be done in house by an IT programmer with self-collected and stored data.

More complex analyses would probably benefit from the in-database option, so sophisticated metrics can be applied as needed.  This may cut down on costs and wait times. Big Data has incredible potential, but it can become an overwhelming mess of information if not handled correctly. With some carefully considered steps, your company can gain the information and edge you need in your online environment.

David A. Steinberg is CEO of Zeta Interactive, a leading big data company.