Big Data in a Data Driven Organization

Back when I started in BI, we used reporting tools to provide users with an interface into very static views of data. Over the years, we’ve seen users continuously want more information displayed in a single report. But that became very time consuming. While the report may have illustrated a point, the time to create it became less and less tenable.

Enter big data: we no longer had the time to synthesize a large swath of data and create a report/infographic that summarized all that detail. Also, with the detail summarized, we sometimes lost the point of the report itself. We needed a better way to advance through analytics that resulted in a number of custom application efforts to provide interactive experiences through the data. The challenge was that these experiences required substantial IT development time to produce even in small use cases. We had to cover concerns with enterprise security, rapidly evolving data types and processing frameworks, and an ever growing demand for analytics - all impediments for a data driven organization..

What Makes a Data Driven Organization?

At Zoomdata, we’re revolutionizing the way people interact with and consume data. Our customers want to do that, but sometimes it's difficult to see how that should change from what they’re used to. As we work with our customers, we have to constantly remind ourselves of three things:

1. Using data to make decisions is the end goal
2. Self-service access to data is critical to achieving that
3. Data freshness is as important as data volume

Imprisoned By Assumptions (Based On 20-Year Old BI Technology)

The other day, I had a meeting with the data architecture team of a large, traditional enterprise. And it was a good discussion, but it made me realize that we all need reminders sometimes as to why we are embarking on this big data journey. Because right now they’re imprisoned by the misconceptions and assumptions of their BI tool vendors who are telling them, “Oh sure, we connect to Hadoop. We’ll connect to whatever data source you have and build you a really modern, fast, and efficient backend.” But the problem is that what they’ll connect it to is 20-year old technology.

We did a demo for this company. We walked through our architecture--our typical sales conversation. And as soon as they understood the architecture and recognized what we can do with Zoomdata Fusion, the architecture team said, “Great, just connect us to Tableau!”

Now, Tableau is kind of like the Kleenex of BI tools. It could have been Qlik. Or Birst. Or Spotfire. Or Pentaho. But this company was very invested in Tableau.

All these tools were built to leverage cached or pre-aggregated data. They were built to do pivots in OLAP cubes with all this supporting infrastructure that’s really meant to deal with small data volumes of high-value, curated data.

But, in big data use cases, data comes at you so fast, it's not realistic to make a business decision to refine that data first before you even know its utility. Because often that data’s shelf life, it’s utility is very short.

Anyway, we paused the conversation and said, “Let’s put that aside for a moment. We’ll come back to that. But let’s look at what Zoomdata can do.” They saw the visualizations, the speed. And… they still said, “Our users are really tied to Tableau. How can we connect to Tableau?”

Changing the Conversation (Getting Big Data To LOB Users)

At that point, we had to remind them of what they wanted to achieve with big data and analytics. They wanted to become a data driven organization. They wanted their LOB users to have self-service analytics access to fresh data. They wanted them to have the ability to drill down to all the detail of the data source. And especially, they wanted to eliminate IT as the bottleneck between data consumers and their analytics. We got a big “YES” to all of that.

To which we replied, “Well, you can’t achieve any of that with Tableau. Even if we connected our intelligent backend and everything else we can do to Tableau.”

“Ohhhhhh, okay. So we should get you in front of our LOBs.”

So we finally got to where we need to for a follow up discussion. But we had to do that education first. Because that’s our focus: enabling the business to consume and utilize data in their day-to-day operations.

There’s a lot of attention paid to data science with big data, and rightly so. You can do a lot of good things with massive amounts of data using the tools of data science. But, from Zoomdata’s perspective, that's a different work stream. That's a business trying to extract greater utility out of its low-utility data for longer term use. To have people become big data driven in their day-to-day lives, you have to eliminate the latency from the data source to the consumer. Zoomdata provides our users with the ability to ask their questions of the most up to date data and drill-down into its lowest level of detail all with the fastest user experience. This is how we are changing the game for organizations and allowing them to become big data-driven.