BI Tool Evaluation for the Modern SaaS Organization

by Will DeCesare, Data & Analytics Manager @ Resilia — Seemingly infinite vendors, each with slight to moderate variations on the same solution. Read to find out how Resilia navigated the process.

Resilia’s Profile

If one were to skim the Locally Optimistic Slack group, they would see similar company profiles ad infinitum. Resilia is not substantially different:

1. Industry: B2B SaaS
2. Stage: Series A; ~50 stakeholders; scaling as we grew our product offering
3. Stack: Snowflake, Segment, dbt, Fivetran, Hex, and Variance
4. Sources: application data on a Heroku-hosted Postgres datastore, Segment event stream, out-of-the-box CRM/payment processing/marketing vendors like Hubspot, Stripe, Facebook, and Google
5. Data Approach: a never-ending journey of becoming data-driven!

Resilia adopted Hex early on. Hex, while a great tool that allows for multiple programming languages, deep exploration, and storytelling, relies on technical acumen to produce analytics. And, as our business teams grew, we became bottlenecked at the source (*cough* yours truly *cough*).

Our Product and RevOps teams, maybe

Understanding the Problem

Resilia’s product team does a fantastic job of understanding the problem before jumping to the solution. We took the same approach in our evaluation for a new BI tool by fully fleshing out our problem statement:

1. How does our team currently consume data & analytics?
2. What are the shortcomings of the current solution?
3. What will our team’s needs be 3/6/12+ months from now?
4. What does the perfect solution look like?

We used datateer’s evaluation matrix as a template to answer these questions and prioritize tool features. We whittled down the most relevant criteria to the following fields:

1. Self-service

How easily can our RevOps manager automate revenue reporting and optimize LTV-to-CAC? How easily can our Product team understand user engagement and optimize PLG strategies through activation measurement?

2. Data Governance

Is the Data & Analytics team able to control data access roles and permissioning? At what depth? How easily can stakeholders understand the data that they consume?

3. Modularity

How strong is vendor lock-in? Does the tool overlap with our current stack?

4. Price

How does the price compare with other tools? What’s the incremental cost as we scale users and features?

5. Embed¹

Is it possible to embed analytics externally on the Resilia app?

Evaluating the Solution

We began our evaluation by identifying and reaching out to a few notable enterprise players in the industry: Looker and Tableau. Having experience with Looker, we didn’t expect that other vendors would compete from a product offering perspective. Looker checked all of our boxes — intuitive self-service, data governance (ie. LookML), efficient modeling/caching, strong product roadmap, and more.

We subsequently learned that we should expand our scope outside of the incumbents and began conversations with newer vendors like Preset, Lightdash, and others. The reason: price. We knew that the enterprise players, particularly Looker, would cost more, but we quickly realized that their pricing levels had changed since our last experiences with the tools².

Through numerous conversations, emails, and Slack chats with account executives, sales engineers, and fellow data professionals, we were able to fill the evaluation doc to near completion.

Our ability to stand up test instances was straightforward and smooth across all vendors. We were consistently impressed with the simplicity of securely connecting to our data warehouse and quickly visualizing data. While it was time-consuming to tackle a few mini-POCs, this approach allowed us to further drill down into each tool and get a feel for specific aspects of the application, such as the ability to self-serve and organizational framework.

Making the Decision

We eliminated Looker on the basis of price.

Similarly, we found Tableau’s product offering to be quite similar to Preset’s, at a higher cost and with a less favorable developer experience.

We had high hopes for Lightdash. Marketed to us as “Looker without the LookML”, it held true to its name. We valued the ability to define the semantic layer in dbt in order to codify and govern dimensions and metrics in one place. Similar to Looker, you signal to Lightdash joinable tables via metadata in dbt. We decided not to move forward with Lightdash because it was less developed than we needed it to be. Particularly, basic chart functionality, organization, and governance needed improvement before we felt comfortable using it as our main BI tool.

We ultimately decided on Preset. Preset is a hosted version of Superset, an internal Airbnb tool released into the Apache incubator and then graduated as an officially supported Apache project.

In time-to-value, Preset was on par with Lightdash. Preset gives the flexibility of modularity, where all (or no, if desired) metrics can be defined in the evolving dbt integration. We were confident in Preset’s data governance strategy by means of data access groups and layered certification. Lastly, they were reasonably priced relative to other vendors and follow a by-user pricing model that we will readjust on a quarterly basis.

We’re working with Preset to improve upon a few features, including:

1. Report organization: labeling (Hex) vs folder structure (Looker)
2. In-platform dataset creation: how can we improve the experience of smartly joining tables within Preset?
3. Column-level RBAC: ability to hide dimensions from specific data access groups

We’re excited to work with Preset as they continue to deliver on their product roadmap.

¹ A potential product use case in the future that we needed to include in the evaluation.

² To the Looker sales team’s credit, they amended the original proposal more than once.

Want to chat all things data and analytics? Reach out to Will at william.decesare@resilia.com