It’s about appropriately measuring Customer Engagement & User Engagement…
With the growth of the SaaS industry, more and more Product Management teams are wondering as to how they can measure a very important aspect, and potentially the strongest leading indicator of success for SaaS Platforms: User Engagement.
User Engagement refers to a set of metrics that can track how many users are actually interacting with your Mobile App or SaaS Platform and to what extent. As a Stakeholder, Product Manager or a CEO of a tech company revolving around a product, it is imperative to keep an eye on actual usage of the product, at a customer as well as an end-user level. Both lead to insights that can help course correct (while you still have time), or create additional options of growth to thrive in an ever-increasing competitive business environment.
The goal of a SaaS or Mobile App CEO should be to increase the LTV (Lifetime Value) from each customer. It is obvious that measuring customer engagement & user engagement is a key tool that will help them achieve that goal. By getting a true measure of engagement, actionable decisions can be made about:
- How to identify customers/users who are about to churn?
- How to identify customers/users appropriate for a cross-sell or upsell?
- How to identify features being heavily used, and ones that are not getting traction (and why)?
- Which factors are potentially causing improvement or deterioration of engagement?
A glimpse into the history
The need for measuring engagement is not new. For the last decade, several studies and reports have been available in the market. Few of them that are worth noticing are:
- A user engagement study by Lehmann et al, 2012 noticed users frequented tech products with different frequencies and suggested the following user categories based on the number of days per month they used the product.
2. Similarly, a study focused on branded mobile phones by Kim, Lin & Sung, 2013 noticed that most of the tech products incorporated at least one of the seven Engagement Attributes: Control; Customization; Vividness; Multi platforming; Motivation; Feedback; Novelty.
3. Another study by O’ Brien, 2015 concluded that there were 3 Primary information sources for measuring user engagement, with different characteristics:
- Self Reported: They pertain to questionnaires, surveys, interviews, app reaction cards, thinking aloud.
- Cognitive Engagement: They pertain to the factors like time spent, follow on tasks, neurological measures (EEG) or physiological measures (eye-tracking, heart rate, etc.)
- Interactive Engagement: They are related to app data analytics, regression & data science.
4. Finally, a study by [Peterson et al, 2016] reported that Tech App Engagement is best measured as a consolidation of 8 Indices: Duration index, Loyalty index, Retention index, Feedback index, Performance index, Recall index, Social index, and Interaction index.
Do solutions for measuring engagement exist?
While the value of being able to measure engagement has increasingly been felt by most Product Owners, Managers, and IT Professionals, it has been tough getting hold of a solution that really delivers. In July 2018, Gartner published the following report that lists some solutions that can help you measure engagement. You can download the guide through this link: Market Guide for Web and Mobile App Analytics.
What is missing in the solution above?
While the solutions above help in getting some idea about the engagement levels, after trying them out, we figured that there were a few gaps. Each of these solutions was providing a bunch of measures or metrics, without helping put them together into a consolidated and meaningful product engagement score.
Also, engagement is a moving target. What engages customers and users today fails to keep them interested in a couple of quarters down the line. Hence, the algorithm to measure this needed to be flexible enough to add or modify parameters as and when required.
We were looking for a solution that could give us a consolidated product engagement score, and then provide insights so that we can drill down on each of those parameters for further analysis. We also wanted to see if we could build a solution that could help influence people engagement positively, and then be able to track if the actions taken actually led to an enhancement of engagement.
We were also looking for a solution that could help us understand how we were doing on engagement, relative to other Apps or SaaS Solutions, and not always as an absolute. And finally, we were hoping that with ML and AI being built into the solution, over time regression would help polish the algorithm and fine-tune the output for even better results.
What we ended up creating:
Based on our research on the history of engagement processes, and the gaps we found in them, we designed our own solution to measure all the aspects of a SaaS customer engagement process that a subscription-based business needs. This product engagement score is designed around 4 indices based on which a holistic insight can be obtained. They are:
- Duration Index
- Retention Index
- Performance Index
- Loyalty Index
Let’s look at them individually.
- Duration Index tracks app usage. It shows the median time (in hours) spent by a user on the platform per day, and also the usage trend, by comparing the length of sessions or time spent, every 15 days. As a product company, one needs to carefully watch app usage and trends, and this index lets one do exactly that.
- Retention Index measures regularly active users through cohort analysis. It measures the percentage of users who continue to revisit the app and use the platform over time. Each of these pre-defined intervals gets a weighted score, which is then used to calculate the composite score for retention. Calculations at shorter intervals than a week would lead to skewed scores. Hence, our index calculation accounts for data available longer than a week.
- Performance Index is a composite of two key sub-parameters – the average time for a page to load, and the percentage of successful items loading for that page. Both affect user retention in today’s digital economy thriving on accessibility and immediacy. The latter is particularly relevant for pages that see high request volumes, giving the business a clearer view of how different pages per section of their app perform. Performance Index is replaced by Rating Index for mobile apps, where we pick up ratings from the Mobile App Store if it is available in the public domain.
- Loyalty Index measures the overall continuity of usage of the platform’s features. It is calculated based on the number of users who are dropping out (churn) which is measured by the lack of logins for a period longer than a month. The index also accounts for the type of app (Mobile or Web) to create an appropriate benchmark.
Conclusion:
- Customer or user engagement is the lifeblood of every SaaS & Mobile App based business because these business models are based on retention.
- If people don’t use your product, they will stop paying for it. Hence, tech subscription businesses simply must keep their customers paying month-over-month for a long period of time in order for the model to work.
- All product teams are building features and user experiences with the goal of driving world-class user engagement. To know if their cost to the company is justified, there must be a consistent way to measure engagement at the product level over time.
- Even given its fundamental importance to the entire business model, the vast majority of SaaS businesses are not measuring user engagement in any systematic or quantifiable way simply because quantifying user engagement is not easy.
- When we talk about quantifying user/customer engagement, we’re talking about creating a model for scoring the engagement of each customer based on varying but relevant parameters.
- We also talk about trying to ensure that this measure is relative and not absolute, wherever the business model allows for it.
While this article is by no way exhaustive, we hope it is enough to showcase how important quantifying user engagement is for your business. It is much more than a simple metric. It is something that can have a significant impact on business processes across an entire organization that is based around a Mobile App or SaaS platform. In fact, we strongly believe it might just be the MOST important metric for any SaaS business.
We developed the Product Engagement Score to make it easy for all SaaS businesses actually quantify user engagement—and integrate it with all their essential business functions. We would love for you to check it out. You can start your 17-day trial today to gauge your product engagement score!
Before you go
We are the SmartKarrot Team. We design, we build, and we derive fun out of solving problems in the Venn overlap of tech and real life. Thanks for reading!
Surojit has over 15 years of experience in quality and implementations. He is a promoter of an extremely light and efficient Agile process to fit business needs. In his prior role as product owner, he built a robust product in a very short span of time.
Published May 13, 2020, Updated January 06, 2021
