Context
It’s often the case that when improving a digital product, there’s an understandable reluctance to introduce big technical improvements as such changes come with a risk of destabilizing the existing product.
There’s a solution to that risk, and we’ll show how we handled it for a B2B chat product for emerging markets.
Problem Identification and Consequences
In this product, the ability to send attachments like photos, videos, and files is crucial. However, the technical backbone supporting this feature in our application was outdated and lacked offline capabilities for sending attachments. This posed a significant risk as the existing solution, while stable, needed an update to meet modern needs.
The challenge was deciding whether to enhance the existing solution or implement a significant change by integrating Android's WorkManager, which supports task scheduling until internet connectivity is restored. The decision was fraught with risk due to the imprecise documentation and previous stability issues of WorkManager. The dilemma was whether the potential benefits of adopting a modern, automated solution outweighed the risks associated with changing a stable system in a critical part of the application.
Solution Implementation
To mitigate the risks associated with implementing new technical changes, we utilized A/B testing—a method commonly applied in UI/UX validation—to assess the new technical implementation:
- Variant A maintained the current, stable implementation.
- Variant B implemented the new solution using WorkManager.
Key metrics for the A/B test included the success ratio of sending an attachment, the number of attachments sent per user, the percentage of users who successfully sent at least one attachment, and the crash ratio. This method allowed us to objectively measure the impact of the new implementation against the existing one without affecting the overall user experience.
Business and Product Gains
After two weeks of testing, we saw that:
| Variant | A | B | Conclusion |
|---|---|---|---|
| Success ratio for sending an attachment | 88% | 94% | Increase of 6.74%. Statistically significant with 99% probability. |
| Number of attachments sent per person. | 1.83 | 1.96 | Increase of 7%, which is close to the previous result. The result is statistically significant with 99% probability. |
| Percentage of users who successfully sent at least one attachment | 9.09% | 9.11% | Statistical significance cannot be demonstrated in terms of the number of users sending attachments with either version due to the sample size being too small to discover statistical significance in both versions. |
| Crash ratio | Baseline | -0.078% | Statistically insignificant decrease in crash ratio. |
These results validated the effectiveness of the new implementation, leading to a decision to roll out WorkManager to all users, thereby enhancing the functionality without increasing technical debts or user disruption.
The A/B testing framework thus proved essential in reducing the risks associated with technical changes, providing a reliable basis for decision-making based on data rather than assumptions. This approach ensured that the new functionality was not only theoretically better but also proven in a live environment.
