Unsuccessful hypotheses: when promising ideas don’t reach their potential

Development is not always a straightforward process. Sometimes, even the most promising mechanics and features can result in negative outcomes for various metrics. However, this is a normal part of the learning process, and it’s crucial to rely on thorough testing before rolling out a new version to all users.

In this article, we’ve gathered a few examples of such cases.

The meta-game that crashed the metrics

Punch Bob is a hyper-casual game where players throw their characters at opponents and solve simple physics puzzles.

The game is set in a fighting world with wrestlers, and to match this theme, we introduced a meta-game in the form of a gym. Players could earn a special currency in the main game and use it to unlock the ‘kill a random opponent’ ability in the gym, which could be further improved. On paper, the idea seemed simple and fitting.

Despite our expectations of increased retention, the update to Punch Bob’s meta-game resulted in a decrease in all metrics. Our average LTV decreased by 15%, retention by 22%, playtime by 10%, and interstitial and rewarded ad views by 15%.

We identified the poorly configured balance as the primary issue. The meta-game was too easy, and players could quickly exhaust its content without sufficient motivation to return to the main game.

As a result, we removed the feature and have not revisited it yet. However, we may explore it again in the future with a better-balanced approach.

Bosses that lowered retention

We also tried to increase the challenge and retention metrics for Punch Bob by adding boss levels. The bosses were tougher than ordinary opponents, requiring players to hit them three times to win.

Unfortunately, the new bosses didn’t have the expected impact. Retention only slightly decreased, the LTV dropped by 5%, but playtime increased by more than 20%. Despite this, we removed the bosses, because we also noticed a decrease in ad views.

City building that distracted from core gameplay

State Connect is a hyper-casual game that challenges players to connect cities on a world map with roads. For more information, you can read our big article.

We decided to add an additional layer of gameplay to it. We introduced a new mode where players could rebuild a city using a separate currency and receive unique car skins to use in the main game.

We rolled out the new feature to 25% of our users, and the overall result was mostly negative. However, there were some positive aspects as well.

Unfortunately, we made some mistakes in the development of the meta-game that were essentially bugs. For instance, players were not able to accumulate gold in the main game while in the city, which led to retention decreasing by 10% and a 20% drop in rewarded ad impressions.

On the other hand, playtime increased by more than 25%. Players enjoyed the city-building aspect and exploring the new feature.

Although it took significant time and effort to create the meta-game, we recognized its potential and decided not to abandon it completely. Instead, we plan to rework the feature, adjust the balance, fix any obvious errors, and run additional tests to see if we can improve its performance.

Clicker mechanics that didn’t catch on in another genre

The hypothesis of introducing clicker mechanics to improve retention and playtime in State Connect was implemented by allowing the player to tap the screen to accelerate the movement of cars and resource accumulation in addition to connecting dots on the map.

The results showed a 30% increase in playtime, but a decrease in rewarded ad impressions by 40%, and a drop in retention and LTV by 5-10%. Players also used the upgrade buttons less, as tapping provided a quicker way to earn coins and level up.

As a result, the clicker mechanics were removed, but the team plans to revisit it in the future with a better balance that encourages players to use both tapping and the upgrade buttons without leveling up too quickly.

Final thoughts

It’s important to acknowledge that even the most promising and seemingly foolproof hypotheses may not work out. What may have been a decisive feature in one project, may not meet the expectations of the audience in another, leading to a negative impact on metrics. Therefore, relying on thorough testing is crucial.

In the long run, such failures can be seen as a learning experience that helps to reduce the number of negative iterations in the future.