Summary
Where: Confidence scores are presented in UA Recommendations, as part of the campaign card. (Highlighted in green below)
What: Confidence scores measure the noisiness in underlying data. The noisier the underlying data, the more variance there will be in a prediction. High confidence scores are better than lower confidence scores
How: In brief, when Ktrl predicts LTV, it calculates a distribution of predictions, and then compares the median and bottom 5th percentile prediction. The closer the bottom 5th percentile is to the median predicted LTV, the higher the confidence score of the prediction
So what: The higher the confidence score, the more certain a UA manager should feel that the D7 target presented, is the right signal to send to the ad network
Classifying risk
Confidence scores are bucketed into:
"High" when the score is above 60%
"Medium" when it is between 30% and 60%
The "Low" risk bucket covers two things:
Campaigns when confidence scores are below 30%
Any non-classical networks such as incentivised and rewarded networks or app store ads
Tips and warnings
⚠️ Recommendations can shift as new actuals update cohort forecasts, especially for early campaigns or volatile geos. Small changes in cohort quality, IAP vs IAA, and organic vs paid ratios can meaningfully affect results
⚠️ Recommendations should be used as directional input, not exact targets. Since Ktrl does not have access to each ad network’s live optimisation ROAS, UA managers should adjust proportionally
⚠️ If Ktrl recommends increasing the D7 target ROAS from 10% → 15%, but the ad network already uses 17%, the UA manager can adjust directionally - e.g. testing an increase from 17% → ~26% and monitoring performance
Advanced information
Why you need confidence scores
Confidence score reflects the noise in the data. This is why the median and 5th percentile prediction will be different
Think of the expected rain in two locations:
London, England: Historically between 10 and 100mm of rain on a given day in winter. Tomorrow, a model predicts 25mm of rain
Cape Town, South Africa: Historically between 10 and 30mm of rain on a given day in winter. Tomorrow, a model predicts 25mm of rain
Here, the London prediction would have a lower confidence, because on any given day, the range of possible values is higher.
Similarly, confidence scores allow a UA manager to contextualise a UA recommendation. While following a UA recommendation may be the best course of action a UA manager can do, it explains the risk associated with it. Low confidence is not always a bad thing: sometimes an ad network may have a higher variance in underlying performance and that is what makes the ad network's traffic unique.
How confidence scores differ from accuracy
Confidence scores are not the accuracy of a model. Accuracy is how correct a model is. It is measured via back tests where a model predicts and then compared to actuals, for a past period.
In the above examples, both the London and Cape Town model can be 99% accurate - e.g. they predict 99 days correctly for their locations, out of 100. However, as the London data has a higher spread - 10 to 100mm - it will have a lower confidence score.
Managing downside risk
As the median is compared to the bottom 5th percentile, Ktrl's confidence scores measure the risk that actual performance falls below expectations. This helps you understand how much cushion you have in your predictions. A high confidence score means there's low risk of underperforming. A low confidence score signals higher uncertainty - the actual LTV could end up lower than predicted, which means you need to be more cautious with taking actions.