We have developed our own proprietary methodology for evaluating the effectiveness of certain geo-behavioral push notification messages.
For any message which recommends a course of action such as the recipient should visit a certain place of interest (perhaps at a certain time, say for an Event), then, subject to certain minimum qualification criteria, we are able to evaluate in near-real time how well a message has performed.
We’ve called this the ‘R4 score’ because we assess a message on four measures:
Reach – The total target group size, i.e. the number of devices that are sent the geo-targeted push notification.
Relevance – The percentage of the Reach group for whom the message is relevant. For recommendations to visit a certain region (during a certain time period, if relevant) this is the group that were located at that region (during that specified time period, if relevant), either because they would have been anyway, or because they were encouraged to go by the message.
Reaction – The percentage of the Reach group that, based on our methodology, we believe modified their behaviour because of the message, i.e. they wouldn’t have visited the recommended region (during the specified time period, if relevant) if the PNS hadn’t been sent.
Result – The percentage uplift in the audience size at the recommended region that we can attribute to the message, based on our methodology.
This advanced attribution analysis of message effectiveness is available via the Engage module in the Colocator console. This functionality is available to console users with the applicable Colocator license permissions.
In the message curation interface within Engage, under the tab labelled “Advanced” a toggle for Impact Analysis can be accessed:
Toggling the Impact Analysis to "on" means that the geo-targeted push notification message will be sent to 4 out of every 5 qualifiers, thereby creating a control group against which the impact of the message can be measured. The message is deliberately NOT sent to the control group.
The system also requires the creation of, and reference to, a Segment which is used to describe the behaviour the system should measure against. For example, this Segment might be to visit a certain Region or group of Regions between time A and time B. In essence, this is the behaviour the message is designed to promote, and the system will monitor those Devices that were both sent the message and those not sent the message to determine whether they exhibited this behaviour.
So when setting up the impact analysis the Segment that describes behaviour you want to measure might read: Entered [Region] any time after [time message was sent/started], or for a time bounded event, Entered [Region] between [time message was sent/started] and [time the event at the Region ends].
For example, if the behaviour you are trying to measure is "Visited the Blue region within 2 hour of recommendation push message being sent", you would set up a Segment with filters "Entered Blue Region between [time recommendation message is sent] and [time + 2hours].
Once a message is live and the system has processed enough data to generate a statistically robust finding, selecting the “Report” tab and clicking on the “Load Data” (or subsequently “Refresh Data”) button will populate a live R4 score for that message as illustrated here:
The key measurement is Reaction, i.e. what percentage of message recipients modified their behaviour because of the message?
- Positive Reaction score means the message made people more inclined to follow the recommendation. The higher the percentage, the better the message has performed in this respect.
- Negative Reaction means message made people less inclined to follow recommendation. In other words, it was actually counterproductive, ‘turning off’ recipients.
Confidence in the measurement varies – it depends on the number of people the message went to, and how many of those followed the recommendation, and how many people in our control group also followed the recommendation (despite not receiving a message).
The width of the cyan band on each of the Relevance, Reaction and Result scores in the chart provides an indicator of confidence. The narrower the cyan band, the higher the confidence.