Example: User engagement modelling

1. User engagement
2. Metrics
3. Interpretations

Metrics, engagement and personalisation

• Talk from Mounia Lalmas – spotify’s Head of personalisation

1. User engagement

User engagement: The quality of the user experience that emphasizes the positive aspects of interaction |

• How to measure the positive experience
• In particular, the fact of wanting to stay on the side of the technology for longer and often; hence leading to loyal users

Why is user engagement important?

• + Can make the user become loyal
• + You can collect more data about the user and can improve on the service
• + User paying for more services
• + User advertising to other people if they have a positive experience
• question to ask is why am I doing?

Mounia’s answer; Why is user engagement important?

• Users have increasingly enhanced expectations about their interactions with technology
  • This leads to increase competition amongst the providers with other services
• Utilitarian factors such as usability experiential factors of interactions (e.g. fun, fulfilment) user engagement

User engagement lifecycle

• 
• 
• New users
• Active users
  • There’s a period where the users are active – they can stay engaged or can be disengaged. Some of them can be re-engaged
  • With user’s behaviour, you can estimate if the person is likely to be engaged/ disengaged based on their interaction
• Dormant user – can be re-engaged or churn
• We need to understand the rectangle!! Of the active user in order to keep users engaged

2. Metrics

Measurements metrics performance indicators key slide

Measurement	Metric	Key Performance Indicator (KPI)
process of obtaining one or more quantity values that can reasonably be attributed to a quantity (number)	Metric is a calculation of the measurement Can aggregate the values to get more meaningful values	Quantifiable measure demonstrating how effectively key business objectives are being achieved What are the values that you wanna measure How do I quantify this Convert it into business value that you want
e.g. clicks on a link (number of clicks)	E.g. (clicks on a link / views of the link) click through rate	E.g. conversion rate – downloads divided by total clicks

• measurement and KPI lies !! parallel !!

We need several metrics - from what we measure to something meaningful/ clever

• Games – users spend a long time per visit
• SNS – users come frequently and stay long
• Search – frequent and short
• Niche – once a week
• News – come periodically

3. Interpretations

1. Abandonment in search – when there is no click on the search result page

• Satisfied – user clicks and abandon; good abandonment

• Dissatisfied – didn’t get what they want; bad abandonment

• Measuring abandonment
  • Cursor trail length – total distance travelled by cursor on search result page
    • Short for good abandonment
  • Movement time – total time cursor moved on the search result page
    • Longer when answers in snippet
  • Cursor speed – average cursor speed
    • Slower when answer is in snippet – good abandonment
• Total time spent on search result page

2. Dwell time in search

• Better proxy for user interest on a news article than click
• E.g. open a link and how long do you spend to look at the link; not moving the cursor but staying in the link
  • Indicat es that user is engaged

3. Dwell time in search: Eye tracking

• Since dwell time is so important they are combining w Eye tracking
  • Can determine bad or good abandonment
• With the tracking can determine if the user is reading the article or if they got lost

• 

• Relevant document vs irrelevant document
  • Left - Engaged w particular document – positive abandonment
  • Right – lost – negative engagement
  • areas and duration of fixation
• The trajectory of the eye
  • 
  • Allow us to know the sequence of the area user is focused on
  • Left – how much time user has spent in each area (e.g. menu, content etc)
  • Right – more engaged coz its more strategic
• WHERE they looked at + for HOW LONG + TRAJECTORY engagement of the user

Summary

• User engagement is applied to:
  • Advertisements
  • Office environment to monitor workers
• Main points
  • User engagement has a lifecycle
    • By separating the lifecycle, the team knows where to improve and what to focus on and divide up the work
  • Metrics depend on the context – in terms of what you want to personalise
• Detecting and understanding implicit signals of user satisfaction are essential for enhancing the quality of recommendations

Applying to YouTube

• What to measure
  • Click time
  • Watch time
  • What is shown to the user and what the user clicks
  • Likes, watch history
  • *Metadata about the videos
    • Topic, authors, likes
• Performance indicators
  • Engagement (watch at least one video)
  • Click on at least one advertisement from video
  • Type of content user is interested in
• Metrics
  • Attention – whenabouts people leave the vid
  • Click through rate
  • If they can predict how long people stay in the vid