In a world with too few leads, you send your leads to the people who are most likely to close them. In Rover.com’s case, we did this by placing our best sitters (determined algorithmically) at the top of search results.
This worked great during normal times, but it produced sub-optimal results during peak holiday seasons when we had many more dog owners seeking sitters than our top sitters could handle.
One little disclaimer: what follows is merely a theoretical framework on how to deal with this sort of problem. Maybe Rover implemented a solution just like this, maybe they did something a little bit different. I’m not about to divulge such secrets to you! But I think you still might find this problem solving framework useful.
Problem: Too Many Dogs, Not Enough Sitters
Imagine this scenario: it’s the week before the 4th of July and you go to Rover.com to search for a sitter for the long weekend. Rover, looking to maximize the likelihood of you booking a stay, serves up sitters ordered from most likely to least likely to close the deal.
You then click on one of the sitters at the top of the results, maybe the sitter at the top. You submit a request for sitting, and you think, “Great, someone will be able to look after my dog this weekend.” Unbeknownst to you, however, 30 other people also submitted requests. And the sitter only has room for one more dog. Let’s say that on average this sitter books 40% of her requests. Think you’ll still have a sitter? I’ll do that math for you: you only have a 1 in 4.5 million chance of none of those other people booking the stay.
This created a bad experience for every stakeholder in the Rover universe.
- Rover missed out on revenue from dog owners not being able to book a stay
- Dog sitters who had availability missed out on a chance to make some money
- Dog owners could potentially get their weekend plans ruined by not having a sitter
Thankfully, with a little bit of math (and when you get to some of the more advanced versions of this problem, a little bit of Python or some other scripting language) you can devise a framework for solving this problem.
Solution 1: Set an Arbitrary Threshold, Hide The Inundated Sitters
Probably the simplest approach would be to set some threshold where if a sitter has X times as many open inquiries as available spots, don’t show her in the search results.
You can just arbitrarily declare “if a sitter has 5 times as many leads as available spots, don’t show her”, though perhaps your boss might feel slightly more comfortable with a slightly less arbitrary declaration like “if there’s a 90% or greater chance that the sitter will be unavailable, don’t show her”.
To find out whether there’s a 90% (or whatever number you decide on) chance of this, you’ll have to do some math.
Let’s look at a sitter, Jane, who books stays on 40% of her inquiries, therefore there’s a 60% chance that any given inquiry does not result in a stay. Let’s also assume that Jane only has room for one more dog.
To calculate the probability that none of the existing inquiries (leads) convert, thereby preventing the next dog owner from being able to book a stay, you would do the following calculation:
Probability of Vacancy = (Probability of Non-Stay)^(Number Of Leads)
In our sitter Jane’s case, let’s say she has 5 inquiries:
Probability of Vacancy = (60%)^5 = 7.78%
So there’s a 7.78% chance Jane will be available, meaning there’s an 92.22% chance she’ll be unavailable. That’s above the 90% threshold that we set arbitrarily, so you would exclude this sitter from the search results.
But there’s that “arbitrary”/“arbitrarily” word popping up again. And while this method may be an improvement, it might still produces sub-optimal outcomes.
Solution 2: An Expected Value Approach
Imagine that there are only two sitters available for the weekend: Jane and Rob. We established earlier that Jane has a 40% chance of booking a stay (or as we’ll treat it below, a 60% chance of not booking a stay). Rob, however, doesn’t book at as high a rate. It may be that he’s just not an appealing a sitter, or perhaps he’s super picky about which dogs he chooses to let stay at his place. Either way, lower conversion rate of 15% (or, again as we’ll treat it below, an 85% chance of not booking a stay).
Given Rob’s lower conversion rate, we rank him below Jane and place him lower in the search results during normal times. But the week before a major holiday isn’t a normal time. So do we steer the visitor toward Jane or to Rob? For simplicity’s sake, let’s assume both would produce $30 in revenue from booking a stay.
Expected Value to Rover (Steer to Jane) = Expected Value from Jane + Expected Value from Rob
= $30 * (1 – (0.6^5)) + 0
If the math looks strange to you, here’s what we did:
- Start with the revenue associated with the stay ($30 in this case)
- Calculate the odds of at least one of the five dog owners (the first four that Jane already had, and the fifth that would come in if we steered this next owner her way) being able to come to an agreement with Jane – in this case you start with100% (or “1”) and subtract the odds of none of them agreeing to a stay (0.6^5). As written above: (1 – (0.6^5)).
- Multiply the revenue associated with the stay by the odds that at least one of the dog owners agrees to a stay to get your expected value: $30 * 0.9222 = $27.76
- You add that to the expected value from not steering anyone to Rob: $0.
Having established that the expected value of steering traffic to Jane is $27.76, let’s run the same analysis for steering the visitor to Rob.
Expected Value to Rover (Steer to Rob) = Expected Value from Jane + Expected Value from Rob
= ($30 * (1 – (0.6^4))) + ($30 * (1 – (0.85)^1))
Walking through that problem:
- Assume revenue for both sitters is $30
- Get Jane’s expected value with four leads rather than five: $30 * (1 – (0.6^4)) = $26.11
- Get Rob’s expected value with one lead: $30 * (1 – (0.85)^1) = $4.50
- Add the two together: $26.11 + $4.50 = $30.61
The expected value of steering this next visitor to Rob is $30.61, which is greater than the $27.76 we’d have gotten if the visitor went to Jane. In practical terms, what we would want to do is either exclude Jane from the search results, or at least place her below Rob–either way, do something to make it more likely that our visitor chooses Rob rather than Jane. Reminder: despite his lower conversion rate on average, in this case it makes more sense to send the lead to Rob rather than Jane.
That was a fairly simple problem. If you’re still following along, next we’ll take a look at how to solve this when you’re dealing with sitters who have not only multiple inquiries, but also multiple vacancies. That will require brushing up on some more of your probably long forgotten math, re-learning things like Pascal’s Triangle, particularly the section on Combinations. Totally worth it if you’re like me and you believe not just that the leads must flow, but that they must flow to people who are going to help you generate the most revenue.