What does it mean to be a Search PM?
About us: Andrew (AM) and Chandrika (CM) met during their MBA program at MIT Sloan and connected over their passion for product, growth and paying forward the help they got transitioning into product management. AM currently works as a Product Manager at Moveworks and CM works as a Product Manager at DocuSign.
In this new series, we are covering our peers in PM roles across different organizations to give you a view into what various roles demand and which role is likely the best fit for you. For this post, we spoke to our friend Sumit who is currently a Search PM at Audible. Prior to starting at Audible, Sumit spent a year working on the search product at TripAdvisor.
As a Product Manager at Audible, Sumit is focused on the end-to-end search experience. In his role, he has worked on developing expertise in data science, machine learning, analytics, and UX design. He works on both the ranking and presentation of search results across various surfaces.
How do you define the business objectives that you are working towards?
It’s worth providing some historical background to understand our business objectives. Established in 1995, Audible has always been seen as the leader in audiobooks. However, as the options and methods by which people consume media continuously converge, it has become apparent that our competition extends far past other providers of audiobooks. We’re essentially competing for what’s called “Share of Ear” with virtually any provider of audio content, whether its streaming services like Spotify, platforms like Apple Podcasts, or even FM/AM radio. With such a high level of competition, it has become a top priority of the organization to provide a world-class product experience for our listeners to meet their expectations and ensure they can seamlessly discover the content they want. With over 90% engagement rate, Search is a crucial part of improving that experience.
Day-to-day, I focus on how to blend different types of results (ie audiobooks, podcasts, or anything else you might search for), rank them at parity, and present them in an intuitive way. This is across multiple distinct surfaces, from when you first click into the search box and start typing (“autocomplete”) to when you hit enter and see search results (“search results page”).
When it comes to defining business objectives, Search sits at an interesting inflection point where you are constantly trying to strike the right balance between driving hard business objectives like revenue and improving the overall user experience. For example, if our goal was to exclusively drive business revenue, we would optimize our UX and ranking algorithms to prioritize content that directly generates revenue (ie chargeable Audiobooks). However, this would lead to customers being unable to find other content that may be valuable to them (ie all-you-can-listen podcasts).
A way to get around this problem is to create a holistic “Search Success” metric, which should capture how valuable a user found the results given to them for their keyword. This can be done by measuring a combination of click-through-rate on the search results page, as well as the subsequent post-click engagement (ie did the user spend time on the page, make a purchase, hit share, etc., or did they bounce and reformulate their query?).
Once this metric is set, you can leverage an applied data science technique called “Causal Analysis” to establish how moving Search Success by X% will have a Y% downstream impact on a key organization KPI, whether its revenue, retention customer LTV, or something else.
It is a crucial part of your job as a search PM to make this as clear and communicable as possible, as it’s how you ultimately make the case for investments in search from the organization’s leadership. In our case, we might be able to say that because we balanced the ranking of all-you-can-listen content with chargeable content to give our users the best possible experience, we extended our average subscription lifetime.
There’s no single standard for how to define this metric, and it can vary widely across different organizations, but a sound Search Success metric should allow you to:
Reliably measure search improvements within the boundaries of your organization’s A/B testing framework
Establish a measurable downstream impact on at least one major business KPI
How do you go about defining requirements for your projects?
I typically start with defining the customer problem we are trying to solve: what are the objectives? What specific metrics or KPIs are we trying to move through this project? That leads to defining the requirements in detail.
How do you prioritize projects?
At Audible, we go through quarterly planning and prioritize projects for the next 3 months. We align with all the teams involved on what we can deliver. At TripAdvisor, we used to align on board half-yearly goals and did bi-weekly sprint planning. Regardless of the process, there are three things I look at:
Expected impact on business KPIs
The risk level of the project / chances of success - if its resource intensive but unsure of impact, will move towards the end.
Strategic infrastructure dependencies to support a model or new CX framework
Overall, who do you think will enjoy the role of a ‘Search PM’?
The Search PM role is quite versatile.
First, you will need to be open to developing a deep understanding of applied machine learning, analytics, and UX design - all three are strongly intertwined and inseparable in the search product. Examples of ML applications used extensively are learn-to-rank (ranking results) and natural language understanding (determining query intent, applying spell correction, etc.).
You will frequently get feedback from the CEO and customers alike flagging an issue with an unexpected search result, and it will be on you as the search PM to quickly and clearly explain why the model ranked it this way, and then identify the right teams to triage it if necessary.
You must also be data fluent and know how to work with SQL to understand the capabilities of your models and the data behind them. Some questions you’ll think about are “What signals is your product capturing?”, “Which ones should be used in a ranking model?”,“What investments need to be made so your data scientists can start leveraging them?”
Last, you should be ready to network extensively with a diverse set of stakeholders - legal, marketing, content, user research, and even customer service teams. In addition to product owners across the organization.
If you found this valuable and want to support us, consider buying us a coffee!