What does it mean to be a Machine Learning PM?
About us: Andrew (AM) and Chandrika (CM) met during their MBA program at MIT Sloan and connected over their passion for product, growth and paying forward the help they got transitioning into product management. AM currently works as a Product Manager at Moveworks and CM works as a Senior Product Manager at DocuSign.
In this new series, we are covering our peers in PM roles across different organizations to give you a view into what various roles demand and which role is likely the best fit for you. For this post, we spoke to our friend Raj who is currently a Sr. PM at StubHub, working on the Seller Experience, including several Machine Learning (ML) Products.
As a Product Manager at StubHub, Raj is improving the full end-to-end Seller experience within the StubHub marketplace, which includes the core flows that sellers interact with on the platform and the algorithms that power seller tools. Also, Raj has witnessed - and been a part of - the massive shift in StubHub’s business during the course of 2020, which makes for an incredible story.
How do you define the business objectives that you are working towards with your project?
As a product manager, one of my responsibilities is to craft my team’s objectives, key metrics, and roadmap for a given fiscal quarter. At any point, I should be able to communicate how my team’s current priorities ladder up to company-wide objectives and Level 1 success metrics so that product leadership can ensure different teams are rowing in the same direction. Our Level 1 success metrics, determined by the executive team, have changed twice in the span of a year. As a product manager, I’ve learned how to quickly reorient myself and my team’s work when the foundational measures of success are a moving target.
The global pandemic was one of the instigators of change, given its negative impact on StubHub’s core offering of providing a marketplace for buyers and sellers of tickets for sports, concerts, theater and other live entertainment events. The mass level of postponements and cancellations across all live events early in the pandemic was unprecedented territory for our industry and these unprecedented times shifted our success metrics; maximizing sales and revenue in the early months of the pandemic wouldn’t make sense. Instead, we were focused on ensuring our continued business health and coming out of this period with a stronger foundational technology platform would serve us in the longer term.
Before the pandemic, we were focused on revenue and getting more people to live events, which we measure by number of orders. Since I work on the seller experience, I defined my goals as improving the seller experience to increase the conversion rates through our flows, developing seller loyalty to increase the repeat rate for selling on our platform, and developing seller tools that increase the percentage of seller listings that ultimately sell. In basic terms, users who are able to successfully list their tickets, times the number of listings per user, times the percent of listings that actually sell, equals the number of orders.
In particular, building tools that improve our sellers’ likelihoods of successful sale to a buyer is an area that requires both UX and machine learning innovations. In the absence of seller tooling, we find that many sellers price their tickets above market value and don’t update the prices of their tickets frequently enough, which means their listings will remain “on the shelf” with stale prices. We have worked on multiple algorithms and UX experiences to tackle this customer problem. For example, we need to algorithmically estimate the quality of the ticket (based on its location in the venue and how much similar tickets transacted for in the past). We also need to determine the market price of the ticket, based on the quality of the seat and how similar quality seats are faring in the marketplace, given the demand for the event and the time left until the event. In addition to shipping algorithm upgrades for these products, we also experiment with different UX experiences to ensure high customer adoption of our tools.
How do you go about defining requirements for your projects?
My requirements need to consider several different perspectives. The requirements should start with the customer problem and what pain points we are trying to solve. There will be collaboration with the frontend teams to agree on what the outputs of the machine learning models are, because some permutation of these outputs will eventually need to be communicated to a user. Machine learning engineering and data engineering teams are central to the process as well, because so much of getting a machine learning model into production requires heavy investments in data pipelines and model serving infrastructure. Given the large potential for business impact, I collaborate heavily with our business and strategic teams to generate the universe of possibilities and engage in healthy debates to push our own thinking forward. Finally, and most importantly, collaborating with ML engineers and data scientists themselves is key. With ML product management, the final requirements depend on detailed discussion about the impact on the model and effort required of any particular adjustment to a model, so these conversations require the technical team’s perspectives to be successful.
How do you prioritize projects?
One of the more challenging aspects of prioritization is explaining “no” to stakeholders in your organization fairly often. If you are not transparent about suggested work being deprioritized, you can end up with well-intentioned stakeholders pinging for status updates and being unhappy with the perceived inaction from you.
A tactical approach is to maintain a PBL or “Product Backlog”, which is a simplified and consolidated version of the more detailed stories being written in JIRA. I have found that JIRA is not effective for communicating a detailed roadmap with stakeholders because many of them don’t use JIRA and it doesn’t quickly convey my team’s current priorities. For example, I use my PBL (which I keep in a spreadsheet) to review milestones with the Marketing team so they know why a new feature is not being prioritized by seeing what other work has been prioritized, along with our estimated dates of completion. In this view, any new requests can have estimated impact and effort, and be slotted into the backlog where appropriate. It gives a strong visual when you have to say “no” to a request, because there is simply too much high impact work preceding in priority and stakeholders can see that for themselves.
Overall, who do you think will enjoy the role of a ‘ML PM’?
First, you must have fluency with data. While there are analytics resources provided, you need to be comfortable with SQL so that you are actively understanding the data and basic trends. As a machine learning PM, there are times where model output seems off and I want to do some quick investigations in SQL before deciding next steps. You’ll need to have an appetite to constantly learn more about ML as the field is quickly evolving and the PM role exposes you to significant technical subject matter in daily conversations and meetings.
Second, you’ll need to be open to extensive context switching and autonomy. On the same day, you might have a meeting with your ML team to discuss new model architecture implementation, then switch to a meeting with your UX team on some initial designs for the customer experience, and then head off to a meeting with executives to discuss high-level strategic priorities and business landscapes. While much of your day is scheduled in meetings, there is a high degree of autonomy required in deciding how best to utilize your time -- and ensuring you are making great decisions for your product.
Finally, a love for creating and nurturing relationships. One of the PM superpowers I enjoy the most is embracing the diverse individuals and teams that I work with. There is no ML product that I can build myself, but through investing in relationships across all corners of our organization, I can bring the right people together to solve tough problems, and not to mention that work becomes so much more fun.
If you found this valuable and want to support us, consider buying us a coffee!