In a year in which the Google Optimize Sunset got all the headlines in the digital analytics and experimentation community, another piece of news was noteworthy in 2023: Spotify announced that they would launch an Experimentation and A/B-testing platform called “Confidence”. They have been developing the product over the last decade or so – just to be used for themselves internally – before realising that the tool is so good that they decided to convert it in a software to be sold on the market. But before we dive into this a bit deeper, first a bit of context.

What is Spotify Confidence?

Spotify Confidence is an experimentation platform designed primarily – according to Spotify – for software development teams. This platform is based on Spotify’s internal A/B testing methods and technologies developed over the years.

The main purpose of Confidence is to simplify the process for teams to set up, run, coordinate, and analyze their own user tests, thereby allowing them to optimize their ideas more effectively. This platform is especially useful for teams that have outgrown their current testing platforms or are seeking an easy and scalable way to start with A/B testing.

Confidence originated from Spotify’s initial efforts in experimentation, which began in the early 2010s with simple, manual A/B tests. Over time, as the company’s needs evolved, Spotify developed a more sophisticated platform named ABBA, which featured flagging and analysis for standardized metrics. This development led to an increase in the scale of experimentation within Spotify, evolving from less than 20 priority experiments per year to hundreds across various teams – or simply to what became one of the world’s best run experimentation programs.

The platform offers a range of features that enable software development teams to implement and manage tests effectively. These include a new Metrics Catalog for self-service analysis, a coordination engine for running multiple experiments simultaneously, and automated scientific best practices.

Confidence is available to customers in three forms:

1. As a managed service, where the experimentation platform is run in a SaaS setup.
2. As a plugin, integrating with Spotify’s existing developer portal ‘Backstage’.
3. Via APIs, allowing maximum customization and integration into a company’s own infrastructure.

Currently, as of December 2023, Confidence is in private beta, with Spotify planning to make it more widely available in the future. This move is part of Spotify’s broader strategy to extend its offerings beyond its core music streaming service and into commercial products for software development.

How will the tool differentiate itself from the competition?

Great question! Many tools currently out there in the market are still looking for that answer. And no, I am not going to name the ones that I think fall under that category. But back to Confidence.

From what I have seen so far from Spotify’s presentations about the product, they have built a software that is designed for developers, but not in the usual tech-heavy way that is only understood by a select few, they want to make the tool easily available also to developers that are new to the world of A/B-testing, experimentation and decision science. Sounds complicated, but let me give you an example:

This is a screenshot of one of the alpha releases. The tool – unlike most competitors currently on the market – does not just output a bunch of metrics and significance calculations that only data scientists would properly understand. It outputs results and recommendations about data quality in plain text and therefore makes it actionable for development teams to take the next steps directly without having to consult stakeholders.

That means that the tool will be able to differentiate itself in the market by the target audience – as most current solutions are primarily (not exclusively!) targeted towards data or experimentation specialists rather than development teams themselves.

Potential impact on the experimentation market

In my opinion, this move will almost exclusively have positive effects for the greater experimentation market. Experimentation is still – after all these years – a small marketing/analytics discipline that is patiently waiting to enjoy widespread adoption. A big-name player like Spotify entering the market will lead to more eyes being on it , which in turn will lead to greater publicity also among other marketing cirles. This impact should drive more companies to start with experimentation, which will benefit all the tool vendors.

In general, the move comes at a great time. The CAC (Customer Acquisition Cost) are going through the roof, driven by high CPC’s because of increasing competition. Smaller companies are looking for ways to get sufficient traffic to keep growing online – which is harder and harder to come by. Here’s where experimentation comes in: Instead of ‘buying’ more traffic, companies would be wise to try to invest in the conversion of the traffic that they already have. Spotify – at least this is my interpretation of the move – is anticipating this development as well and therefore chose the current market environment to launch their product. Hoping that their market entry can provide the necessary spark to push the experimentation discipline into widespread relevance.