Some background:

The biggest internal stakeholder of our analytics team is the sales team. The CGO / CSO is hands on, joins all of our analytics-sales team meetings, and outranks the head of our analytics team, who is also present during all of our meetings.

Our company sells a B2B product. The bulk of our analytics projects involve running analyses to show that this product works in the particular customer context (sorry for the vagueness, trying to be anonymous here). The sales team presents these analyses to these customers to convince them to buy it.

We belong in an industry where it seems that customers infrequently conduct their own analyses or experiments to verify that products (like ours) works. So, weirdly to me, customers tend to just trust the analytics we present.

That’s the set-up. Now the troubling parts:

The sales team and analytics teams have tried running A/B tests in the past to demonstrate product value, but the results showed the product was of no benefit on the single, key metric of interest. Since then, the sales team has discouraged the use of A/B tests because “we don’t understand the results.” The head of the analytics team goes along with that.

We run observational, causal analyses to demonstrate an effect and the raw results vary from weak negative effect, to no effect, to weak positive effect. It’s a noisy analysis.

The few times one of our external customers has tried to A/B test the product in a trial run before fully buying it, the results have always been null.

To my eyes, all of the above averages out to *the product does not work*.

Whenever we find a negative or null effect in an analysis, the sales team pressures us into re-running the analysis looking at different sub-groups until we find a positive effect / “good story for the client”. They do this in sometimes subtle ways (e.g. “I think our data isn’t capturing a subtlety in this customer’s industry”) to more obvious pressure (e.g. “look, we have a client meeting on Friday. We need a good story. Try looking at it X way or Y way in the data). I have ample email evidence of this.

I’ve tried explaining to both teams that it is bad statistical practice (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4840791/) to do multiple testing / fishing / p-hacking to find good stories, but no one seems to understand this or perhaps care. So we keep doing this practice, and the sales team finds a way to spin it into a narrative for the customer.

I’ve spoken with the head of the analytics team, the HR team, and even the CEO about the situation. I’m diplomatic, so in all of these conversations I’ve made sure to not heap blame on individuals but calmly insist that the system is at fault and provide suggestions on how to fix it. In all of these cases, people give me platitudes about the need for integrity, thank me for speaking up, but literally nothing happens.

I’m still at the company but currently actively looking around for new jobs. What is an employee supposed to do in this situation? It’s a private company so there’s no whistleblowing to be done, I imagine. Would it be right to secretly inform our customers of the situation? Am I being overly dramatic?