Today, the go to solution to measure performance is probably to use an APM(DataDog, Sentry, …), continuously analyzing your production environment. However, since those solutions are operating on real environments they need real users to experience poor performance in order to report issues and unfortunately, performance remains an afterthought appearing only at the end of the development cycle.

Another possibility to measure performance is to create benchmarks while developing and to run them on a regular basis to have an idea of the performance trend of your project. However, with this approach, the variance in the results creates a lot of noise and it’s rarely possible to compare your results with the ones from a co-worker or a production environment.

To make consistent performance measurement as easy as unit testing and fully integrated in CI workflows, we chose a benchmark based solution. And, to eliminate the usual variance associated with running them, we measure the number of instructions and memory/cache accesses through CPU instrumentation performed with Valgrind. This approach gives repeatable and consistent results that couldn’t be obtained with a time based statistical approach, especially in extremely noisy CI and cloud environments.

We have been in closed beta for a few months, already being used by popular open-source projects such as Prisma and Pydantic. Notably, CodSpeed helped Pydantic through their Rust migration, empowering them to make the library 17x faster: https://docs.pydantic.dev/latest/blog/pydantic-v2/#performan...

Today, we’re super excited to finally make the product available to everyone. We currently support Python, Node.js and Rust and are looking forward to integrate with more languages soon.

The product is and will be free forever for open-source projects. Also, we have a per-seat pricing for private repository usage. We have a lot of exciting features planned regarding additional integrations, such as Database and GPU integrations that should come in upcoming months.

Don’t hesitate to try out the product and give your honest feedback. We’re looking forward to your comments!