Go on a Bughunt in Production, but without a map!

Talk with Dirk Kröhan

Debugging is an integral part of the everyday life of a software developer. Bugs happen, bugs are normal, bugs will always exist. There are many measures to detect bugs as early as possible, such as pair programming, code reviews, automated tests or a QA environment.
But one thing is for sure: Even with the best precautions, bugs will make it to production.

Bugs are best investigated in the environment where they occur. But for logs from production, often stricter rules apply, be it a generally higher log level or for reasons of data protection. Often exactly the information needed for an error analysis is missing. Reconstructing the error locally or on another environment is tedious and costs a lot of time, valuable time.

For our microservice architecture, we have therefore built a solution called „On-Demand Debug Logging“. This allows us to activate debug logs for individual users of our system for a certain time interval. Within a few seconds, without any code change or deployment. It allows us to see debug logs for everything that a user does on our API and across all microservices. All other logs, especially those from other users, remain unchanged.

In this talk you will get to know how we implemented the whole thing with the power of Observability, what advantages it brings and what impact it had on our error analyses.