Instrumenting my espresso machine with OpenTelemetry

Instrumenting an Espresso Machine with OpenTelemetry

The process of refining espresso involves managing a complex interplay of variables such as grind size, water pressure, and temperature. These variables are often difficult to track and analyze without proper instrumentation. In this context, the author explores the use of OpenTelemetry to monitor an espresso machine, treating it as a distributed system. This approach allows for the collection and analysis of data from various components of the machine, such as the boiler temperature, pressure, and flow rate.

The Gaggia espresso machine is equipped with an ESP32 controller, which provides a platform for implementing OpenTelemetry. The ESP32 is a microcontroller with limited resources, which presents unique challenges in implementing the OpenTelemetry protocol. The author addresses these challenges by flattening the OTLP protobuf schema to fit within the memory constraints of the ESP32. This involves creating a self-contained otlp.proto file that includes only the necessary fields for the espresso machine's telemetry data.

One of the key challenges in implementing OpenTelemetry on the ESP32 is managing network I/O without disrupting the espresso brewing process. The author uses a dual-core approach, where the control loop and sensor event handlers run on one core, and the exporter runs on another core. This ensures that network operations do not interfere with the brewing process. The use of a snapshot pattern for shared state helps prevent blocking operations that could affect the brewing process.

Another challenge is ensuring accurate timestamping of telemetry data. The ESP32 initially boots with a default timestamp of 1970, which is not useful for tracking events in the present. The author implements a mechanism to wait for NTP confirmation before exporting any telemetry data, ensuring that timestamps are accurate and meaningful.

The implementation of OpenTelemetry on the ESP32 demonstrates the versatility of the protocol in handling diverse use cases, from cloud infrastructure to embedded systems. The use of ClickStack, which includes ClickHouse for storage and HyperDX for visualization, allows for the aggregation and analysis of telemetry data from the espresso machine. This setup enables the author to query and analyze data from each shot, providing insights into the brewing process and helping to refine the quality of the espresso.

The project highlights the importance of careful design and implementation when working with limited resources. The use of a snapshot pattern for shared state, the dual-core approach for managing network I/O, and the implementation of accurate timestamping are all critical to the success of the project. These strategies ensure that the espresso machine's telemetry data is collected and analyzed effectively without disrupting the brewing process.

Source: ClickHouse. Analysis based on publicly available information; no hands-on testing is claimed.