Datasource Micrometer Reference Documentation

1. Legal

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

2. Getting Started

If you are getting started with Datasource Micrometer, start by reading this section. It answers the basic “what?”, “how?” and “why?” questions. It includes an introduction to Datasource Micrometer, along with installation instructions.

2.1. Introducing Datasource Micrometer

The Datasource Micrometer provides Micrometer Observation API instrumentation for JDBC operations.

Currently, this project provides two modules - core instrumentation and its Spring Boot Auto Configuration.

If you are a Micrometer user but not using Spring Boot, you can directly use the datasource-micrometer module, which doesn’t have a dependency on the Spring Framework.
If you are a Spring Boot 3 user, then you can add the datasource-micrometer-spring-boot module to the classpath. This will automatically instruments the DataSource and provide tracing capabilities for JDBC operations.

The instrumentation implementation uses datasource-proxy to provide a proxy for JDBC operations.

2.1.1. Background

The Micrometer v1.10.0 introduced the new Observation API and Micrometer Tracing module. The Observation API allows measuring(observing) any interested behavior from the code in action. Then, notifies it to the registered handlers. The Micrometer Tracing module provides an observation handler implementation that creates distributed traces(spans). It uses a tracer that has abstracted the popular tracer implementations and gives the vendor free APIs for tracing.

In Spring Boot 2.x, the Spring Cloud Sleuth provided the instrumentation to the many components including JDBC operations. It was a central library that provides tracing instrumentation to the Spring Boot applications. However, with the new observability, the responsibility for instrumentation has shifted to the individual component. For example, Spring Framework will provide native instrumentation support for its components using the Observation API. As a result, there will be no Spring Cloud Sleuth for Spring Boot 3.

This Datasource Micrometer project provides instrumentation on the JDBC operations to cover what was provided by the Spring Cloud Sleuth but with the Observation API. The initial version aims users to smoothly transition from Spring Boot 2.x with Spring Cloud Sleuth to the Spring Boot 3 in JDBC instrumentation. In addition, since the Observation API and Micrometer Tracing are independent from Spring ecosystem, the instrumentation is available to the non-spring applications as well.

2.1.2. Modules

datasource-micrometer

This module provides the instrumentation on the JDBC operations. The implementation is provided as a datasource-proxy listener. Non-spring applications can directly use this module.

datasource-micrometer-spring-boot

This module provides an auto-configuration for the Spring Boot 3 applications.

datasource-micrometer-bom

This module provides a Bill of Materials (BOM) for dependency management.

2.2. Installation

2.2.1. Maven and Gradle

datasource-micrometer

Maven

<dependency>
    <groupId>net.ttddyy.observation</groupId>
    <artifactId>datasource-micrometer</artifactId>
    <version>1.1.1</version>
</dependency>

Gradle

dependencies {
    implementation "net.ttddyy.observation:datasource-micrometer:1.1.1"
}

datasource-micrometer-spring-boot

Maven

<dependency>
    <groupId>net.ttddyy.observation</groupId>
    <artifactId>datasource-micrometer-spring-boot</artifactId>
    <version>1.1.1</version>
</dependency>

Gradle

dependencies {
    implementation "net.ttddyy.observation:datasource-micrometer-spring-boot:1.1.1"
}

datasource-micrometer-bom

Maven

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>net.ttddyy.observation</groupId>
            <artifactId>datasource-micrometer-bom</artifactId>
            <version>1.1.1</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

Gradle

dependencies {
    implementation platform("net.ttddyy.observation:datasource-micrometer-bom:1.1.1")
}

2.2.2. Setup

datasource-micrometer

To setup the observation for JDBC operations, create a proxy of your DataSource, then register the corresponding tracing observation handlers.

// Register the tracing observation handlers
ObservationRegistry observationRegistry = ObservationRegistry.create();
ObservationConfig observationConfig = observationRegistry.observationConfig();
observationConfig.observationHandler(new ConnectionTracingObservationHandler(tracer));
observationConfig.observationHandler(new QueryTracingObservationHandler(tracer));
observationConfig.observationHandler(new ResultSetTracingObservationHandler(tracer));
//  add other necessary handlers

// Create a DataSource proxy with the observation listener
DataSource dataSource = ...
DataSourceObservationListener listener = new DataSourceObservationListener(observationRegistry);
DataSource instrumented = ProxyDataSourceBuilder.create(dataSource).listener(listener).methodListener(listener).build;

// Use the instrumented DataSource

datasource-micrometer-spring-boot

The auto-configuration class automatically sets up the observation on your DataSource bean.

2.3. Migration from Spring Cloud Sleuth

Datasource Micrometer deliberately provides similar property names to ease the migration from Spring Cloud Sleuth. Most of the JDBC related properties from spring.sleuth.jdbc and spring.sleuth.jdbc.datasource-proxy map to the jdbc and jdbc.datasource-proxy properties.

Please reference the list of application properties in Spring Cloud Sleuth and Datasource Micrometer.

3. Using Datasource Micrometer

This section goes into more detail about how you should use Datasource Micrometer.

3.1. Types of Observations

The Datasource Micrometer creates Connection, Query, Generated Keys(from v1.1), and ResultSet observations.

The Connection observation represents the database connection operations. It is the base observation, as any database access requires a connection. The Query observation provides query execution details, such as execution time, SQL query, bind parameters, etc. The Generated Keys observation records generated keys when auto-generated keys feature is used for the insert statements. The ResultSet observation shows how the operations fetched the data from the query result, including the number of retrieved rows.

To configure these observations, see How to Add Tracing Observation Handlers for JDBC operations.
For Spring Boot, see How to Choose What To Observe.

3.2. Features

3.2.1. HikariCP Support

jdbc.datasource.driver and jdbc.datasource.pool tags are available when the target datasource is a HikariDataSource.

The HikariJdbcObservationFilter provides this feature and this observation filter needs to be registered to the ObservationRegistry.

ObservationRegistry registry = ...
registry.observationConfig().observationFilter(new HikariJdbcObservationFilter());

It is auto configured in datasource-micrometer-spring-boot.

3.2.2. Remote IP and Port

For spans, remote IP and port are retrieved from the datasource url.

3.2.3. Remote Service Name

The datasource name is used as the remote service name in spans. The name is specified when creating a proxy datasource by datasource-proxy.

DataSource instrumented =
    ProxyDataSourceBuilder.create(dataSource)
        .name("myDS")    // Specify datasource name
        .listener(listener)
        .methodListener(listener)
        .build;

For datasource-micrometer-spring-boot, the datasource name is resolved by looking the catalog name at start up (or the connection pool name for Hikari, then fallback to its beanname) by default.

To specify a custom datasource name, see How to Customize the ProxyDataSource Name and How to Customize the ProxyDataSource Creation section.

3.2.4. Application Events

Since version 1.1, datasource-micrometer-spring-boot can publish Spring’s application events for query executions and method invocations on proxied JDBC classes. This feature is disabled by default and can be enabled by setting the property jdbc.event.enabled=true. When enabled, it publishes events - JdbcQueryExecutionEvent and JdbcMethodExecutionEvenet.

3.3. Limitations

3.3.1. Open Session In View

Unfortunately, Open Session In View (OSIV) is not supported. This is because OSIV delays closing the database connection until the HTTP response is sent.

Observation scope orders without OSIV:

Open HTTP request observation scope
Open DB connection observation scope
(Other observation scopes)
Close DB connection observation scope
Close HTTP request observation scope

However, with OSIV enabled, the DB connection closing is delayed, leading to the following order:

Open HTTP request observation scope
Open DB connection observation scope
(Other observation scopes)
(SWAPPED) Close HTTP request observation scope
(SWAPPED) Close DB connection observation scope

Since observation scopes must be closed in the reverse order of their creation, this swapped ordering causes observation leaks.

To disable OSIV, set spring.jpa.open-in-view=false.

4. “How-to” Guides

This section provides answers to some common “how do I do that…?” questions. Its coverage is not exhaustive, but it does cover quite a lot.

We are also more than happy to extend this section. If you want to add a “how-to”, send us a pull request.

4.1. datasource-micrometer

4.1.1. How to instrument `DataSource`

The DataSourceObservationListener provides the observation logic. It is implemented as a datasource-proxy listener. Follow the datasource-proxy usage to create a proxied DataSource with the listener. Then adds tracing observation handlers to the ObservationRegistry.

ObservationRegistry observationRegistry = ...
DataSourceObservationListener listener = new DataSourceObservationListener(observationRegistry);
DataSource instrumented = ProxyDataSourceBuilder.create(dataSource).listener(listener).methodListener(listener).build;

4.1.2. How to Add Tracing Observation Handlers for JDBC operations

There are 3 tracing observation handlers that react to the observations from DataSourceObservationListener.

ConnectionTracingObservationHandler
QueryTracingObservationHandler
ResultSetTracingObservationHandler

generated-keys are also handled by ResultSetTracingObservationHandler.

ObservationRegistry registry = ...
registry.observationConfig().observationHandler(new ConnectionTracingObservationHandler(tracer));
registry.observationConfig().observationHandler(new QueryTracingObservationHandler(tracer));
registry.observationConfig().observationHandler(new ResultSetTracingObservationHandler(tracer));

4.1.3. How to Instrument `ResultSet`

By default, datasource-proxy does not create a proxy for ResultSet. This, in turn, does not instrument the ResultSet. You need to explicitly enable the ResultSet proxy creation. Then, ResultSet get instrumented automatically.

ProxyDataSourceBuilder builder = ProxyDataSourceBuilder.create(dataSource).listener(listener).methodListener(listener);
builder.proxyResultSet();  // enable ResultSet proxy creation
DataSource instrumented = builder.build();

4.1.4. How to Instrument Generated Keys

By default, datasource-proxy does not create a proxy for the generated-keys. You need to explicitly enable the generated-keys proxy creation.

ProxyDataSourceBuilder builder = ProxyDataSourceBuilder.create(dataSource).listener(listener).methodListener(listener);
builder.proxyGeneratedKeys();  // enable Generated-Keys proxy creation
DataSource instrumented = builder.build();

4.1.5. How to Include Bind Parameter Values

Bind parameter values - values from setInt, setString, etc operations on prepared and callable statement - are not tagged to spans by default. The DataSourceObservationListener class has a toggle to enable this. When it is enabled, values are tagged to the query spans as jdbc.params[]

DataSourceObservationListener listener = ...;
listener.setIncludeParameterValues(true);

4.2. datasource-micrometer-spring-boot

4.2.1. How to Disable JDBC Instrumentation

Set the jdbc.datasource-proxy.enabled property to false.

4.2.2. How to Choose What To Observe

Specify jdbc.includes property. By default, the property is set to include(observe) all(CONNECTION, QUERY, KEYS, FETCH) types.

4.2.3. How to Include the Bind Parameter Values in Spans

Set the jdbc.datasource-proxy.include-parameter-values property to true.

4.2.4. How to Enable and Configure Query Logging

To enable the query logging, set the jdbc.datasource-proxy.query.enable-logging property to true.

jdbc.datasource-proxy.query.enable-logging=true

# logging configuration
jdbc.datasource-proxy.logging=slf4j
jdbc.datasource-proxy.query.log-level=DEBUG
jdbc.datasource-proxy.query.logger-name=my.query-logger
jdbc.datasource-proxy.multiline=false

# spring boot log level property
logging.level.my.query-logger=DEBUG

4.2.5. How to Customize the ProxyDataSource Name

Create a custom DataSourceNameResolver bean. It replaces the default bean, DefaultDataSourceNameResolver.

4.2.6. How to Customize the ProxyDataSource Creation

The ProxyDataSourceBuilderCustomizer beans are automatically called before creating a proxy datasource. This callback API allows you to customize the ProxyDataSourceBuilder.

For example, you can use ProxyDataSourceBuilderCustomizer to specify the datasource proxy name. In turn, it becomes the remote service name of the spans.

@Bean
public ProxyDataSourceBuilderCustomizer myCustomizer(){
    return (builder, dataSource, beanName, dataSourceName) -> {
        builder.name("MyAppDataSource");
    };
}

4.2.7. How to Modify the Query in Span

If you want to modify the query string in the span (in high cardinality tags), use ObservationFilter to update the tag value. For example, you could perform sanitization, truncation, etc on the query.

This approach is generally applicable to modify any tags in span.

ObservationFilter is applied when the observation stops. The modification will not be available for timers created by DefaultMeterObservationHandler since it sets the timers at observation start.

@Bean
public ObservationFilter observationFilter() {
    return (context) -> {
        String tagKey = QueryHighCardinalityKeyNames.QUERY.name();
        KeyValue tag = context.getHighCardinalityKeyValue(tagKey);
        if(tag != null) {
            String query = tag.getValue();

            // ... modify query

            context.addHighCardinalityKeyValue(KeyValue.of(tagKey, modifiedQuery));
        }
        return context;
    };
}

4.2.8. How to Enable Application Events

Set the jdbc.event.enabled property to true.

4.2.9. How to Selectively Tag Queries

Instead of tagging all queries, you can define an ObservationPredicate bean to selectively tag queries.

@Bean
ObservationPredicate myObservationPredicate() {
    return (name, context) -> {
        if(context instanceof QueryContext queryContext) {
            return queryContext.getQueries().stream().noneMatch(query -> query.contains("QUERY TO IGNORE"));
        }
        return true;
    };
}

A Kotlin-flavored implementation can be found on this issue comment.

4.2.10. How to Use Custom ObservationConvention

Define beans for your custom observation conventions, which will then be automatically detected and applied to the listener.

4.2.11. How to Switch to Use ProxyDataSource

Since version 1.1, the instrumented DataSource is now a pure JDK proxy. To revert to the previous behavior using ProxyDataSource, set the property: jdbc.datasource-proxy.type=CONCRETE

5. Appendix

5.1. Common Spring Boot application properties

Various properties can be specified inside your application.properties file, inside your application.yml file, or as command line switches. This appendix provides a list of common Datasource Micrometer properties and references to the underlying classes that consume them.

Property contributions can come from additional jar files on your classpath, so you should not consider this an exhaustive list. Also, you can define your own properties.

Name Default Description

Name	Default	Description
jdbc.datasource-proxy.enabled	`true`	Whether to enable JDBC instrumentation.
jdbc.datasource-proxy.include-parameter-values	`false`	Whether to tag actual query parameter values.
jdbc.datasource-proxy.json-format	`false`	Use json output for logging query. @see ProxyDataSourceBuilder#asJson()
jdbc.datasource-proxy.logging	`slf4j`	Logging to use for logging queries.
jdbc.datasource-proxy.multiline	`true`	Use multiline output for logging query. @see ProxyDataSourceBuilder#multiline()
jdbc.datasource-proxy.query.enable-logging	`false`	Enable logging all queries to the log.
jdbc.datasource-proxy.query.log-level	`DEBUG`	Severity of query logger.
jdbc.datasource-proxy.query.logger-name		Name of query logger.
jdbc.datasource-proxy.slow-query.enable-logging	`false`	Enable logging slow queries to the log.
jdbc.datasource-proxy.slow-query.log-level	`WARN`	Severity of slow query logger.
jdbc.datasource-proxy.slow-query.logger-name		Name of slow query logger.
jdbc.datasource-proxy.slow-query.threshold	`300`	Number of seconds to consider query as slow.
jdbc.datasource-proxy.type	`proxy`	Type for the generating DataSource.
jdbc.event.enabled	`false`	Enable publishing query/method execution events.
jdbc.excluded-data-source-bean-names		List of DataSource bean names that will not be decorated.
jdbc.includes		Which types of tracing we would like to include.

jdbc.datasource-proxy.enabled

true

Whether to enable JDBC instrumentation.

jdbc.datasource-proxy.include-parameter-values

false

Whether to tag actual query parameter values.

jdbc.datasource-proxy.json-format

false

Use json output for logging query. @see ProxyDataSourceBuilder#asJson()

jdbc.datasource-proxy.logging

slf4j

Logging to use for logging queries.

jdbc.datasource-proxy.multiline

true

Use multiline output for logging query. @see ProxyDataSourceBuilder#multiline()

jdbc.datasource-proxy.query.enable-logging

false

Enable logging all queries to the log.

jdbc.datasource-proxy.query.log-level

DEBUG

Severity of query logger.

jdbc.datasource-proxy.query.logger-name

Name of query logger.

jdbc.datasource-proxy.slow-query.enable-logging

false

Enable logging slow queries to the log.

jdbc.datasource-proxy.slow-query.log-level

WARN

Severity of slow query logger.

jdbc.datasource-proxy.slow-query.logger-name

Name of slow query logger.

jdbc.datasource-proxy.slow-query.threshold

300

Number of seconds to consider query as slow.

jdbc.datasource-proxy.type

proxy

Type for the generating DataSource.

jdbc.event.enabled

false

Enable publishing query/method execution events.

jdbc.excluded-data-source-bean-names

List of DataSource bean names that will not be decorated.

jdbc.includes

Which types of tracing we would like to include.

5.2. Observability Metrics and Spans

5.2.1. Observability - Conventions

Below you can find a list of all GlobalObservationConvention and ObservationConvention declared by this project.

Table 1. ObservationConvention implementations
ObservationConvention Class Name	Applicable ObservationContext Class Name
`net.ttddyy.observation.tracing.ConnectionObservationConvention`	`ConnectionContext`
`net.ttddyy.observation.tracing.QueryObservationConvention`	`QueryContext`
`net.ttddyy.observation.tracing.GeneratedKeysObservationConvention`	`ResultSetContext`
`net.ttddyy.observation.tracing.ResultSetObservationConvention`	`ResultSetContext`

5.2.2. Observability - Metrics

Below you can find a list of all metrics declared by this project.

Connection

Span created when a JDBC connection takes place.

Metric name jdbc.connection. Type timer.

Metric name jdbc.connection.active. Type long task timer.

KeyValues that are added after starting the Observation might be missing from the *.active metrics.

Micrometer internally uses nanoseconds for the baseunit. However, each backend determines the actual baseunit. (i.e. Prometheus uses seconds)

Fully qualified name of the enclosing class net.ttddyy.observation.tracing.JdbcObservationDocumentation.

All tags must be prefixed with jdbc prefix!

Table 2. Low cardinality Keys
Name	Description
`jdbc.datasource.driver` (required)	Name of the JDBC datasource driver. (HikariCP only)
`jdbc.datasource.name` (required)	Name of the JDBC datasource.
`jdbc.datasource.pool` (required)	Name of the JDBC datasource pool. (HikariCP only)

Since, events were set on this documented entry, they will be converted to the following counters.

Connection - jdbc connection acquired

When the connection is acquired. This event is recorded right after successful "getConnection()" call.

Metric name jdbc.connection.acquired. Type counter.

Connection - jdbc connection commit

When the connection is committed.

Metric name jdbc.connection.commit. Type counter.

Connection - jdbc connection rollback

When the connection is rolled back.

Metric name jdbc.connection.rollback. Type counter.

Generated Keys

Span created when generated keys are returned.

Metric name jdbc.generated-keys. Type timer.

Metric name jdbc.generated-keys.active. Type long task timer.

KeyValues that are added after starting the Observation might be missing from the *.active metrics.

Micrometer internally uses nanoseconds for the baseunit. However, each backend determines the actual baseunit. (i.e. Prometheus uses seconds)

Fully qualified name of the enclosing class net.ttddyy.observation.tracing.JdbcObservationDocumentation.

All tags must be prefixed with jdbc prefix!

Table 3. Low cardinality Keys
Name	Description
`jdbc.datasource.name` (required)	Name of the JDBC datasource.

Query

Span created when executing a query.

Metric name jdbc.query. Type timer.

Metric name jdbc.query.active. Type long task timer.

KeyValues that are added after starting the Observation might be missing from the *.active metrics.

Micrometer internally uses nanoseconds for the baseunit. However, each backend determines the actual baseunit. (i.e. Prometheus uses seconds)

Fully qualified name of the enclosing class net.ttddyy.observation.tracing.JdbcObservationDocumentation.

All tags must be prefixed with jdbc prefix!

Table 4. Low cardinality Keys
Name	Description
`jdbc.datasource.name` (required)	Name of the JDBC datasource.

Result Set

Span created when working with JDBC result set.

Metric name jdbc.result-set. Type timer.

Metric name jdbc.result-set.active. Type long task timer.

KeyValues that are added after starting the Observation might be missing from the *.active metrics.

Micrometer internally uses nanoseconds for the baseunit. However, each backend determines the actual baseunit. (i.e. Prometheus uses seconds)

Fully qualified name of the enclosing class net.ttddyy.observation.tracing.JdbcObservationDocumentation.

All tags must be prefixed with jdbc prefix!

Table 5. Low cardinality Keys
Name	Description
`jdbc.datasource.name` (required)	Name of the JDBC datasource.

5.2.3. Observability - Spans

Below you can find a list of all spans declared by this project.

Connection Span

Span created when a JDBC connection takes place.

Span name connection.

Fully qualified name of the enclosing class net.ttddyy.observation.tracing.JdbcObservationDocumentation.

All tags must be prefixed with jdbc prefix!

Table 6. Tag Keys
Name	Description
`jdbc.datasource.driver` (required)	Name of the JDBC datasource driver. (HikariCP only)
`jdbc.datasource.name` (required)	Name of the JDBC datasource.
`jdbc.datasource.pool` (required)	Name of the JDBC datasource pool. (HikariCP only)

Table 7. Event Values
Name	Description
`acquired`	When the connection is acquired. This event is recorded right after successful "getConnection()" call.
`commit`	When the connection is committed.
`rollback`	When the connection is rolled back.

Generated Keys Span

Span created when generated keys are returned.

Span name generated-keys.

Fully qualified name of the enclosing class net.ttddyy.observation.tracing.JdbcObservationDocumentation.

All tags must be prefixed with jdbc prefix!

Table 8. Tag Keys
Name	Description
`jdbc.datasource.name` (required)	Name of the JDBC datasource.
`jdbc.generated-keys` (required)	Generated keys.

Query Span

Span created when executing a query.

Span name query.

Fully qualified name of the enclosing class net.ttddyy.observation.tracing.JdbcObservationDocumentation.

All tags must be prefixed with jdbc prefix!

Table 9. Tag Keys
Name	Description
`jdbc.datasource.name` (required)	Name of the JDBC datasource.
`jdbc.params[%s]` (required)	JDBC query parameter values. (since the name contains `%s` the final value will be resolved at runtime)
`jdbc.query[%s]` (required)	Name of the JDBC query. (since the name contains `%s` the final value will be resolved at runtime)
`jdbc.row-affected` (required)	Result of "executeUpdate()", "executeLargeUpdate()", "executeBatch()", or "executeLargeBatch()" on "Statement". For batch operations, the value is represented as array.

Result Set Span

Span created when working with JDBC result set.

Span name result-set.

Fully qualified name of the enclosing class net.ttddyy.observation.tracing.JdbcObservationDocumentation.

All tags must be prefixed with jdbc prefix!

Table 10. Tag Keys
Name	Description
`jdbc.datasource.name` (required)	Name of the JDBC datasource.
`jdbc.row-count` (required)	Number of SQL rows.