Logging to a database in a Spring Boot environment

[ppkarwasz]

Spring Boot supports automatic connection establishment with many databases (see Data).
It might be tempting to reuse those connections to forward logs directly to a database. Alas, a couple of problems appear:

• Database connections appear very late in an application's lifecycle, while the logging subsystem is required basically from the start. Currently Spring:

  1. At startup relies on the native auto-configuration of logging backends. For Log4j Core this would pick up a log4j2.xml that does not contain any ${spring:...} lookups (since the Spring configuration is not available yet).
  2. When the Environment becomes available, Spring reconfigures Log4j Core, based on its logging.* properties.
  3. The database client beans unfortunately become available only after the ApplicationContext is refreshed.

• Database clients also support logging. Such log events can not be delivered to the database, since they would cause recursive logging.

What do you think is the best approach to log events through Spring Boot's managed database clients?
Is it even advisable to use the same database client for logging and business?

Some pros and cons I collected:

• Pros: The JDBC appender consumes a lot of connections (see Rationalize the usage of Connection in the JDBC appender #1916). I might be an advantage to attach it to Spring Boot's connection pool instead of handling multiple pools.
• Cons: Logging might require different configuration parameters. For example we recommend to use a separate JPA Persistence Unit for logging (see JPA Appender).

[ppkarwasz]

One of the possible solutions to retrieve a database client from Spring Boot is:

• Write a singleton bean that upon initialization:

  1. Retrieves the database client (can be auto-wired).
  2. Stores it in the logger context via LoggerContext.putObject.

• Write a custom ConfigurationFactory that:

  1. If the logger context contains a database client, creates a database appender programmatically,
  2. Otherwise, it delegates initialization to another factory.

[vy]

I would not implement such a sophisticated configuration mechanism, because:

• There is not really a compelling reason to do so. Shaving off one extra connection pool, IMHO, is not such a big win for a modern application which is composed of dozens of such pools.
• The solution is pretty complicated.
• The solution has other disadvantages that you already cited.
• Keeping logging system apart simplifies the architecture for both the application and the logging system implementation immensely. Given how complicated the things already are, I would avoid adding more to the existing complexity.

[ppkarwasz]

• Keeping logging system apart simplifies the architecture for both the application and the logging system implementation immensely. Given how complicated the things already are, I would avoid adding more to the existing complexity.

Makes sense, although if writing logs to a database is required, I would still use Spring Boot's configuration properties to create a database client. We can provide some copy/paste examples of such an integration in our documentation.

The real question probably is whether logging to a database is required. In a big cloud deployment logging to stdout should be enough, but for smaller deployments, logging to a database might mean less moving parts.

[hydra3230]

Firstly, I don't think that writing logs into database synchronously and fully is valid requirement and design.
Secondly, assuming that we really need to do so, it should be asyschronously with a independent thread, retrive your log key data and message in your own logic implementation.
Spring Boot framework has not designed for this scenario, log4j2 works and loaded following Spring class load design.

If writing key logs into database for key logic and key business, there's no Spring database client issues, just use MDC or NDC of Log4j to do it.