Gerrit Productivity Hacks – Flexible Logging: Set DEBUG level logging without Gerrit restart

Many of us have been thrown into production issue, equipped (in the best case ;)) with only vague reproduction instructions and get stuck looking into logs and have only one wish: if I could only magically enable more detailed logging and replay the scenario… just without putting restart procedure into motion… One can argue for having DEBUG level enabled all the time but this is impractical for at least two reasons: too much information is as bad as too little and log files will explode with size. Well, our prayers were answered! Since CollabNet TeamForge Git Integration 8.6.1 release one can modify log level of all or particular classes on the fly.

Example: enable DEBUG for everything

There are two handy SSH Gerrit commands that allow viewing (logging ls-level) and setting/resetting (logging set-level) log level on demand. But there is one catch: user needs to have Administrate Server capability. So get it for yourself or make sure that during hot period there is someone who has it. After being warned, let’s go back to our typical scenario: having only a vague idea what is wrong (out of mysterious error stack trace and some reproduction instructions) one can enable DEBUG for all classes to get more context:

ssh -p 29418 admin@hostname gerrit logging set DEBUG

Both long set-level and short set (call above) versions of command can be used and by default they produce no output (was kind of a strange to me, but it means that everything went fine) and in case of error one gets something like:

Capability administrateServer is required to access this resource

Get yourself Administrate Server capability!

Note that the logging level will stay set until Gerrit is restarted or one calls reset command that will restore log levels of all classes to their initial values (according to the configuration that was provided during Gerrit start):

ssh -p 29418 admin@hostname gerrit logging set reset

If you don’t believe commands that produce no output (I don’t ;)) here is how one can display current level:

ssh -p 29418 admin@hostname gerrit logging ls

Again one can use longer ls-level and shorter ls (call above) version and output would be similar to that:

: INFO
/api: INFO
/gerrit: INFO
/git/api: INFO
GitApplication: INFO
com.collabnet.gerrit: INFO
...
com.google: INFO
com.google.gerrit: INFO
com.google.gerrit.audit.AuditService: INFO
com.google.gerrit.common.ChangeHookRunner: INFO
com.google.gerrit.common.SiteLibraryLoaderUtil: INFO
com.google.gerrit.common.Version: INFO
com.google.gerrit.gpg.GpgModule: INFO
com.google.gerrit.httpd.DirectChangeByCommit: INFO
com.google.gerrit.httpd.RunAsFilter: INFO
...

Example: enable DEBUG for certain package or …

After getting as much as possible with DEBUG level enabled for all packages, one can proceed to narrow it down to particular package:

ssh -p 29418 admin@hostname gerrit logging set DEBUG org.eclipse.jgit

Note that it will affect all classes within this package as well as subpackages – see logging ls command output for package in question:

...
org.eclipse.jgit.internal.storage.file.ObjectDirectory: DEBUG
org.eclipse.jgit.internal.storage.file.RefDirectory: DEBUG
org.eclipse.jgit.util.FS: DEBUG
...

In fact while set command is executed check is being performed if given parameter is included in package qualified class name. Knowing that one can set logging to DEBUG for all classes that have name starting with given prefix regardless their packages:

ssh -p 29418 admin@ctf-centos72-dev-box gerrit logging set DEBUG Jar

Here is ls command output for given pattern:

...
com.google.gerrit.server.plugins.JarPluginProvider: DEBUG
org.eclipse.jetty.util.resource.JarFileResource: DEBUG
org.eclipse.jetty.util.resource.JarResource: DEBUG
...

In order to limit it to particular class it is advised to provide class name qualified with full package name e.g. org.eclipse.jetty.util.resource.JarResource.

What comes next?

Check CollabNet blog for more Gerrit productivity hacks!

Tagged with: , , , , , , , ,
Posted in DevOps/CI, Enterprise Git, Git, SCM

Leave a Reply

Your email address will not be published. Required fields are marked *

*