The default core Kinetica log that records database interactions, startup/shutdown, error information, and more is located at /opt/gpudb/core/logs/gpudb.log. This log file can be configured to use a custom format that conforms to a rolling logging standard, enable various appenders and/or loggers, change the logging level, and more using the /opt/gpudb/core/etc/gpudb_logger.conf file.
Customizing the Log
Review the available Log Levels.
- OFF
- FATAL
- ERROR
- WARN
- UERR
- INFO
- DEBUG
- TRACE
- NOTSET
Review the available Appenders.
- ToConsole
- ToSysLog
- ToFile
- ToRollingFile
- ToRollingDatedFile
- ToBacktraceFile
Review the available Loggers.
Open /opt/gpudb/core/etc/gpudb_logger.conf with a text editor.
Update the rootLogger setting with the desired level and appender(s):
log4cplus.rootLogger=<log-level>, <appender1>[, <appender2>, ..., <appenderN>]
Uncomment the desired appender setting section(s):
# --------------------------------------------------------------------------- # Rolling dated log file appender. # ... log4cplus.appender.ToRollingDatedFile=log4cplus::RollingDatedFileAppender ...
Adjust the appender settings as necessary, e.g., log size, threshold, file name, etc.
Adjust additional logger levels and/or appender(s) as necessary:
# =========================================================================== # Loggers # ... # Log memory info, logs at DEBUG. log4cplus.logger.MEMORY=<log-level>, <appender1>[, <appender2>, ..., <appenderN>] ...
Save and exit the file.
Restart the host manager and the database:
service restart gpudb_host_manager && service restart gpudb
Log Levels
Log levels can be attributed to the rootLogger (the base logger, default level is INFO), the individual loggers at the bottom of the file, or as the threshold for an appender. The log levels can be ranked in order from least information to most information logged like so: OFF, FATAL, ERROR, WARN, UERR, INFO, DEBUG, TRACE. Some loggers will not provide their information unless a certain level is set.
| Level | Description |
|---|---|
| OFF | Does not log at all. |
| FATAL | Logs unrecoverable errors that occur and cause the database or a component to exit. |
| ERROR | Logs serious and unexpected errors that may severely impact the database, e.g., network errors, disk full, etc. |
| WARN | Logs unfavorable conditions or states that may impair the database, e.g., network outage, slow disks, etc. |
| UERR | Logs malformed user requests or requests that could not be processed, e.g., invalid table names, bad options, etc. |
| INFO | Logs informational messages recorded during startup and shutdown. There are two INFO logs per request by the head node: one when the request is received and one when the request is completed. |
| DEBUG | Logs verbose messages about program processing and state. |
| TRACE | Logs even more verbose messages about program processing and state. |
| NOTSET | Logs at the level of the next logger in the chain, up to the rootLogger. This is equivalent to commenting out the logger. |
Some loggers may have chained sub loggers; these sub loggers will use the same level as the next logger up the chain (if set). If none of the loggers or sub loggers in the chain are set, the logger will use the rootLogger level. For example, if the following sub logger is NOTSET:
log4cplus.logger.GlobalManager.pub.socket=NOTSET
It will use the level of the next logger up the chain:
log4cplus.logger.GlobalManager.pub=INFO
Appenders
The gpudb_logger.conf file comes with preconfigured appenders that can
be used to create different log types. Multiple appenders can be used if
necessary as long as the appropriate settings are uncommented and the appender
name is added to the rootLogger appender list. Default database setups have
the ToConsole appender enabled.
Important
For sizing considerations, a request typically logs for about 500 bytes, but joins and multi-step queries will log quite a bit more.
ToConsole
The Console appender pipes the output of the /opt/gpudb/core/bin/gpudb script to a /opt/gpudb/core/logs/gpudb-<hostname>-<date>.log file. This file is symbolically linked to /opt/gpudb/core/logs/gpudb.log.
| Setting | Description |
|---|---|
| layout | Dictates the type of layout the logging of which the logging will take form. |
| layout.ConversionPattern | The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details. |
ToSysLog
The SysLog appender sends the log output to the centralized SysLog host.
| Setting | Description |
|---|---|
| layout | Dictates the type of layout the logging of which the logging will take form. |
| layout.ConversionPattern | The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details. |
| syslogHost | Hostname for the SysLog host. |
| Facility | Source of the log messages. |
| Threshold | The level of messages to log. See Log Levels for more information. |
ToFile
The File appender sends each Kinetica process' (rank and host manager) logs to a separate file.
Important
This appender is demanding in terms of disk space.
| Setting | Description |
|---|---|
| File | The name and directory of the files to which to send the logs. The files will have an -r<rank-number> or -hm appended for each rank and the host manager respectively. |
| Threshold | The log level required to log messages. See Log Levels for more information. |
| Append | Enables appending log messages to the end of the files. Leave as true. |
| ImmediateFlush | Enables flushing the output stream to the files with each append operation. |
| layout | Dictates the type of layout the logging of which the logging will take form. |
| layout.ConversionPattern | The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details. |
| UseLockFile | Enables the use of a logger lockfile to create a single log file per host. Important Setting this value to true results in slower performance as the system will need to check the file lock for each append. It's recommended this setting be left as false. |
ToRollingFile
The Rolling File appender processes logs to a rolling log file where a new log file is created once the previous log file reaches the maximum file size. The maximum file size and number of files to keep is configurable.
| Setting | Description |
|---|---|
| File | The name and directory of the files to which to send the logs. The files will have an -r<rank-number> or -hm appended for each rank and the host manager respectively. As files are periodically replaced, older files are renamed <filename>.1 to <filename>.n where n is equal to MaxBackupIndex. |
| Threshold | The log level required to log messages. See Log Levels for more information. |
| MaxFileSize | The maximum file size a log file can reach before a new one is created. |
| MaxBackupIndex | The maximum number of log files to keep until older files are deleted to make room for new ones. Note that this setting defines the maximum backup index for each process, e.g., if MaxBackupIndex is set to 4, the host manager can have up to four files, rank 0 can have up to four files, etc. |
| Append | Enables appending log messages to the end of the file. Leave as true. |
| ImmediateFlush | Enables flushing the output stream to the file with each append operation. |
| layout | Dictates the type of layout the logging of which the logging will take form. |
| layout.ConversionPattern | The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details. |
| UseLockFile | Enables the use of a logger lockfile to create a single log file per host. Important Setting this value to true results in slower performance as the system will need to check the file lock for each append. It's recommended this setting be left as false. |
ToRollingDatedFile
The Rolling Dated File appender processes logs to a dated rolling log file where a new log file is created (with the time the file is created appended to the name) once the previous log file reaches the maximum file size. The maximum file size and number of files to keep is configurable.
| Setting | Description |
|---|---|
| File | The name and directory of the files to which to send the logs. The files will have an -r<rank-number> or -hm appended for each rank and the host manager respectively. As files are periodically replaced, older files are renamed <filename-datetime>-1 to <filename-datetime>-n where n is equal to MaxBackupIndex. |
| Threshold | The log level required to log messages. See Log Levels for more information. |
| MaxFileSize | The maximum file size a log file can reach before a new one is created. |
| MaxBackupIndex | The maximum number of log files to keep until older files are deleted to make room for new ones. Note that this setting defines the maximum backup index for each process, e.g., if MaxBackupIndex is set to 4, the host manager can have up to four files, rank 0 can have up to four files, etc. |
| Append | Enables appending log messages to the end of the file. Leave as true. |
| ImmediateFlush | Enables flushing the output stream to the file with each append operation. |
| layout | Dictates the type of layout the logging of which the logging will take form. |
| layout.ConversionPattern | The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details. |
| UseLockFile | Enables the use of a logger lockfile to create a single log file per host. Important Setting this value to true results in slower performance as the system will need to check the file lock for each append. It's recommended this setting be left as false. |
ToBacktraceFile
The Backtrace File appender redirects the backtrace logs generated by the ErrorBacktrace logger so they do not appear in the rootLogger output. Add the ERROR log level and the ToBacktraceFile appender to the ErrorBacktrace like so to properly setup this appender:
log4cplus.ErrorBacktrace=ERROR, ToBacktraceFile
Important
It's advised that this appender also be added to the rootLogger so that logs with a threshold of ERROR are also logged to give context to a backtrace.
| Setting | Description |
|---|---|
| File | The name and directory of the file to which to send the logs. |
| Threshold | The log level required to log messages. See Log Levels for more information. |
| Append | Enables appending log messages to the end of the file. Leave as true. |
| ImmediateFlush | Enables flushing the output stream to the file with each append operation. |
| layout | Dictates the type of layout the logging of which the logging will take form. |
| layout.ConversionPattern | The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details. |
| additivity | If using an alternate appender for the ErrorBacktrace logger, set the additivity to false to avoid duplicated the backtrace logs in the rootLogger output. |
Loggers
Loggers control specific types of information. Each logger can have its own log level and appender. If the level or appender are not specified, the rootLogger level and/or appender(s) are used.
Loggers are divided into sections:
- System and process information -- memory, CPU
- GlobalManager and GlobalRegistry -- publish and subscribe sockets
- All ranks (not exclusive to a particular rank) -- filter planner, job manager, security manager
- Head node -- endpoint manager, graph client, WMS
- Worker nodes -- bytes and GPU manager, queue, tasks
Auditing
The location of the request auditor logs can also be controlled via the
gpudb_logger.conf file. Auditing must be enabled before the logging
can be configured; see Auditing for more information.
The auditor uses its own appender to separate audit messages from other log messages. All audit messages are at the INFO or ERROR level; it's recommended the audit log level be kept to at least INFO. If desired, the auditor can use a different appender.
The audit log comes with a few preconfigured appender types that are similar to the rootLogger appenders.
Customizing the Log
Review the available Log Levels.
- OFF
- FATAL
- ERROR
- WARN
- UERR
- INFO
- DEBUG
- TRACE
- NOTSET
Review the available Appenders.
- ToConsole
- ToSysLog
- ToFile
- ToRollingFile
- ToRollingDatedFile
- ToBacktraceFile
Open /opt/gpudb/core/etc/gpudb_logger.conf with a text editor.
Adjust the default logger.Audit setting as necessary:
log4cplus.logger.Audit=INFO, AuditAppender
Optionally, update the additivity setting if audit information is desired in both the rootLogger and the audit log. In most cases, this will not need to be updated.
Optionally, uncomment one of the other audit appenders (File or Rolling File). If uncommenting a new appender, the Console appender will need to be commented out.
Adjust the appender settings as necessary, e.g., log size, threshold, file name, etc.
Save and exit the file.
Restart the host manager and the database:
service restart gpudb_host_manager && service restart gpudb
Audit Appenders
Console
The Console appender for the audit log will direct output to the rootLogger file(s).
| Setting | Description |
|---|---|
| layout | Dictates the type of layout the logging of which the logging will take form. |
| layout.ConversionPattern | The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details. |
File
The File appender for the audit log will direct output to a file.
| Setting | Description |
|---|---|
| File | The name and directory of the files to which to send the logs. The files will have an -r<rank-number> or -hm appended for each rank and the host manager respectively. |
| layout | Dictates the type of layout the logging of which the logging will take form. |
| layout.ConversionPattern | The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details. |
Rolling File
The Rolling File appender for the audit log will direct output to a rolling file.
| Setting | Description |
|---|---|
| File | The name and directory of the files to which to send the logs. The files will have an -r<rank-number> or -hm appended for each rank and the host manager respectively. As files are periodically replaced, older files are renamed <filename>.1 to <filename>.n where n is equal to MaxBackupIndex. |
| MaxFileSize | The maximum file size a log file can reach before a new one is created. |
| MaxBackupIndex | The maximum number of log files to keep until older files are deleted to make room for new ones. Note that this setting defines the maximum backup index for each process, e.g., if MaxBackupIndex is set to 4, the host manager can have up to four files, rank 0 can have up to four files, etc. |
| Append | Enables appending log messages to the end of the file. Leave as false. |
| layout | Dictates the type of layout the logging of which the logging will take form. |
| layout.ConversionPattern | The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details. |
