Kamailio (OpenSER) - Debug and syslog messages

Authors of initial tutorial:
Daniel-Constantin Mierla
  <miconda (at) gmail.com>

Elena-Ramona Modroiu
  <ramona (at) asipto.com>

Abstract

Tutorial about how to print debug and syslog messages with Kamailio (OpenSER) SIP Server.

Please note: this tutorial is partly obsolete, logging backend in the server core was reworked for the 1.3 release. The logging methods are renamed from e.g. L_ERR to LM_ERR.

Kamailio (OpenSER) was designed to log debug and error messages either to “syslog” facility or to standard error display “stderr”. The switching between the two options can be done via command line parameters or configuration file. By default, Kamailio (OpenSER) prints log messages to “syslog”.

The messages are printed only if their log level exceeds the threshold set with the “debug” parameter in the configuration file or specified by the command line parameter “-d”.

Kamailio (OpenSER) can be configured to report the log messages to a custom syslog facility. You need to to configure your “syslog” daemon and set the “log_facility” Kamailio (OpenSER) configuration option to the appropriate value. Later in this document, this feature will be presented in more details.

The Kamailio (OpenSER) API offers two functions for printing debug and log messages that can be used when write extensions for Kamailio.

It allows printing messages to different log levels. See the section “Log Levels” for more about the log levels.

        o   level - the level where to print the messages.
        o   string - the string to print. It may contain specifiers in the format of “printf()” C function.
        o   ... - further parameters, in case the “string” parameter includes printing specifiers. 

Prints debug messages.

        o   string - the string to print. It may contain specifiers in the format of “printf()” C function.
        o   ... - further parameters, in case the “string” parameter includes printing specifiers. 

Prints only error messages.

This level should be used if the error requires immediate action.

This level should be used if the error is a critical situation.

This level should be used to report errors during data processing which do not cause system malfunctioning.

This level should be used to write warning messages.

This level should be used to report unusual situations.

This level should be used to write informational messages

This level should be used to write messages for debugging.

Logging in Kamailio (OpenSER) can be configured using the following options. Core options are located at the beginning of the configuration file, before loading modules and any routing block.

• debug - set log level, this is number between 0 and 9. Default is 0. The higher the number the more information will be written to the log. The default level (if no debug is present in the config file and no -d parameter on the command line) is 2 (L_NOTICE).
• log_stderror - if set to “yes”, the server will print its debugging information to standard error output. If set to “no”, syslog will be used. Default is “no” (printing to syslog).
• memlog - debugging level for final memory statistics report. Default is “L_DBG” – memory statistics are dumped only if “debug” option has higher value.
• log_facility - is used to specify what type of program is logging the message. Valid values to use with OpenSER are “LOG_LOCAL0” through “LOG_LOCAL7”. See the manual page of syslog for more information.

The debug level and logging type can be also specified as command line parameters.

• -E - enable logging to stderr
• -d - enable debugging mode (multiple -d increase the debug level, e.g., -ddddd)

Admins can write log messages from configuration file. This eases the troubleshooting and monitoring of the system.

Typically, Kamailio (OpenSER) is configured to issue a message to the syslog on errors or extraordinary situations of which the system administrator should be notified. The format of the logs is loose, depending on the type of event being reported. Usually the situation is described by one or two sentences and some processing data is logged as well when available.

Kamailio (OpenSER) core exports a method to log plain text messages while a module (xlog) allow writing more complex log messages based on specifiers.

In addition to the log messages issued by Kamailio (OpenSER) internally, the administrators can configure Kamailio (OpenSER) to write more log messages using the “log” method explicitly in configuration file. The prototype of the method is:

log([level,] message)

The meaning of the parameters:

• level - an integer value specifying the log level
• message - the message to write in logs. It does not support specifiers, the message will be printed as it is.

If no log level is specified, the messages are issued at log level 4 (L_DBG).

Example 1. Logging example

...
log("This is a log message\n");
log(1, "This is another log message\n");
...

To print formatted log messages from Kamailio (OpenSER) configuration script, one can use “xlog” module. The module exports two functions that allow to use specifiers which will be replaced with an appropriate value when printing the message. An specifier for printing has the same format and meaning as pseudo-variable (http://www.kamailio.org/dokuwiki/doku.php/pseudovariables:devel);

Complete documentation of this module and its facilities can be found in module's “README” file located in sip_router/modules/xlog directory.

Example 2. Logging with xlog

...
xdbg("SIP Request: method [$rm] from [$fu] to [$tu]\n");
xlog("L_INFO", "SIP Request: method [$rm] from [$fu] to [$tu]\n");
xlog("LOG_LOCAL6","L_INFO", "SIP Request: method [$rm] from [$fu] to [$tu]\n");
...

The logging features in Kamailio (OpenSER) can be tuned using several define flags at the compile time. One of them is the possibility to debug memory operations.

If you want to change these flags, you have to edit the “Makefile.defs” file from “sip_router” directory. You have to change the value of the “DEFS” variable (somewhere around the line 290). Next example shows a possibility of how this variable could look.

Example 3. Compile flags

...
DEFS+= $(extra_defs) \
	 -DNAME='"$(MAIN_NAME)"' -DVERSION='"$(RELEASE)"' -DARCH='"$(ARCH)"' \
	 -DOS='"$(OS)"' -DCOMPILER='"$(CC_VER)"' -D__CPU_$(ARCH) -D__OS_$(OS) \
	 -DCFG_DIR='"$(cfg-target)"'\
	 -DPKG_MALLOC \
	 -DSHM_MEM  -DSHM_MMAP \
	 -DDNS_IP_HACK \
	 -DUSE_IPV6 \
	 -DUSE_MCAST \
	 -DUSE_TCP \
	 -DDISABLE_NAGLE \
	 -DF_MALLOC
...

The precompiler flags in Kamailio (OpenSER) that influences the message logging are presented in the next list.

• DNO_DEBUG - turns off some of the debug messages (DBG(…))
• DNO_LOG - completely turns off all the logging (and DBG(…))
• DEXTRA_DEBUG - enables more debug messages
• DDBG_QM_MALLOC - will cause OpenSER to keep and display lot of debugging information about memory operations: file name, function, line number of malloc/free call for each block, extra error checking (trying to free the same pointer twice, trying to free a pointer alloc'ed with a different malloc etc.)

When memory debugging is turned on at compilation, then Kamailio (OpenSER) prints a huge amount of debug messages (note that in order to get the report on shared memory you will need to send signal USR1 to the master process). Some CVS versions of OpenSER (mainly before a release) will have memory debugging turned on by default.

If you want to get rid of all these memory debug messages edit the “Makefile.defs” file and delete -DDBG_QM_MALLOC from the list of compilation flags (from the value of the “DEFS” variable).

Kamailio (OpenSER) uses a special memory manager designed to be fast for operations and sizes that Kamailio (OpenSER) usually deals with. It is incompatible with memory debugging, so when you turn off the memory debugging you have to add the flag that enables fast memory allocation. This flag is named F_MALLOC (-DF_MALLOC in the value of DEFS variable.)

Next example shows what to edit in the “Makefile.defs” to disable memory debugging and enable the fast memory manager.

NOTE: after changing the “Makefile.defs” file you have to recompile and reinstall OpenSER and all the modules you use (you can do it with the following commands: make proper; make all; make install).

Example 4. Disable memory debugging

...
DEFS+= $(extra_defs) \
	 -DNAME='"$(MAIN_NAME)"' -DVERSION='"$(RELEASE)"' -DARCH='"$(ARCH)"' \
	 -DOS='"$(OS)"' -DCOMPILER='"$(CC_VER)"' -D__CPU_$(ARCH) -D__OS_$(OS) \
	 -DCFG_DIR='"$(cfg-target)"'\
	 -DPKG_MALLOC \
	 -DSHM_MEM  -DSHM_MMAP \
	 -DDNS_IP_HACK \
	 -DUSE_IPV6 \
	 -DUSE_MCAST \
	 -DUSE_TCP \
	 -DDISABLE_NAGLE \
	 -DDBG_QM_MALLOC \    <=== delete this line
	 -DDBG_F_MALLOC \     <=== add this line
...

In a Linux/Unix environment it is possible to configure syslog to write log messages to a separate file for a specific facility. It is possible to configure Kamailio (OpenSER) to use a particular facility and then redirect that facility into a separate file. This can greatly improve the readability and manageability of Kamailio (OpenSER) logs.

The default syslog file depends on your OS distribution. For example, in Debian Linux is “/var/log/syslog” and in RedHat Linux is “/var/log/messages”. To make Kamailio print log messages in another file you have to follow the next instructions.

Set option “log_facility=LOG_LOCAL0” in Kamailio (OpenSER) configuration file. Then configure syslog to use a special file for log messages with facility “LOG_LOCAL0”. The configuration file of syslog can be usually found in “/etc/syslog.conf”. Example 5, “Changes in syslog.conf” shows a syslog configuration file modified to report Kamailio error messages into a separate file.

Example 5. Changes in syslog.conf

...
#
# don't log messages with LOG_LOCAL0 in /var/log/syslog anymore
*.*;auth,authpriv.none,local0.none		-/var/log/syslog

#
# log messages with LOG_LOCAL0 in /var/log/kamailio.log
local0.*			-/var/log/kamailio.log
...

NOTE: The '-' in front of “/var/log/kamailio.log” is to omit synchronizing the log file after every log message. If you choose to configure “syslog” in synchronous mode you must be aware that it has big impact over performances when the signaling traffic is high.

In most of current Linux distributions, you just have to add a file:

/etc/logrotate.d/kamailio 

Containing:

/var/log/kamailio.log {
    missingok
    size=50M
    create 0644 root root
    postrotate
	   /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Otherwise, add the piece of config above in /etc/logrotate.conf.

Resources that will help understanding Kamailio (OpenSER)'s logging system: - manual pages of “syslog” (syslog.conf, syslogd, …) - Kamailio (OpenSER) Core Cookbook - XLOG Module Documentation