Syslog Logging

Installations of Singularity from Ctrl IQ sources come with a syslog plugin enabled by default. This allows a container audit trail to be formed by recording container startup and usage data.

Ensuring Syslog Logging

Syslog logging can be ensured by running the following command:

$ singularity plugin list
ENABLED  NAME
    yes  github.com/ctrliq/syslog-plugin

We can see from the output that the plugin is installed and enabled. If you wish to disable this plugin on your system, you can simply run:

$ sudo singularity plugin disable github.com/ctrliq/syslog-plugin
$ singularity plugin list
ENABLED  NAME
     no  github.com/ctrliq/syslog-plugin

And now the plugin will not be used by Singularity.

Logged Parameters

The syslog plugin creates a log record when Singularity starts a container and when that container process exits. Messages are whitespace separated lists of container related parameters that are formatted with an equals sign(=) separating the parameter name and its value. The format of parameters for each type of message described below along with a description of the parameter itself.

Note

It is important to note that joining a running instance with exec, shell or run to execute commands will not trigger a message to syslog and will not be reflected in the resource usage of the original instance container process when it exits.

Container Process Startup

UID (int)

The ID of the user that created the container.

PID (int)

The process ID of the container process.

IMAGE (string)

The path of the container image.

SHA256 (hexadecimal)

The sha256 sum of the container image. This will will display N/A for a root filesystem sandbox.

COMMAND (string)

The Singularity command used to start the container. E.g. exec, shell or run.

ARGS (string array)

The command line arguments supplied to Singularity for execution by the container process. The array is bracket enclosed with spaces separating elements.

GPU (bool)

This indicates if the --nv or --rocm options were used to bind in GPU libraries to the container automatically.

CONTAIN (bool)

This indicates that a minimal /dev was bound into the container and typically bound directories, like /tmp, were not bound from the host unless specifically set by another option.

FAKEROOT (bool)

This indicates that the container process was executed using the Fakeroot feature.

CWD (string)

The path to the current working directory on the host at invocation.

HOMEDIR (string[:string])

The path of the home directory within the container. If the directory is bound from the host do a different location in the container, the source and destination paths will be separated by a colon.

BINDPATHS (string[:string[:string]] array)

The paths bound into the container from the host, including bind options if specified. Bound paths are listed in the form source:destination:options where all three are provided if options are set and destination is provided if it differs from source. The array is bracket enclosed with whitespace separating elements.

OVERLAYS (string array)

The paths of external overlay images applied on top the container filesystem. The array is bracket enclosed with whitespace separating elements.

WORKDIR (string)

The host path used to create /tmp, /var/tmp and $HOME bind points when a user specifies the --workdir option.

SCRATCHDIR (string array)

The paths for ephemeral scratch directories within the container. If --workdir is set, the scratch directories will persist within the working directory on the host. The array is bracket enclosed with spaces separating elements.

Container Process Exit

UID (int)

The ID of the user that created the container.

PID (int)

The process ID of the container process.

IMAGE (string)

The path of the container image.

UTIME (string,string) seconds, microseconds

The time the container process spent executing in user mode. The time is formatted with a s unit for seconds and us for microseconds.

STIME (string,string) seconds, microseconds

The time the container process spent executing in kernel mode. The time is formatted with a s unit for seconds and us for microseconds.

MAXRSS (int)

The maximum resident set size of the container process in kilobytes.

MINFLT (int)

The number of page faults that did not require I/O for the container process.

MAJFLT (int)

The number of page faults requiring I/O for the container process.

INBLOCK (int)

The number of times the filesystem performed input for the container process.

OUBLOCK (int)

The number of times the filesystem performed output for the container process.

NVCSW (int)

The number of voluntary context switches of the container process.

NIVCSW (int)

The number of involuntary context switches of the container process.

Note

UTIME, STIME, MAXRSS, MINFLT, MAJFLT, INBLOCK, OUBLOCK, NVCSW and NIVCSW are all derived from the rusage information of the container process when it exits. See https://man7.org/linux/man-pages/man2/getrusage.2.html.

Example Log Messages

Whenever a container is run on the system, we can see the important parameters of that container execution and understand who ran what. When we run the command below:

$ singularity exec alpine.sif echo Hello World!

We get the following message when the container is started:

Dec  8 17:37:01 pc singularity[225128]: UID=1000 PID=225143 IMAGE=/home/john/alpine.sif SHA256=f51a1f82e512c7daa68df0847540e6f849fc3b9ec8c29acf44f59f84b7eb9a17 COMMAND=exec ARGS=[echo Hello World!] GPU=NONE CONTAIN=false FAKEROOT=false CWD=/home/john HOMEDIR=/home/john BINDPATHS=[] OVERLAYS=[] WORKDIR= SCRATCHDIR=[]

And this message when the container process exits:

Dec  8 17:37:01 pc singularity[225128]: UID=1000 PID=225143 IMAGE=/home/john/alpine.sif UTIME=0s,20979us STIME=0s,16983us MAXRSS=18976 MINFLT=3267 MAJFLT=5 INBLOCK=1344 OUBLOCK=0 NVCSW=700 NIVCSW=15

From these messages we can see that the alpine.sif container image was located in /home/john, nothing was bound into it from the host other than the default directories, including /home/john, and the echo command that was executed in the container.