Syslog Logging¶
Installations of Singularity from Ctrl IQ sources come with a syslog plugin enabled by default. This allows a container audit trail to be formed by recording container startup and usage data.
Ensuring Syslog Logging¶
Syslog logging can be ensured by running the following command:
$ singularity plugin list
ENABLED NAME
yes github.com/ctrliq/syslog-plugin
We can see from the output that the plugin is installed and enabled. If you wish to disable this plugin on your system, you can simply run:
$ sudo singularity plugin disable github.com/ctrliq/syslog-plugin
$ singularity plugin list
ENABLED NAME
no github.com/ctrliq/syslog-plugin
And now the plugin will not be used by Singularity.
Logged Parameters¶
The syslog plugin creates a log record when Singularity starts a container and
when that container process exits. Messages are whitespace separated lists of
container related parameters that are formatted with an equals sign(=
)
separating the parameter name and its value. The format of parameters for each
type of message described below along with a description of the parameter
itself.
Note
It is important to note that joining a running instance with exec
,
shell
or run
to execute commands will not trigger a message to
syslog and will not be reflected in the resource usage of the original
instance container process when it exits.
Container Process Startup¶
UID (int
)
The ID of the user that created the container.
PID (int
)
The process ID of the container process.
IMAGE (string
)
The path of the container image.
SHA256 (hexadecimal
)
The sha256 sum of the container image. This will will display N/A
for a root
filesystem sandbox.
COMMAND (string
)
The Singularity command used to start the container. E.g. exec
, shell
or
run
.
ARGS (string array
)
The command line arguments supplied to Singularity for execution by the container process. The array is bracket enclosed with spaces separating elements.
GPU (bool
)
This indicates if the --nv
or --rocm
options were used to bind in GPU
libraries to the container automatically.
CONTAIN (bool
)
This indicates that a minimal /dev
was bound into the container and
typically bound directories, like /tmp
, were not bound from the host unless
specifically set by another option.
FAKEROOT (bool
)
This indicates that the container process was executed using the Fakeroot feature.
CWD (string
)
The path to the current working directory on the host at invocation.
HOMEDIR (string[:string]
)
The path of the home directory within the container. If the directory is bound from the host do a different location in the container, the source and destination paths will be separated by a colon.
BINDPATHS (string[:string[:string]] array
)
The paths bound into the container from the host, including bind options if
specified. Bound paths are listed in the form source:destination:options
where all three are provided if options
are set and destination
is
provided if it differs from source
. The array is bracket enclosed with
whitespace separating elements.
OVERLAYS (string array
)
The paths of external overlay images applied on top the container filesystem. The array is bracket enclosed with whitespace separating elements.
WORKDIR (string
)
The host path used to create /tmp
, /var/tmp
and $HOME
bind points
when a user specifies the --workdir
option.
SCRATCHDIR (string array
)
The paths for ephemeral scratch directories within the container. If
--workdir
is set, the scratch directories will persist within the working
directory on the host. The array is bracket enclosed with spaces separating
elements.
Container Process Exit¶
UID (int
)
The ID of the user that created the container.
PID (int
)
The process ID of the container process.
IMAGE (string
)
The path of the container image.
UTIME (string,string
) seconds, microseconds
The time the container process spent executing in user mode. The time is
formatted with a s
unit for seconds and us
for microseconds.
STIME (string,string
) seconds, microseconds
The time the container process spent executing in kernel mode. The time is
formatted with a s
unit for seconds and us
for microseconds.
MAXRSS (int
)
The maximum resident set size of the container process in kilobytes.
MINFLT (int
)
The number of page faults that did not require I/O for the container process.
MAJFLT (int
)
The number of page faults requiring I/O for the container process.
INBLOCK (int
)
The number of times the filesystem performed input for the container process.
OUBLOCK (int
)
The number of times the filesystem performed output for the container process.
NVCSW (int
)
The number of voluntary context switches of the container process.
NIVCSW (int
)
The number of involuntary context switches of the container process.
Note
UTIME
, STIME
, MAXRSS
, MINFLT
, MAJFLT
, INBLOCK
,
OUBLOCK
, NVCSW
and NIVCSW
are all derived from the rusage
information of the container process when it exits. See
https://man7.org/linux/man-pages/man2/getrusage.2.html.
Example Log Messages¶
Whenever a container is run on the system, we can see the important parameters of that container execution and understand who ran what. When we run the command below:
$ singularity exec alpine.sif echo Hello World!
We get the following message when the container is started:
Dec 8 17:37:01 pc singularity[225128]: UID=1000 PID=225143 IMAGE=/home/john/alpine.sif SHA256=f51a1f82e512c7daa68df0847540e6f849fc3b9ec8c29acf44f59f84b7eb9a17 COMMAND=exec ARGS=[echo Hello World!] GPU=NONE CONTAIN=false FAKEROOT=false CWD=/home/john HOMEDIR=/home/john BINDPATHS=[] OVERLAYS=[] WORKDIR= SCRATCHDIR=[]
And this message when the container process exits:
Dec 8 17:37:01 pc singularity[225128]: UID=1000 PID=225143 IMAGE=/home/john/alpine.sif UTIME=0s,20979us STIME=0s,16983us MAXRSS=18976 MINFLT=3267 MAJFLT=5 INBLOCK=1344 OUBLOCK=0 NVCSW=700 NIVCSW=15
From these messages we can see that the alpine.sif
container image was
located in /home/john
, nothing was bound into it from the host other than
the default directories, including /home/john
, and the echo
command that
was executed in the container.