Difference between revisions of "RHEL9 systems"

From GridPP Wiki
Jump to: navigation, search
(Performance Bottleneck)
Line 28: Line 28:
 
In high verbosity environments (multiple-podman containers, or dCache) incorrectly tuning journalctl can lead to problems with performance and useful debugging messages being lost.
 
In high verbosity environments (multiple-podman containers, or dCache) incorrectly tuning journalctl can lead to problems with performance and useful debugging messages being lost.
 
Edinburgh is investigating a good set of configuration parameters to recommend for high-verbosity environments running on hdd moving forward.
 
Edinburgh is investigating a good set of configuration parameters to recommend for high-verbosity environments running on hdd moving forward.
 +
 +
Config from Edinburgh:
 +
<pre>
 +
[root@neeps ~]# mkdir -p /var/log/journal/
 +
[root@neeps ~]# restorecon -R -v /var/log/journal/
 +
[root@neeps ~]# cat /etc/systemd/journald.conf
 +
[Journal]
 +
Storage=persistent
 +
Compress=true
 +
SyncIntervalSec=30s
 +
RateLimitInterval=0
 +
RateLimitBurst=0
 +
SystemMaxUse=10G
 +
SystemKeepFree=1G
 +
RuntimeMaxUse=10G
 +
RuntimeKeepFree=1G
 +
MaxRetentionSec=3month
 +
MaxFileSec=3month
 +
MaxLevelStore=debug
 +
MaxLevelSyslog=debug
 +
MaxLevelKMsg=notice
 +
[root@neeps ~]# systemctl restart systemd-journald
 +
[root@neeps ~]# journalctl --sync
 +
</pre>
  
 
== Security ==
 
== Security ==

Revision as of 12:28, 12 April 2023

RHEL9 Experiences

Experiences with RHEL9 based systems and changes that are good to know.

Missing Packages from EPEL??

Some things require PowerTools repo in EL8 which is now named CRB in EL9.

XFS incompatibility

XFS filesystems made under 9 aren't backwards compatible with 7/8. At Edinburgh we couldn't mount/edit our VM template from a 7/8 host unless it was made using EXT4.

I don't know how/if this will impact proxmox.

JournalCtl

Logs in RAM

Journalctl by default put all of the system journal in /run which is in tmpfs. After some period of time the system locked up due to memory exhaustion which required a reboot and re-configure of journalctl. (Most of the noise in the logs was from external scanning services/tools probing http(s) endpoints, but on one host a significant few GB or so was due to the box being hammered with ssh requests.)

This was the case for Alma 9.0/9.1, it's unknown if Rocky configures this differently.

Performance Bottleneck

In high verbosity environments (multiple-podman containers, or dCache) incorrectly tuning journalctl can lead to problems with performance and useful debugging messages being lost. Edinburgh is investigating a good set of configuration parameters to recommend for high-verbosity environments running on hdd moving forward.

Config from Edinburgh:

[root@neeps ~]# mkdir -p /var/log/journal/
[root@neeps ~]# restorecon -R -v /var/log/journal/
[root@neeps ~]# cat /etc/systemd/journald.conf 
[Journal]
Storage=persistent
Compress=true
SyncIntervalSec=30s
RateLimitInterval=0
RateLimitBurst=0
SystemMaxUse=10G
SystemKeepFree=1G
RuntimeMaxUse=10G
RuntimeKeepFree=1G
MaxRetentionSec=3month
MaxFileSec=3month
MaxLevelStore=debug
MaxLevelSyslog=debug
MaxLevelKMsg=notice
[root@neeps ~]# systemctl restart systemd-journald
[root@neeps ~]# journalctl --sync

Security

Certificate Key Length Policy

To fix this for SSH see: https://access.redhat.com/solutions/6973518

This is mainly to allow connections back to legacy systems and I don't think changing the system policy to allow usage on RHEL9+ is encouraged

Certificate Encryption Type

SHA-1 at the time of writing is used by

update-crypto-policies --set DEFAULT:SHA1