Setup Chrony to avoid PRVG-13606 in a VirtualBox/RAC env

Configure Local Chrony Server on our RAC Nameserver ns1.example.com

1) Install chrony and disable NTP
[root@ns1 etc]# yum install -y chrony
[root@ns1 etc]# service ntpd stop
[root@ns1 etc]# chkconfig ntpd off
[root@ns1 etc]#  chkconfig --list ntpd
ntpd               0:off    1:off    2:off    3:off    4:off    5:off    6:off

2) Edit the Server Config file '/etc/chrony.conf' and make the following changes.

#Comment the server entries
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

# Allow NTP client access from local network.
allow 192.168/16

#Not synchronized to any external source. (local mode)
local stratum 10

3) Start the server and enable at boot time
[root@ns1 etc]# systemctl start chronyd
[root@ns1 etc]# systemctl enable chronyd

4) Verify Chrony tracking
[root@ns1 etc]# chronyc tracking
Reference ID    : 127.127.1.1 ()
Stratum         : 10
Ref time (UTC)  : Sat Apr  8 08:58:22 2017
System time     : 0.000000000 seconds fast of NTP time
Last offset     : +0.000000000 seconds
RMS offset      : 0.000000000 seconds
Frequency       : 0.000 ppm fast
Residual freq   : +0.000 ppm
Skew            : 0.000 ppm
Root delay      : 0.000000 seconds
Root dispersion : 0.000001 seconds
Update interval : 0.0 seconds
Leap status     : Not synchronised

The Reference ID field shows the IP Address or name of the NTP Server to which we are synchronized.
The value 127.127.1.1 indicates "local" mode (Not synchronized with any external source).

Configure Chrony Client for our RAC VMs

1) Install chrony
[root@ractw21 etc]# yum install -y chrony

2) Edit the Client Config file '/etc/chrony.conf' and make the following changes.

#Comment the server entries
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

#Add local Chrony server entry pointing to our RAC Nameserver
server 192.168.5.50  iburst

3) Start the server and enable at boot time
[root@ractw21 etc]# systemctl start chronyd
[root@ractw21 etc]# systemctl enable chronyd

4) Verify Chrony tracking

[root@ractw21 etc]# chronyc tracking
Reference ID    : 192.168.5.50 (ns1.example.com)
Stratum         : 11
Ref time (UTC)  : Sat Apr  8 09:02:32 2017
System time     : 0.000008410 seconds slow of NTP time
Last offset     : +0.000009758 seconds
RMS offset      : 0.012060679 seconds
Frequency       : 4.164 ppm slow
Residual freq   : +0.001 ppm
Skew            : 0.482 ppm
Root delay      : 0.000209 seconds
Root dispersion : 0.000002 seconds
Update interval : 65.1 seconds
Leap status     : Normal

The Reference ID field shows the IP Address or name of the NTP Server to which we are synchronized.

5) Verify Chrony Sources and resync time with our Nameservers time

[root@ractw21 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ns1.example.com              10   6    77    30    -23us[ +206us] +/-   91us

NOTE: To quickly synchronize with the server run ntpdate
[root@ractw21 ~]# ntpdate 192.168.5.50
8 Apr 11:10:56 ntpdate[25464]: adjust time server 192.168.5.50 offset 0.000664 sec


Chrony not working after reboot

Chrony status after reboot
[root@ractw21 ~]#  chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? ns1.example.com               0   6     0   10y     +0ns[   +0ns] +/-    0ns

Local chrony is not synchronized with our Master Server 


[root@ractw21 ~]# systemctl status chronyd.service
● chronyd.service - NTP client/server
   Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-04-14 11:15:17 CEST; 2min 42s ago
  Process: 858 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
  Process: 805 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 811 (chronyd)
   CGroup: /system.slice/chronyd.service
           └─811 /usr/sbin/chronyd

Apr 14 11:15:17 ractw21.example.com systemd[1]: Starting NTP client/server...
Apr 14 11:15:17 ractw21.example.com chronyd[811]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DE...CHASH)
Apr 14 11:15:17 ractw21.example.com chronyd[811]: Could not resolve address of initstepslew server ns1.example.com
Apr 14 11:15:17 ractw21.example.com chronyd[811]: Frequency -3.525 +/- 2.346 ppm read from /var/lib/chrony/drift
Apr 14 11:15:17 ractw21.example.com systemd[1]: Started NTP client/server.
Hint: Some lines were ellipsized, use -l to show in full.

Status
- chronyc Client reports a problem reaching our Chrony Master server during the Boot process
- Not sure whether this problem is related to VirtualBox or due to problems with the parallel
  OEL 7.3 Linux startup scripts

Anyway a quick and dirty fix is to use rc.local the set Chrony ONLINE when fully network connectivity is established  
- First set  server attribute to OFFLINE  chrony.conf 
    server ns1.example.com offline
- Set chrony ONLINE in rc.local. Add line
   ( sleep 5; chronyc -a  online ) &>/tmp/CHRONY.log 
   Don't forget to give proper permissions 
   [root@ractw21 ~]# chmod +x /etc/rc.d/rc.local'


After a reboot chronyc looks like 
[root@ractw21 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ns1.example.com              10   6   377    18   -384us[ -686us] +/-  168us

Working Chrony Client Setup

  • Don’t forget the set chrony ONLIE in rc.local
Configuration:

192.168.5.50    : IP Address of our Nameserver running Chrony Master
ns1.example.com : Hostname of our Nameserver running Chrony Master

Client File: /etc/chrony.conf
server ns1.example.com offline
driftfile /var/lib/chrony/drift
logdir /var/log/chrony
log measurements statistics tracking
keyfile /etc/chrony.keys
local stratum 10
initstepslew 20 ns1.example.com
allow 192.168.5.50

Chrony Commands

Check Chrony version 
[root@ractw21 ~]# rpm -qa |grep chrony
chrony-2.1.1-4.el7_3.x86_64


If chronyc command fails with:  501 Not authorised     
[root@ractw21 ~]# chronyc online
501 Not authorised

Login to chrony in interactive Mode 
chronyc> authhachrony.confsh SHA1
chronyc> password  HEX:F88F845A9773AEEC9340EFA825AA60435D78D773
chronyc> online 
200 OK

Using chrony from command line when login in required 
[root@ractw21 ~]# chronyc -a  offline
200 OK
200 OK
[root@ractw21 ~]# chronyc -a  online
200 OK
200 OK

Display current chrony Status 
[root@ractw21 ~]# systemctl  status chronyd
● chronyd.service - NTP client/server
   Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-04-14 10:05:43 CEST; 6min ago
  Process: 4461 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
  Process: 4457 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 4459 (chronyd)
   CGroup: /system.slice/chronyd.service
           └─4459 /usr/sbin/chronyd

Apr 14 10:05:37 ractw21.example.com systemd[1]: Starting NTP client/server...
Apr 14 10:05:37 ractw21.example.com chronyd[4459]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +D...CHASH)
Apr 14 10:05:37 ractw21.example.com chronyd[4459]: Frequency -2.683 +/- 0.064 ppm read from /var/lib/chrony/drift
Apr 14 10:05:41 ractw21.example.com chronyd[4459]: System's initial offset : 0.082425 seconds fast of true (slew)
Apr 14 10:05:43 ractw21.example.com systemd[1]: Started NTP client/server.
Apr 14 10:05:47 ractw21.example.com chronyd[4459]: Selected source 192.168.5.50
Hint: Some lines were ellipsized, use -l to show in full.



Check sources stats
[root@ractw21 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ns1.example.com              10   6   377    18   -384us[ -686us] +/-  168us

The columns are as follows:
M
    This indicates the mode of the source. 
     ^ means a server 
     = means a peer  
     # indicates a locally connected reference clock. 
S
    This column indicates the state of the sources. 
    * indicates the source to which chronyd is current synchronised. 
    + indicates other acceptable sources. 
    ? indicates sources to which connectivity has been lost.


Chrony Status Change to Synchronized - this may take a while  
[root@ractw21 ~]#  chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? ns1.example.com              10   6     3    62  -1487ms[-1487ms] +/-  221us

[root@ractw21 ~]#  chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? ns1.example.com              10   6     3    64  -1487ms[-1487ms] +/-  221us
[root@ractw21 ~]#  chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ns1.example.com              10   6     7     0  -5616ns[-1487ms] +/-  976us

Status is set to synchronized with source when Reach Count > 3 - so be patient !


Verify Chrony tracking status
[root@ractw21 etc]# chronyc tracking
Reference ID    : 192.168.5.50 (ns1.example.com)
Stratum         : 11
Ref time (UTC)  : Sat Apr  8 09:02:32 2017
System time     : 0.000008410 seconds slow of NTP time
Last offset     : +0.000009758 seconds
RMS offset      : 0.012060679 seconds
Frequency       : 4.164 ppm slow
Residual freq   : +0.001 ppm
Skew            : 0.482 ppm
Root delay      : 0.000209 seconds
Root dispersion : 0.000002 seconds
Update interval : 65.1 seconds
Leap status     : Normal

The Reference ID field shows the IP Address or name of the NTP Server used for time synchronizing.


Reference:

How are logical volumes like /dev/dm-0 mapped to /dev/sdx disks/partitions ?

Display volume manager devices
#  dmsetup ls --tree -o device
vg_oel64-lv_swap (252:1)
 ├─ (8:3)                         <-- Major, Minor number from /dev/sdX 
 └─ (8:2)
vg_oel64-lv_root (252:0)
 └─ (8:2)

Check /dev/mapper 
# ls -l  /dev/mapper/vg*
lrwxrwxrwx. 1 root root 7 Mar 24 09:07 /dev/mapper/vg_oel64-lv_root -> ../dm-0
lrwxrwxrwx. 1 root root 7 Mar 24 09:07 /dev/mapper/vg_oel64-lv_swap -> ../dm-1

Display physical VG
# pvs
  PV         VG       Fmt  Attr PSize  PFree
  /dev/sda2  vg_oel64 lvm2 a--  40.34g    0
  /dev/sda3  vg_oel64 lvm2 a--   6.04g 3.04g

Display logical volumes
# lvdisplay /dev/mapper/vg_oel64-lv_swap
  --- Logical volume ---
  LV Path                /dev/vg_oel64/lv_swap
  LV Name                lv_swap
  VG Name                vg_oel64
# lvdisplay /dev/mapper/vg_oel64-lv_root
  --- Logical volume ---
  LV Path                /dev/vg_oel64/lv_root
  LV Name                lv_root
  VG Name                vg_oel64

Match major/minor number returned from above dmsetup output
# ls -l /dev/sda2 /dev/sda3
brw-rw----. 1 root disk 8, 2 Mar 24 09:07 /dev/sda2
brw-rw----. 1 root disk 8, 3 Mar 24 09:07 /dev/sda3

--> Swap volume vg_oel64-lv_swap is mapped  to /dev/sda2 and /dev/sda3 partitions
    Root volume vg_oel64-lv_root is only mapped to /dev/sda2
    Both logical volumes are using phy. VG vg_oel64