kernel:NMI watchdog: BUG: soft lockup – CPU#0 stuck for 21s!

If you think that overcommitment is the reason, you can use the following:


echo [time] >  /proc/sys/kernel/watchdog_thresh

where time cannot be more than 60.


How to boot with old kernel version in RHEL7 ?

  • By default, the key for the GRUB_DEFAULT directive in the /etc/default/grub file is the word saved. This instructs GRUB 2 to load the kernel specified by the saved_entry directive in the GRUB 2 environment file, located at /boot/grub2/grubenv. One can set another GRUB record to be the default, using the grub2-set-default command, which will update the GRUB 2 environment file.
  • By default, the saved_entry value is set to the name of latest installed kernel of package type kernel. This is defined in /etc/sysconfig/kernel by the UPDATEDEFAULT and DEFAULTKERNEL directives. The file can be viewed by the root user as follows:
    $ cat /etc/sysconfig/kernel
    # UPDATEDEFAULT specifies if new-kernel-pkg should make
    # new kernels the default
    # DEFAULTKERNEL specifies the default kernel package type
  • To force a system to always use a particular menu entry, use the menu entry name as the key to the GRUB_DEFAULT directive in the /etc/default/grub file. To list the available menu entries, run the following command as root:
    ~]# awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
    ~]#  awk -F\' '$1=="menuentry " {print $2}' /etc/grub2-efi.cfg 
    Red Hat Enterprise Linux Server (3.10.0-693.el7.x86_64) 7.3 (Maipo)           <<==== Entry 0
    Red Hat Enterprise Linux Server (3.10.0-514.el7.x86_64) 7.3 (Maipo)           <<==== Entry  1
    Red Hat Enterprise Linux Server (0-rescue-d3c598b9d2204138bd2e1001316a5cc6) 7.3 (Maipo)
  • GRUB 2 supports using a numeric value as the key for the saved_entry directive to change the default order in which the kernel or operating systems are loaded. To specify which kernel or operating system should be loaded first, pass its number to the grub2-set-defaultcommand. For example:
    ~]# grub2-set-default 1
  • Check the below file to see the kernel which will be loaded at next boot, crosscheck the numeric value with the menuentry in the /etc/default/grub file.
    ~]# cat /boot/grub2/grubenv |grep saved
    ~]# cat /boot/grub2/grubenv |grep saved
  • Changes to /etc/default/grub require rebuilding the grub.cfg file as follows:
  • Rebuild the /boot/grub2/grub.cfg file by running the grub2-mkconfig -o command as follows:
    • On BIOS-based machines: ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
    • On UEFI-based machines: ~]# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

kernel:NMI watchdog: BUG: soft lockup – CPU#0 stuck for 21s!

  • Consulting the files /etc/grub.conf and /boot/grub/grub.conf, in RHEL 6 and below, or /etc/sysconfig/grub in RHEL 7, it should be verified if the console output is redirected to a console, i.e. using console=ttyS1 or console=ttyS1,9600. In both of these cases the output is restricted to 9600 baud, limiting the output and possibly causing issues.
  • A fix might be to not log to the serial console, or explicitly configure a higher baudrate, i.e. using console=ttyS1,115200. Please note, in some situations also 115200 baud might be a limiting factor.

Otherwise, investigate further root cause conditions

  • Determine if the system was under extremely high load at the time the soft lockups were seen in the logs. If the sysstat package was already installed, it will have recorded load average every 10 minutes using a cron job.
  • Then Load average can be found by searching for ldavg in /var/log/sa/sar<day> where day is the number date of the day when soft lockups were seen. If load average is significantly higher than the amount of logical CPU cores on the system it indicates the soft lockups probably occured because of extremely high workloads.
    In this case it would be best to determine what processes caused the load to go so high and make changes so that the processes don’t cause the issue again.
  • Since it is also possible that defects in the kernel could have caused the soft lockups, full logs needs to be investigated around the time of the soft lockups to see if the issue is a bug or is fixed by errata. It can help to look in the changelog of the latest kernel available on Red Hat Network and see if any soft lockup issues were fixed since the version of the installed kernel.
  • Another way is to eliminate the possibility of a known issue which has already been fixed by testing the system by running it with the latest kernel and see if the soft lockups happen again. Red Hat support may be required to conclusively determine if the issue is a bug.
  • Also verify with a hardware vendor that the issue is not hardware related. One way to verify that the issue is not a known and solved hardware problem is to update the firmware or BIOS to the latest available from the hardware vendor.
  • On virtual systems, soft lockups can indicate that the underlying hypervisor is overcommitted. Please see this article addressing this issue: VMware virtual machine guest suffers multiple soft lockups at the same time
  • If all of the above have been verified to not be the cause it could be a case where soft lockups do not indicate a problem; for example on systems with very large numbers of CPU cores.

If this is encountered in RHEL 5, then increase the threshold at which the messages appear using the following procedures:

  • Run following command and check whether “soft lockup” errors are still encountered on the system:
    # sysctl -w kernel.softlockup_thresh=30
  • To make this parameter persistent across reboots by adding following line in /etc/sysctl.conf file:

In RHEL 6 and above, the threshold is now named “watchdog_thresh” and can be set to no higher than 60:
– To make this change in RHEL 6 and above, set the tuneable kernel.watchdog_thresh in sysctl.conf

Additional Notes:

  • The softlockup_thresh kernel parameter was introduced in Red Hat Enterprise Linux 5.2 in kernel-2.6.18-92.el5 thus it is not possible to modify this on older versions.

Root Cause

  • Soft lockups are situations in which the kernel’s scheduler subsystem has not been given a chance to perform its job for more than the limit set by the watchdog threshold, in seconds; they can be caused by defects in the kernel, by hardware issues or by extremely high workloads.
  • If lockups are encountered on a virtual system, it is important to ensure that the hypervisor is not overcommitted.
  • Hardware issues related to newly installed memory might cause soft lockups.
  • Also misconfigurations might cause the issue, like redirecting console output to a serial device and limiting it to i.e. 9600 baud.
  • On systems with a very large numbers of CPU cores soft lockups might not indicate a problem.

Trying to modify a kernel

I’m a noob in linux/android, yet I have to modify a kernel.

For one specific reason I’m using this guide (it’s somewhat understandable when translated to english using google).

The problem is that I’m stuck at part where you have to “enter the following command to view the address of these two functions”. The only addresses I get when entering those commands are 00000000, which doesn’t seem quite right.

I don’t really understand why is that happening. It may be because the guy who created a guide is using adb for getting addresses, while I’m trying to get them using terminal in android. I can’t quite use adb, because I’m running MEmu emulator and that’s where I need addresses from.


The address is not being shown because you are not running the command under the root user.
This issue has been explained in this answer.

In your case, you need to obtain super-admin rights using either the sudo -s or su command. Once admin, your shell prompt should end with a #. On my one plus, the prompt looks like this when I am admin: A0001:/ #

If it does not work, be sure that the file /proc/sys/kernel/kptr_restrict contains a 0. You can do so by executing the command cat /proc/sys/kernel/kptr_restrict.

To set its value to 0, you should execute the command echo 0 > /proc/sys/kernel/kptr_restrict with administrative rights.

Hope it helps!

Useful ansible stuff


inventory_hostname‘ contains the name of the current node being worked on…. (as in, what it is defined in your hosts file as) so if you want to skip a task for a single node –

- name: Restart amavis
  service: name=amavis state=restarted
  when: inventory_hostname != "boris"

(Don’t restart Amavis for boris,  do for all others).

You could also use :

  when: inventory_hostname not in groups['group_name']

if your aim was to (perhaps skip) a task for some nodes in the specified group.


Need to check whether you need to reboot for a kernel update?

  1. If /vmlinuz doesn’t resolve to the same kernel as we’re running
  2. Reboot
  3. Wait 45 seconds before carrying on…
- name: Check for reboot hint.
  shell: if [ $(readlink -f /vmlinuz) != /boot/vmlinuz-$(uname -r) ]; then echo 'reboot'; else echo 'no'; fi
  ignore_errors: true
  register: reboot_hint

- name: Rebooting ...
  command: shutdown -r now "Ansible kernel update applied"
  async: 0
  poll: 0
  ignore_errors: true
  when: kernelup|changed or reboot_hint.stdout.find("reboot") != -1
  register: rebooting

- name: Wait for thing to reboot...
  pause: seconds=45
  when: rebooting|changed

Fixing ~/.ssh/known_hosts

Often an ansible script may create a remote node – and often it’ll have the same IP/name as a previous entity. This confuses SSH — so after creating :

- name: Fix .ssh/known_hosts. (1)
  local_action: command  ssh-keygen -f "~/.ssh/known_hosts" -R hostname

If you’re using ec2, for instance, you could do something like :

- name: Fix .ssh/known_hosts.
  local_action: command  ssh-keygen -f "~/.ssh/known_hosts" -R {{ item.public_ip }} 
  with_items: ec2_info.instances

Where ec2_info is your registered variable from calling the ‘ec2’ module.

Debug/Dump a variable?

- name: What's in reboot_hint?
  debug: var=reboot_hint

which might output something like :

"reboot_hint": {
        "changed": true, 
        "cmd": "if [ $(readlink -f /vmlinuz) != /boot/vmlinuz-$(uname -r) ]; then echo 'reboot'; else echo 'no'; fi", 
        "delta": "0:00:00.024759", 
        "end": "2014-07-29 09:05:06.564505", 
        "invocation": {
            "module_args": "if [ $(readlink -f /vmlinuz) != /boot/vmlinuz-$(uname -r) ]; then echo 'reboot'; else echo 'no'; fi", 
            "module_name": "shell"
        "rc": 0, 
        "start": "2014-07-29 09:05:06.539746", 
        "stderr": "", 
        "stdout": "reboot", 
        "stdout_lines": [

Which leads on to —

Want to run a shell command do something with the output?

Registered variables have useful attributes like :

  • changed – set to boolean true if something happened (useful to tell when a task has done something on a remote machine).
  • stderr – contains stringy output from stderr
  • stdout – contains stringy output from stdout
  • stdout_lines – contains a list of lines (i.e. stdout split on \n).

(see above)

- name: Do something
  shell: /usr/bin/something | grep -c foo || true
  register: shell_output

So – we could :

- name: Catch some fish (there are at least 5)
  shell: /usr/bin/somethingelse 
  when: shell_output.stdout > "5"

Default values for a Variable, and host specific values.

Perhaps you’ll override a variable, or perhaps not … so you can do something like the following in a template :

max_allowed_packet = {{ mysql_max_allowed_packet|default('128M') }}

And for the annoying hosts that need a larger mysql_max_allowed_packet, just define it within the inventory hosts file like :

busy-web-server mysql_max_allowed_packet=256M

Install Virtualbox on Centos 7

1. Change to root User


su -
## OR ##
sudo -i

2. Install Fedora or RHEL Repo Files


cd /etc/yum.repos.d/

## Fedora 26/25/24/23/22/21/20/19/18/17/16 users

## CentOS 7.4/6.9 and Red Hat (RHEL) 7.4/6.9 users

3. Update latest packages and check your kernel version

Update packages


## Fedora 26/25/24/23/22 ##
dnf update

## Fedora 21/20/19/18/17/16 and CentOS/RHEL 7/6/5 ##
yum update

While executing yum update if you get any dependency errorfor glibc use the following to resolve it:

rpm -qa | grep glibc
You may see duplicates for either glibc or glibc common. There might be a common version in between any two packages of glibc and glibc-common. Remove the package which has uncommon version.

You can also check for duplicates with the following command:
yum list –showduplicates glibc

rpm -e glibc-common-2.17-196.el7

rpm -qa kernel |sort -V |tail -n 1

uname -r

Note: If you got kernel update or run older kernel than newest installed then reboot:



4. Install following dependency packages

CentOS 7/6/5 and Red Hat (RHEL) 7/6/5 needs EPEL repository, install it with following command:


## CentOS 7 and RHEL 7 ##
rpm -Uvh

## CentOS 6 and RHEL 6 ##
rpm -Uvh

## CentOS 5 and RHEL 5 ##
rpm -Uvh

## Fedora 26/25/24/23/22 ##
dnf install binutils gcc make patch libgomp glibc-headers glibc-devel kernel-headers kernel-devel dkms

## Fedora 21/20/19/18/17/16 and CentOS/RHEL 7/6/5 ##
yum install binutils gcc make patch libgomp glibc-headers glibc-devel kernel-headers kernel-devel dkms

## PAE kernel users install ##
## Fedora 24/23/22 ##
dnf install binutils gcc make patch libgomp glibc-headers glibc-devel kernel-headers kernel-PAE-devel dkms

## Fedora 21/20/19/18/17/16 and CentOS/RHEL 7/6/5 ##
yum install binutils gcc make patch libgomp glibc-headers glibc-devel kernel-headers kernel-PAE-devel dkms

5. Install VirtualBox Latest Version 5.1 (currently 5.1.28)


## Fedora 26/25/24/23/22 ##
dnf install VirtualBox-5.1

## Fedora 21/20/19/18/17/16/15/14 and CentOS/RHEL 7/6/5 ##
yum install VirtualBox-5.1

This command create automatically vboxusers group and VirtualBox user must be member of that group.
This command also build needed kernel modules.
Package is VirtualBox-5.1 not VirtualBox.

Rebuild kernel modules with following command:


## Fedora 26/25/24/23/22/21/20/19 and CentOS/RHEL 7 ##
/usr/lib/virtualbox/ setup

## Fedora 18/17/16 and CentOS/RHEL 6/5 ##
/etc/init.d/vboxdrv setup
## OR ##
service vboxdrv setup

You might get the following error: Stopping VirtualBox services. Building VirtualBox kernel modules.
This system is not currently set up to build kernel modules (system extensions).
Running the following commands should set the system up correctly:

yum install kernel-devel-3.10.0-327.28.2.el7.x86_64
(The last command may fail if your system is not fully updated.)
yum install kernel-devel failed: Look at /var/log/vbox-install.log to find out what went wrong.

Execute the following: yum install kernel-devel

If the error still persists look in the log: /var/log/vbox-install.log
If you find the following error: Error: unable to find the sources of your current Linux kernel. Specify KERN_DIR=<directory> and run Make again. Stop.

## Current running kernel on Fedora ##
KERN_DIR=/usr/src/kernels/`uname -r`
## Current running kernel on CentOS and Red Hat (RHEL) ##
KERN_DIR=/usr/src/kernels/`uname -r`-`uname -m`
## Fedora example ##
## CentOS and Red Hat (RHEL) example ##
## Export KERN_DIR ##
export KERN_DIR

Make sure your system is rebooted.

6. Add VirtualBox User(s) to vboxusers Group

Replace user_name with your own user name or some another real user name.


usermod -a -G vboxusers user_name

7. Start VirtualBox

Use launcher from menu or simply run:



fswatch – Monitors Files and Directory Changes or Modifications in Linux

fswatch is a cross-platform, file change monitor that gets notification alerts when the contents of the specified files or directories are altered or modified.

It executes four types of monitors on different operating systems such as:

  1. A monitor build on the File System Events API of Apple OS X.
  2. A monitor based on kqueue, a notification interface present in FreeBSD 4.1 also supported on many *BSD systems, OS X inclusive.
  3. A monitor based on File Events Notification API of the Solaris kernel plus its spin-offs.
  4. A monitor based on inotify, a kernel subsystem that shows file system modifications to apps.
  5. A monitor based on ReadDirectoryChangesW, a Windows API that records alters to a directory.
  6. A monitor that regularly check that status of file system, keeps file modification times in memory, and manually determine file system changes (which works anywhere, where stat can be used).

Features of fswatch

  1. Supports several OS-specific APIs
  2. Allows recursive directory monitoring
  3. Performs path filtering using including and excluding regular expressions
  4. Supports customizable record format
  5. Additionally, it supports periodic idle events

How To Install fswatch in Linux Systems

Unfortunately, fswatch package is not available to install from the default system repositories in any Linux distributions. The only way to install the latest version of fswatch is to build from source tarball as show in the following installation instructions.

First grab the latest fswatch tarball using following wget command and install it as shown:

$ wget
$ tar -xvzf fswatch-1.9.3.tar.gz
$ cd fswatch-1.9.3
$ ./configure
$ make
$ sudo make install 

Important: Make sure you’ve GNU GCC (C and C++ Compiler) and Development Tools (build-essential on Debian/Ubuntu) installed on the system, before you compile fswatch from source. If not, install it using following command on your respective Linux distributions..

# yum group install 'Development Tools'		[On CentOS/RHEL]
# dnf group install 'Development Tools'		[On Fedora 22+ Versions]
$ sudo apt-get install build-essential          [On Debian/Ubuntu Versions]

On Debian/Ubuntu distributions, you might get following error while executing fswatch command..

fswatch: error while loading shared libraries: cannot open shared object file: No such file or directory

To fix it, you need to execute the command below, this will help refresh the links and cache to the dynamic libraries before you can start using fswatch.

$ sudo ldconfig

How do I use fswatch on Linux?

The general syntax for running fswatch is:

$ fswatch [option] [path]

On Linux, it is recommended that you use the default inotify monitor, you can list available monitors by employing the -M or - list-monitors option:

$ fswatch -M
$ fswatch --list-monitors

fswatch - List Monitors

The command below enables you to watch the changes in the current directory (/home/tecmint), with events being delivered to standard output every 4 seconds.

The -l or –-latency option allows you to set the latency in seconds, the default being 1 second.

$ fswatch -l 4 .

fswatch - Monitor Home Directory Changes

The next command monitors changes to the /var/log/auth.log file every 5 seconds:

$ fswatch -l 5 /var/log/auth.log

Using -t or --timestamp option prints the time stamp for every event, to print the time in UTC format, employ -u or --utf-time option. You can as well format time using -f or --format-time format option:

$ fswatch --timestamp /var/log/auth.log

Next, -x or --event-flags tells fswatch to print the event flags along side the event path. You can use –event-field-seperator option to print events using the particular separator.

$ fswatch --events-flags ~ /var/log/auth.log

To print the numeric value of an event indicating changes in your home directory and /var/log/auth.log file, use -n or --numeric option as below:

$ fswatch --numeric ~ /var/log/auth.log 

Perhaps you can look through the fswatch man page for detailed usage options and information:

$ man fswatch