How to fix the "Leap Second that ate the Internet" without rebooting
The gist is that on Saturday, the timekeepers held their button on the clocks to give us an extra second to keep things in check. This Leap Second is arranged to deal with seasonal and climatic events which affect the earth's rotational speed. In other words, the 86400 SI second mean solar day, is a model. We know it is wrong. Nature is much messier than we were lead to believe in 5th grade science class. The earth doesn't rotate precisely once every 24 hours or at a completely constant speed. So some eggheads at the International Earth Rotation and Reference System Service occasionally insert or remove a second as needed. They've been at it since 1972, and since then have always given us an extra second. So sometimes there are 61 seconds in a minute. One day there might be 59.
The trouble is that many of the servers are the internet keep time with the Network Time Protocol (NTP). At midnight on Saturday, the extra second was inserted. Java, mysql, and other commonly used software on the Internet didn't handle this very well. We observed lots of CPU spikes (100%). Suspiciously all of our host machines spiked at the exact same time! While there is a bug or two that needs to be fixed in the Linux kernel, Java, MySQL and other software, there is a workaround for now.
A reboot will fix the problem. If you cant reboot, there is another workaround that one of our engineers Patrick French, has applied to our servers:
- at the commandline, run: date - It will return a number or a formatted date. - if output is formatted, then run: /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; - if output is not formatted, and is a number (which means it's in UTC mode), run: /etc/init.d/ntp stop; date; date -s "`date`"; date;
The CPU usage of Java and other processes should return to normal. In the event it does not, you may need to restart those processes.