綾小路龍之介の素人思考

[linux] nfsのレスポンスが遅くなり、set_rtc_mmss: can't update from ** to **のエラー

nfsでファイルを読み書きしているときに、たまにものすごくレスポンスが遅くなる場合がある。原因はntpサーバマシンとntpクライアントマシンの間で時刻が大きく食い違っていたことが原因と考えられる。時刻同期に失敗していたnfsサーバマシンの時刻あわせを行い、様子見運用。ntpレスポンスが遅くなったときにそのときに/var/log/messageに流れたエラーらしきものが下。。

Apr  5 07:58:34 debian kernel: [2368986.028400] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:35 debian kernel: [2368987.028164] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:36 debian kernel: [2368988.028188] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:37 debian kernel: [2368989.028175] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:38 debian kernel: [2368990.028150] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:39 debian kernel: [2368991.028182] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:40 debian kernel: [2368992.028190] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:41 debian kernel: [2368993.028162] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:42 debian kernel: [2368994.028160] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:43 debian kernel: [2368995.028301] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:44 debian kernel: [2368996.028338] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:45 debian kernel: [2368997.028446] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:46 debian kernel: [2368998.028164] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:47 debian kernel: [2368999.028171] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:48 debian kernel: [2369000.028168] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:49 debian kernel: [2369001.028088] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:50 debian kernel: [2369002.028161] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:51 debian kernel: [2369003.028177] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:52 debian kernel: [2369004.028209] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:53 debian kernel: [2369005.028168] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:54 debian kernel: [2369006.028157] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:55 debian kernel: [2369007.028242] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:56 debian kernel: [2369008.028155] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:57 debian kernel: [2369009.028132] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:58 debian kernel: [2369010.028145] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:58:59 debian kernel: [2369011.028145] set_rtc_mmss: can't update from 0 to 58
Apr  5 07:59:00 debian kernel: [2369012.028069] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:01 debian kernel: [2369013.028138] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:02 debian kernel: [2369014.028132] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:03 debian kernel: [2369015.028123] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:04 debian kernel: [2369016.028063] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:05 debian kernel: [2369017.028067] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:06 debian kernel: [2369018.028059] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:07 debian kernel: [2369019.028068] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:08 debian kernel: [2369020.028058] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:09 debian kernel: [2369021.028057] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:10 debian kernel: [2369022.028059] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:11 debian kernel: [2369023.028067] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:12 debian kernel: [2369024.028063] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:13 debian kernel: [2369025.028067] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:14 debian kernel: [2369026.028066] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:15 debian kernel: [2369027.028073] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:16 debian kernel: [2369028.028057] set_rtc_mmss: can't update from 0 to 59
Apr  5 07:59:17 debian kernel: [2369029.028161] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:18 debian kernel: [2369030.028145] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:19 debian kernel: [2369031.028108] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:20 debian kernel: [2369032.028195] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:21 debian kernel: [2369033.028153] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:22 debian kernel: [2369034.028113] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:23 debian kernel: [2369035.028074] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:24 debian kernel: [2369036.028064] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:25 debian kernel: [2369037.028061] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:26 debian kernel: [2369038.028069] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:27 debian kernel: [2369039.028066] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:28 debian kernel: [2369040.028067] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:29 debian kernel: [2369041.028072] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:30 debian kernel: [2369042.028064] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:31 debian kernel: [2369043.028072] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:32 debian kernel: [2369044.028069] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:33 debian kernel: [2369045.028247] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:34 debian kernel: [2369046.028074] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:35 debian kernel: [2369047.028064] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:36 debian kernel: [2369048.028060] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:37 debian kernel: [2369049.028120] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:38 debian kernel: [2369050.028165] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:39 debian kernel: [2369051.028195] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:40 debian kernel: [2369052.028182] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:41 debian kernel: [2369053.028749] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:42 debian kernel: [2369054.028152] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:43 debian kernel: [2369055.028183] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:44 debian kernel: [2369056.028170] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:45 debian kernel: [2369057.028115] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:46 debian kernel: [2369058.028157] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:47 debian kernel: [2369059.028188] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:48 debian kernel: [2369060.028157] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:49 debian kernel: [2369061.028156] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:50 debian kernel: [2369062.028171] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:51 debian kernel: [2369063.028184] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:52 debian kernel: [2369064.028177] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:53 debian kernel: [2369065.028163] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:54 debian kernel: [2369066.028162] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:55 debian kernel: [2369067.028169] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:56 debian kernel: [2369068.028201] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:57 debian kernel: [2369069.028207] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:58 debian kernel: [2369070.028169] set_rtc_mmss: can't update from 1 to 59
Apr  5 07:59:59 debian kernel: [2369071.028152] set_rtc_mmss: can't update from 1 to 59

まずはnfsサーバマシンとクライアントマシンの時刻を確認すると大きく(3分くらい)違っている。

nfsserver:~$ date
Sun Apr  5 08:46:19 JST 2009
nfsclient:~$ date
Sun Apr  5 08:39:16 JST 2009

両方のマシンでのntpqの結果からnfsserverでは、時刻がずれ過ぎて時刻同期に失敗していることがわかる。offsetが168700[msec]程度なので、ほぼ2分48秒のずれがある。30分以上ずれると同期に失敗すると言う話を聞いたことがあるが、そうじゃないのかな。

nfsserver:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 doga.jp         133.243.238.243  2 u   61   64   17   39.279  168739. 26565.7
 ns.doga.co.jp   133.243.238.243  2 u   23   64   37   48.324  168745. 128116.
 suisho.attritio 133.100.9.2      2 u   31   64   37    9.928  168748. 128445.
 www.4koma.com   133.243.238.243  2 u   25   64   37   23.192  168751. 128115.
nfsclient:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+211.19.56.138   192.168.0.5      2 u  177 1024  377   12.837    2.072   0.340
*doga.jp         133.243.238.243  2 u  678 1024  377   17.071    1.699   6.645
+ns2.ktroad.net  210.173.160.57   3 u  204 1024  377   17.039    1.324   0.061
-7c29477b.i-revo 61.114.187.55    2 u  167 1024  377   16.798    0.937   1.085

時刻がずれすぎている場合は手作業で直す必要がある。そのためにntpdateコマンドを使う。コマンドが見つからないのでインストールする。

nfsserver:~# ntpdate doga.jp
-su: ntpdate: command not found
nfsserver:~# aptitude search  ntp
p   chntpw                  - NT SAM password recovery utility
p   cyrus-nntpd-2.2         - Cyrus mail system (NNTP support)
v   cyrus21-nntpd           -
v   cyrus22-nntpd           -
p   libnet-ntp-perl         - Perl module to query NTP servers
p   libnews-nntpclient-perl - News::NNTPClient, Perl support for accessing NNTP servers
p   mountpy                 - script for quick mounting of removable devices
p   msntp                   - A very simple and portable SNTP client/server
i   ntp                     - Network Time Protocol daemon and utility programs
p   ntp-doc                 - Network Time Protocol documentation
p   ntpdate                 - client for setting system time from NTP servers
p   openntpd                - OpenBSD NTP daemon
nfsserver:~# aptitude -R install ntpdate
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Reading task descriptions... Done
The following NEW packages will be installed:
  ntpdate
The following packages are RECOMMENDED but will NOT be installed:
  lockfile-progs
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 59.8kB of archives. After unpacking 209kB will be used.
Writing extended state information... Done
Get:1 http://ftp.jp.debian.org lenny/main ntpdate 1:4.2.4p4+dfsg-8 [59.8kB]
Fetched 59.8kB in 0s (80.4kB/s)
Selecting previously deselected package ntpdate.
(Reading database ... 93881 files and directories currently installed.)
Unpacking ntpdate (from .../ntpdate_1%3a4.2.4p4+dfsg-8_i386.deb) ...
Processing triggers for man-db ...
Setting up ntpdate (1:4.2.4p4+dfsg-8) ...
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Writing extended state information... Done
Reading task descriptions... Done

導入が終わったら時刻あわせ。ntpクライアントデーモンが動いているとntpdateが使うソケットが使用中となって同期できないので、ntpdateする前にntpクライアントデーモンをstopする。同期完了したら再度ntpクライアントデーモンをstartしておく。

nfsserver:~# date
Sun Apr  5 08:46:19 JST 2009
nfsserver:~# ntpdate ntp1.jst.mfeed.ad.jp
 5 Apr 08:47:04 ntpdate[3568]: the NTP socket is in use, exiting
nfsserver:~# date
Sun Apr  5 08:47:12 JST 2009
nfsserver:~# /etc/init.d/ntp stop
Stopping NTP server: ntpd.
nfsserver:~# ntpdate ntp1.jst.mfeed.ad.jp
 5 Apr 08:50:49 ntpdate[3577]: step time server 210.173.160.27 offset 173.438520 sec
nfsserver:~# date
Sun Apr  5 08:50:53 JST 2009
nfsserver:~# /etc/init.d/ntp start
Starting NTP server: ntpd.

同期確認のためにntp -qする。すぐには反映されないので30分くらい待って再確認。同期されていることがわかる。

nfsserver:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ns.doga.co.jp   133.243.238.243  2 u    -   64    1   62.996  -10.454   7.343
 www.hypercore.c 210.173.160.27   3 u    1   64    1   16.511    1.519   0.507
 kanda.likk.net  133.100.9.2      2 u    -   64    1   10.775  -15.372   0.526
nfsserver:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+ns.doga.co.jp   133.243.238.243  2 u   29   64   37   74.702   46.415  23.793
+www.hypercore.c 210.173.160.27   3 u    9   64   37   15.582   64.689   7.138
*kanda.likk.net  133.100.9.2      2 u   42   64   37    9.673   64.877   7.862
  1. set_rtc_mmss - Google 検索
  2. set_rtc_mmss: can't update from ** to ** - tomisima::tmp - cucumberグループ
  3. ntp.conf - Google 検索
  4. NTPDの設定
  5. ntpq - Google 検索
  6. ntpq出力内容の説明

解決したと思ったら同じようなエラーメッセージがまたでる。マザーボードの電池切れという可能性も否定できなくなってきた。と思ったら、同じ人の記事にntpdで更新していれば問題ないとの記述あり。うちはntpdで同期作業を行っているんだけどなぁ。

  1. hiroshi.in|なーなー、なんでそれでも地球まわるん?
  2. hiroshi.in|なーなー、なんでそれでも地球まわるん?

ntpdを一回終了して、ntpdateで時刻あわせ、hwclockでソフトクロックにハードクロックを同期。この状態でwatch ntpq -pすると、初めにoffset値-15msec程度だったのが数分後にoffsetが4000msecくらいに増加(同じ参照サーバで)。結局再起動してしまったよ。これで解決したとしてもなんだかなぁ。今のところ、再起動前に比べてwatch ntpq -pのoffset値が大きくならない。

ただし、同期が出来ているマシンのntpq -pとは若干異なっている。offsetが比較的大きい。

nfsserver:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+ntp.kiba.net    197.54.93.103    2 u   21   64  377   19.300   35.570   2.853
*www.4koma.com   133.243.238.163  2 u   17   64  377   21.404   35.848   3.424
-211.19.56.141   211.19.56.138    3 u   27   64  377   12.831   34.355   2.384
+suisho.attritio 133.100.9.2      2 u   17   64  377    8.733   36.825   1.484
nfsclient:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*211.19.56.138   192.168.0.5      2 u  652 1024  377   16.197   -1.138   1.616
+doga.jp         133.243.238.243  2 u  136 1024  377   17.160   -1.188   2.200
+ns2.ktroad.net  210.173.160.57   3 u  693 1024  377   17.490   -0.976   1.004
+7c29477b.i-revo 61.114.187.55    2 u  645 1024  377   16.552    0.463   1.151

ソーシャルブックマーク

  1. はてなブックマーク
  2. Google Bookmarks
  3. del.icio.us

ChangeLog

  1. Posted: 2009-04-05T09:00:00+09:00
  2. Modified: 2009-04-05T00:18:24+09:00
  3. Modified: 2009-04-07T16:38:08+09:00
  4. Generated: 2017-11-17T23:10:21+09:00