Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error...

Symptom: (mysql-5.6.24 with ndb-7.4.6)


[mysql@localhost ndb_data]$ tail -10 ndb_11_out.log
2015-07-02 15:08:42 [ndbd] INFO     -- Grant nodes to start phase: 5, nodes: 0000000000001800
2015-07-02 15:08:42 [ndbd] INFO     -- NDB start phase 4 completed
2015-07-02 15:08:42 [ndbd] INFO     -- As master we will wait for other nodes to reach the state waitpoint52 as well
2015-07-02 15:08:42 [ndbd] INFO     -- Node 12 have reached completion of NDB start phase 4
2015-07-02 15:08:42 [ndbd] INFO     -- Start NDB start phase 5 (only to DBDIH)
2015-07-02 15:08:47 [ndbd] INFO     -- /export/home/pb2/build/sb_0-14878975-1427908966.28/mysql-cluster-com-7.4.6/storage/ndb/src/kernel/blocks/backup/Backup.cpp
2015-07-02 15:08:47 [ndbd] INFO     -- BACKUP (Line: 3846) 0x00000002
2015-07-02 15:08:47 [ndbd] INFO     -- Error handler shutting down system
2015-07-02 15:08:47 [ndbd] INFO     -- Error handler shutdown completed - exiting
2015-07-02 15:08:47 [ndbd] ALERT    -- Node 11: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.



[mysql@localhost ndb_data]$ more ndb_11_error.log
Current byte-offset of file-pointer is: 1067

Time: Thursday 2 July 2015 - 15:08:47
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: Backup.cpp
Error object: BACKUP (Line: 3846) 0x00000002
Program: ndbd
Pid: 25840
Version: mysql-5.6.24 ndb-7.4.6
Trace: /var/local/mysql/server5-ndb/ndb_data/ndb_11_trace.log.2 [t1..t1]
***EOM***

Root Cause:

And I found it was caused by some files inside "datadir" not owned by "mysql" any more, it was changed to "root".
After change permission back to "mysql", this problem resolved. 

[mysql@localhost server2-ndb]$ find . -exec ls -l {} \; |grep root
-rw-r--r--. 1 root  root   1244 Jul  1 15:11 T52F0.Data
-rw-r--r--. 1 root  root   1244 Jul  1 15:11 T52F1.Data
-rw-r--r--. 1 root root 1244 Jul  1 15:11 ./ndb_data/ndb_11_fs/LCP/1/T52F1.Data
-rw-r--r--. 1 root root 1244 Jul  1 15:11 ./ndb_data/ndb_11_fs/LCP/1/T52F0.Data

http://bugs.mysql.com/bug.php?id=70217