MAXAIO导致Oracle启动hang问题
Oracle数据库,10.2.0.4 for linux x86，在正常重启时，到open阶段僵死。在操作系统上看到一些因计划任务启动的用户进程CPU使用率几乎100%，很明显处于等待状态。在Oracle的bdump目录下也很快生成有trc文件。这些文件的内容关键点是这样：
WARNING:io_submit failed due to kernel limitations MAXAIO for process=0 pending aio=0
WARNING:asynch I/O kernel limits is set at AIO-MAX-NR=65536 AIO-NR=65536
WARNING:Oracle process running out of OS kernel I/O resources (1)
从字面上理解是，是操作系统的MAXAIO限制了Oracle用户进程操作。
查了查资料，又说是bug，但给出了两种解决方法：一，增加操作系统内核参数AIO-MAX-NR的值；二，禁用磁盘AIO机制。我采用了修改系统内核参数AIO-MAX-NR的方法来解决这个问题。
1、可以临时修改内核参数aio-max-nr
# echo > /proc/sys/fs/aio-max-nr 1048576
2、永久修改内核参数aio-max-nr，需要在/etc/sysctl.conf加上下面这句
fs.aio-max-nr = 1048576
用下列命令使参数生效
#/sbin/sysctl -p
附，top显示结果
Tasks: 568 total, 6 running, 562 sleeping, 0 stopped, 0 zombie
Cpu(s): 20.4%us, 0.1%sy, 0.0%ni, 79.1%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 132051284k total, 117157820k used, 14893464k free, 197072k buffers
Swap: 5751260k total, 2404292k used, 3346968k free, 114662552k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12975 oracle 25 0 1687m 25m 19m R 99.8 0.0 9:38.01 ora_p004_oncz
12981 oracle 25 0 1687m 25m 19m R 99.8 0.0 9:38.00 ora_p007_oncz
12983 oracle 25 0 1687m 25m 19m R 99.8 0.0 9:38.01 ora_p008_oncz
12985 oracle 25 0 1687m 25m 19m R 99.8 0.0 9:38.00 ora_p009_oncz
12002 oracle 25 0 1968m 1.6g 1.3g R 90.5 1.3 21:25.03 ora_j000_ofdb
附，bdump目录下的trc文件信息
/u01/app/oracle/admin/oncz/bdump/oncz_p008_12983.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1
System name: Linux
Node name: db-172-17-2-8
Release: 2.6.18-348.el5
Version: #1 SMP Tue Jan 8 17:53:53 EST 2013
Machine: x86_64
Instance name: oncz
Redo thread mounted by this instance: 1
Oracle process number: 29
Unix process pid: 12983, image: oracle@db-172-17-2-8 (P008)
*** SERVICE NAME:() 2013-02-19 15:55:08.764
*** SESSION ID:(142.1) 2013-02-19 15:55:08.764
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Additional information: 3
Additional information: 128
Additional information: 65536
WARNING:io_submit failed due to kernel limitations MAXAIO for process=0 pending aio=0
WARNING:asynch I/O kernel limits is set at AIO-MAX-NR=65536 AIO-NR=65536
WARNING:Oracle process running out of OS kernel I/O resources (1)
WARNING:Oracle process running out of OS kernel I/O resources (1)
WARNING:Oracle process running out of OS kernel I/O resources (1)
WARNING:Oracle process running out of OS kernel I/O resources (1)