为Linux版本Oracle 11gR2配置HugePage
HugePage是Oracle在Linux版本下一种性能优化手段。对于共享内存区域（SGA）的数据库系统，Oracle通常都推荐在操作系统层面配置上HugePage，为Oracle实例准备更大的可用共享内存和整块的内存。
本文主要介绍在Oracle 11gR2版本上，进行HugePage配置全过程。记录下来，共需要的朋友待查。
1、环境介绍
笔者使用Oracle 11gR2系统，具体版本为11.2.0.4。
SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE 11.2.0.4.0 Production
TNS for Linux: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 – Production
当前数据库配置内存策略为ASMM，也就是分别设置PGA和SGA的大小范围。Oracle根据负载情况动态自己调节内部各个子池的空间。注意：在当前版本中，AMM（Automatic Memory Management）和HugePage之间还是不能兼容。所以，如果要求配置HugePage，就只能退而求其次选择ASMM。
数据库设置的SGA空间如下：
SQL> show parameter sga;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
lock_sga boolean FALSE
pre_page_sga boolean FALSE
sga_max_size big integer 12G
sga_target big integer 12G
12G的SGA，Oracle在启动过程中，针对大内存情况给出了提示在alert log中。
ue Jan 12 11:31:33 2016
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = 64 KB
Total Shared Global Region in Large Pages = 0 KB (0%)
Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB
RECOMMENDATION:
Total System Global Area size is 12 GB. For optimal performance,
prior to the next instance restart:
1. Increase the number of unused large pages by
at least 6145 (page size 2048 KB, total size 12 GB) system wide to
get 100% of the System Global Area allocated with large pages
2. Large pages are automatically locked into physical memory.
Increase the per process memlock (soft) limit to at least 12 GB to lock
100% System Global Area's large pages into physical memory
首先Oracle发现当前系统并没有设置HugePage，只有默认的2M大小的Page可选。于是，从建议的角度，给出了提示意见，建议开启HugePage选项在操作系统层面。此时，操作系统层面也没有专门的设置。
[oracle@PROD_PC-DB trace]$ grep Huge /proc/meminfo
AnonHugePages: 36864 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
2、配置HugePage
在11.2的小版本中，HugePage处理的行为其实是在变化的。在11.2.0.2中，Oracle推出了一个参数use_large_page，作为从实例层面控制Oracle使用Huge Page的开关。
SQL> show parameter large_pages
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
use_large_pages string TRUE
该参数默认取值为true，是一种向上最优策略。如果设置为true，Oracle实例会去检查是否操作系统开启了HugePage，如果有就采用，如果没有就不用。另外两个取值分别为false和only，如果选择为false，表示Oracle实例不管操作系统设置，都不会使用HugePage。如果选择为only，就表示Oracle实例只会选择HugePage上运行。
此外，在行为上也有一些变化。在11.2.0.2版本中，如果操作系统没有足够的HugePage使用，SGA内存就只有使用系统剩下的小页（Small Pages）。如果余下的小页small page不够用，或者内核使用的比较多，系统会报错ora-4030错误。
这个情况在11.2.0.3版本中有一些变化。Oracle首先会使用可用的HugePage，如果使用尽了，同样会采用剩下的small pages作为SGA使用。变化在于对使用HugePage的方式上，根据官方的说法：
With this new behavior additional shared memory segments are an expected side effect. Part of the change is to ensure that each shared memory segment making up the SGA only contains sub-areas with an identical alignment requirement - hence the SGA will spread over more separate SHM segments. In this supported mixed page mode the database will exhaust the available hugepages, before switching to regular sized pages.
新版本的操作方式可以更好的使用可用的大页资源。当前操作系统版本为Linux 2.6内核版本。
[oracle@PROD_PC-DB trace]$ uname -r
2.6.32-431.el6.x86_64
这个过程中涉及到不同操作系统环境、不同Oracle配置下的推荐HugePage数量问题。Oracle在MOS上发布出一个Shell脚本hugepages_settings.sh，可以动态计算出推荐的HugePage数量。
具体脚本内容可以从MOS Doc ID 401749.1上进行下载，在笔者之前关于HugePage的文章中也有所涉及。执行脚本后结果如下：
[oracle@PROD_PC-DB trace]$ ls -l | grep hugepag
-rw-r--r--. 1 oracle oinstall 3129 Jan 12 13:32 hugepages_settings.sh
[oracle@PROD_PC-DB trace]$ chmod +x hugepages_settings.sh
[oracle@PROD_PC-DB trace]$ ls -l | grep hugepag
-rwxr-xr-x. 1 oracle oinstall 3129 Jan 12 13:32 hugepages_settings.sh
[oracle@PROD_PC-DB trace]$ uname -r
2.6.32-431.el6.x86_64
[oracle@PROD_PC-DB trace]$ ./hugepages_settings.sh
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
* For ASM instance, it needs to configure ASMM instead of AMM.
* The 'pga_aggregate_target' is outside the SGA and
you should accommodate this while calculating SGA size.
* In case you changes the DB SGA size,
as the new SGA will not fit in the previous HugePages configuration,
it had better disable the whole HugePages,
start the DB with new SGA size and run the script again.
And make sure that:
* Oracle Database instance(s) are up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not setup
(See Doc ID 749851.1)
* The shared memory segments can be listed by command:
# ipcs -m
Press Enter to proceed...
Recommended setting: vm.nr_hugepages = 6148
操作系统层面配置HugePage，需要在两个层面进行配置。事先是limits.conf文件，开启对用户的内存限制。
[root@PROD_PC-DB security]# vi limits.conf
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
（篇幅原因，有省略……）
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft memlock 32879476 --对应内存总量
oracle hard memlock 32879476
确认修改。
[oracle@PROD_PC-DB ~]$ ulimit -l
32879476
将脚本计算出的结果，修改入/etc/sysctl.conf文件。
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
fs.file-max = 6815744
vm.nr_hugepages = 6148
"/etc/sysctl.conf" 67L, 1631C written
执行sysctl –p落实修改。
[root@PROD_PC-DB security]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
（篇幅原因，有省略……）
fs.file-max = 6815744
vm.nr_hugepages = 6148
重新启动实例数据库。
SQL> conn / as sysdba
Connected.
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 1.2827E+10 bytes
Fixed Size 2265224 bytes
Variable Size 2214596472 bytes
Database Buffers 1.0603E+10 bytes
Redo Buffers 7307264 bytes
Database mounted.
Database opened.
SQL>
在操作系统层面上，可以看到HugePage使用情况。
[oracle@PROD_PC-DB ~]$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 786435 oracle 640 67108864 22
0x00000000 819204 oracle 640 12817793024 22
0x617ab988 851973 oracle 640 2097152 22
[oracle@PROD_PC-DB ~]$ grep Huge /proc/meminfo
AnonHugePages: 20480 kB
HugePages_Total: 6148
HugePages_Free: 5531
HugePages_Rsvd: 5528
HugePages_Surp: 0
Hugepagesize: 2048 kB
同时，启动过程中的推荐提示信息消失。
Tue Jan 12 14:26:30 2016
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = 31 GB
Total Shared Global Region in Large Pages = 12 GB (100%)
Large Pages used by this instance: 6145 (12 GB)
Large Pages unused system wide = 3 (6144 KB)
Large Pages configured system wide = 6148 (12 GB)
Large Page size = 2048 KB
********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 8
Number of processor cores in the system is 8
Number of processor sockets in the system is 1
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
3、结论
对于投产在Linux版本下的Oracle共享内存系统而言，配置HugePage还是有现实意义的。