Memcached存储机制
主要通过测试，推理memcached的存储机制。
平台 windows7
版本 memcached-1.2.6-win32
启动日志：
E:\memcached\memcached-1.2.6-win32-bin>memcached -m 32 -p 12001 -vv
slab class 1: chunk size 88 perslab 11915 slab class 2: chunk size 112 perslab 9362 slab class 3: chunk size 144 perslab 7281 slab class 4: chunk size 184 perslab 5698 slab class 5: chunk size 232 perslab 4519 slab class 6: chunk size 296 perslab 3542 slab class 7: chunk size 376 perslab 2788 slab class 8: chunk size 472 perslab 2221 slab class 9: chunk size 592 perslab 1771 slab class 10: chunk size 744 perslab 1409 slab class 11: chunk size 936 perslab 1120 slab class 12: chunk size 1176 perslab 891 slab class 13: chunk size 1472 perslab 712 slab class 14: chunk size 1840 perslab 569 slab class 15: chunk size 2304 perslab 455 slab class 16: chunk size 2880 perslab 364 slab class 17: chunk size 3600 perslab 291 slab class 18: chunk size 4504 perslab 232 slab class 19: chunk size 5632 perslab 186 slab class 20: chunk size 7040 perslab 148 slab class 21: chunk size 8800 perslab 119 slab class 22: chunk size 11000 perslab 95 slab class 23: chunk size 13752 perslab 76 slab class 24: chunk size 17192 perslab 60 slab class 25: chunk size 21496 perslab 48 slab class 26: chunk size 26872 perslab 39 slab class 27: chunk size 33592 perslab 31 slab class 28: chunk size 41992 perslab 24 slab class 29: chunk size 52496 perslab 19 slab class 30: chunk size 65624 perslab 15 slab class 31: chunk size 82032 perslab 12 slab class 32: chunk size 102544 perslab 10 slab class 33: chunk size 128184 perslab 8 slab class 34: chunk size 160232 perslab 6 slab class 35: chunk size 200296 perslab 5 slab class 36: chunk size 250376 perslab 4 slab class 37: chunk size 312976 perslab 3 slab class 38: chunk size 391224 perslab 2 slab class 39: chunk size 489032 perslab 2 <96 server listening <112 server listening <116 send buffer was 8192, now 268435456 <116 server listening (udp) <120 new client connection <120 exit >120 ERROR <120 quit <120 connection closed. <120 new client connection
日志只是显示了将来的分配策略，并未真正分配内存。
Slab表示块大小的级别，chunk表示slab中的一个块，还有个比较重要的概念就是page，page表示同一级slab大小的块的集合。
查看数据状态：
stats STAT pid 12212 STAT uptime 10 STAT time 1367559145 STAT version 1.2.6 STAT pointer_size 32 STAT curr_items 0 STAT total_items 0 STAT bytes 0 --已存数据大小 STAT curr_connections 3 STAT total_connections 4 STAT connection_structures 4 STAT cmd_get 0 STAT cmd_set 0 STAT get_hits 0 STAT get_misses 0 STAT evictions 0 STAT bytes_read 11 STAT bytes_written 7 STAT limit_maxbytes 33554432 ---最大容量 STAT threads 1 END
Slab的大小是怎么算出来的呢？
slab大小的计算方式： 从0M到1M，按等比数列划分，默认比例因子是1.25.例如上例中：88*1.25=112.
存数据，测试代码：
public static void main(String[] arg){
byte[] arr=new byte[1024*6]; //使用7k的块
int i=0;
int n=i+140;
for(;i<n;i++){
boolean status=MemcachedUtil.getInstance().set(String.valueOf(System.currentTimeMillis()+""+i), arr,60*60);
if(!status){
System.out.println("------------------------------");
}
System.out.println(status);
}
}结果：
stats slabs STAT 20:chunk_size 7040 --块大小 STAT 20:chunks_per_page 148 ----每个page中块数量 STAT 20:total_pages 1 --------只有一个page STAT 20:total_chunks 148 ---第20个slab级别的块数量总和= total_pages* chunks_per_page STAT 20:used_chunks 148 STAT 20:free_chunks 0 STAT 20:free_chunks_end 8 STAT active_slabs 1 STAT total_malloced 1041920 ---所有数据占用内存=140*(7040+key所占内存) END stats items STAT items:20:number 140 STAT items:20:age 241 STAT items:20:evicted 0 STAT items:20:outofmemory 0 END stats sizes 6240 140 END
使用了第20个slab的块（7k的块），使用了140个，还剩8个。
再执行8次：
int i=0; int n=i+8;
结果：
stats slabs STAT 20:chunk_size 7040 STAT 20:chunks_per_page 148 STAT 20:total_pages 1 STAT 20:total_chunks 148 STAT 20:used_chunks 148 STAT 20:free_chunks 0 STAT 20:free_chunks_end 0 ---第一个page中148个chunk全部占用 STAT active_slabs 1 STAT total_malloced 1041920 END stats items STAT items:20:number 148 STAT items:20:age 407 STAT items:20:evicted 0 STAT items:20:outofmemory 0 END
那么再放一个
7k的块会怎么分配呢？
测试：
int i=0; int n=i+1;
结果：
stats slabs STAT 20:chunk_size 7040 STAT 20:chunks_per_page 148 STAT 20:total_pages 2 -----新加了一个page STAT 20:total_chunks 296 STAT 20:used_chunks 296 STAT 20:free_chunks 0 STAT 20:free_chunks_end 147 STAT active_slabs 1 STAT total_malloced 2083840 END stats items STAT items:20:number 149 ---总过有149个块被使用 STAT items:20:age 1053 STAT items:20:evicted 0 STAT items:20:outofmemory 0 END
那么我用7k的块，把32M内存占完，之后再存数据看会是什么情况：
先测一下多少数据可以把内存沾满，经测试，当到以下状态时，再放数据，total_chunks不会增加，所以可见4736个7k的块可以把内存放满：
stats slabs STAT 20:chunk_size 7040 STAT 20:chunks_per_page 148 STAT 20:total_pages 32 STAT 20:total_chunks 4736 STAT 20:used_chunks 4736 STAT 20:free_chunks 0 STAT 20:free_chunks_end 0 STAT active_slabs 1 STAT total_malloced 33341440 END
修改测试程序：
byte[] arr=new byte[1024*6]; //使用7k的块
MemcachedUtil.getInstance().set("first1", arr,60*60);
MemcachedUtil.getInstance().set("first2", arr,60*60);
MemcachedUtil.getInstance().set("first3", arr,60*60);
MemcachedUtil.getInstance().set("first4", arr,60*60);
int i=1;
int n=i+4732;
for(;i<n;i++){
boolean status=MemcachedUtil.getInstance().set(String.valueOf(i), arr,60*60);
}
MemcachedUtil.getInstance().set("last1", arr,60*60);
MemcachedUtil.getInstance().set("last2", arr,60*60);
MemcachedUtil.getInstance().set("last3", arr,60*60);
MemcachedUtil.getInstance().set("last4", arr,60*60);
System.out.println(MemcachedUtil.getInstance().get("first1"));
System.out.println(MemcachedUtil.getInstance().get("first2"));
System.out.println(MemcachedUtil.getInstance().get("first3"));
System.out.println(MemcachedUtil.getInstance().get("first4"));
System.out.println(MemcachedUtil.getInstance().get("last1"));
System.out.println(MemcachedUtil.getInstance().get("last2"));
System.out.println(MemcachedUtil.getInstance().get("last3"));
System.out.println(MemcachedUtil.getInstance().get("last4"));输出：
null null null null [B@46a5c4 [B@2d09e0 [B@e38fca [B@1f528ab
数据状态：
stats items STAT items:20:number 4736 STAT items:20:age 87 STAT items:20:evicted 4 STAT items:20:outofmemory 0 END
可见，
last把first全部替换出去了。
那么再放一个其他大小块的数据呢？
byte[] arr=new byte[1024*8]; //使用8k的块
MemcachedUtil.getInstance().set("new1", arr,60*60);
System.out.println(MemcachedUtil.getInstance().get("new1"));输出：
[B@1ff0a34
数据状态：
stats slabs =================slab20的状态没变化 STAT 20:chunk_size 7040 STAT 20:chunks_per_page 148 STAT 20:total_pages 32 STAT 20:total_chunks 4736 STAT 20:used_chunks 4736 STAT 20:free_chunks 0 STAT 20:free_chunks_end 0 ============数据被放在了slab21中，新开了一个slab21的page STAT 21:chunk_size 8800 STAT 21:chunks_per_page 119 STAT 21:total_pages 1 STAT 21:total_chunks 119 STAT 21:used_chunks 119 STAT 21:free_chunks 0 STAT 21:free_chunks_end 118 STAT active_slabs 2 STAT total_malloced 34388640 END stats items STAT items:20:number 4736 STAT items:20:age 283 STAT items:20:evicted 4 STAT items:20:outofmemory 0 STAT items:21:number 1 STAT items:21:age 22 STAT items:21:evicted 0 STAT items:21:outofmemory 0 END
那么把slab21的这个page放满之后，如果再放数据会是什么情况呢？是继续新增slab21的page还是做替换呢？：
byte[] arr=new byte[1024*8]; //使用8k的块
int i=1;
int n=i+118;
for(;i<n;i++){
boolean status=MemcachedUtil.getInstance().set("newnew"+String.valueOf(i), arr,60*60);
}
System.out.println(MemcachedUtil.getInstance().get("newnew118"));输出：
[B@c0fc8e
stats slabs STAT 20:chunk_size 7040 STAT 20:chunks_per_page 148 STAT 20:total_pages 32 STAT 20:total_chunks 4736 STAT 20:used_chunks 4736 STAT 20:free_chunks 0 STAT 20:free_chunks_end 0 STAT 21:chunk_size 8800 STAT 21:chunks_per_page 119 STAT 21:total_pages 1 STAT 21:total_chunks 119 STAT 21:used_chunks 119 STAT 21:free_chunks 0 STAT 21:free_chunks_end 0 ---已全部被占用 STAT active_slabs 2 STAT total_malloced 34388640 END stats items STAT items:20:number 4736 STAT items:20:age 579 STAT items:20:evicted 4 STAT items:20:outofmemory 0 STAT items:21:number 119 STAT items:21:age 318 STAT items:21:evicted 0 STAT items:21:outofmemory 0 END
再放一个：
MemcachedUtil.getInstance().set("newnewnew0", arr,60*60);
System.out.println(MemcachedUtil.getInstance().get("newnewnew0"));输出：
[B@1ff0a34
stats slabs STAT 20:chunk_size 7040 STAT 20:chunks_per_page 148 STAT 20:total_pages 32 STAT 20:total_chunks 4736 STAT 20:used_chunks 4736 STAT 20:free_chunks 0 STAT 20:free_chunks_end 0 STAT 21:chunk_size 8800 STAT 21:chunks_per_page 119 STAT 21:total_pages 1 -----page没有增加，还是1 STAT 21:total_chunks 119 STAT 21:used_chunks 119 STAT 21:free_chunks 0 STAT 21:free_chunks_end 0 STAT active_slabs 2 STAT total_malloced 34388640 END stats items STAT items:20:number 4736 STAT items:20:age 725 STAT items:20:evicted 4 STAT items:20:outofmemory 0 STAT items:21:number 119 STAT items:21:age 162 STAT items:21:evicted 1 ----替换掉了一个slab21的块 STAT items:21:outofmemory 0 END
再放一个2k的块也一样：
byte[] arr=new byte[1024*2]; //使用2k的块
MemcachedUtil.getInstance().set("newnewnewnew0", arr,60*60);
System.out.println(MemcachedUtil.getInstance().get("newnewnewnew0"));
stats slabs STAT 15:chunk_size 2304 STAT 15:chunks_per_page 455 STAT 15:total_pages 1 STAT 15:total_chunks 455 STAT 15:used_chunks 455 STAT 15:free_chunks 0 STAT 15:free_chunks_end 454 STAT 20:chunk_size 7040 STAT 20:chunks_per_page 148 STAT 20:total_pages 32 STAT 20:total_chunks 4736 STAT 20:used_chunks 4736 STAT 20:free_chunks 0 STAT 20:free_chunks_end 0 STAT 21:chunk_size 8800 STAT 21:chunks_per_page 119 STAT 21:total_pages 1 STAT 21:total_chunks 119 STAT 21:used_chunks 119 STAT 21:free_chunks 0 STAT 21:free_chunks_end 0 STAT active_slabs 3 STAT total_malloced 35436960 END stats items STAT items:15:number 1 STAT items:15:age 55 STAT items:15:evicted 0 STAT items:15:outofmemory 0 STAT items:20:number 4736 STAT items:20:age 1251 STAT items:20:evicted 4 STAT items:20:outofmemory 0 STAT items:21:number 119 STAT items:21:age 688 STAT items:21:evicted 1 STAT items:21:outofmemory 0 END
可见，当内存被沾满后，如果再存放一种新的slab大小的数据，那么memcached会新开一个且只开一个这种slab大小的page，当这个page被沾满后，就会执行数据替换。
这样算下来，memcached中的实际 数据总量可能就会超出启动时设置的32M，因为当一个slab大小的数据把内存沾满后，还可能为每个其他的slab大小的数据额为分配一个page，每个page的大小是1M。例如，如果有39个slab，那么就可能多占用38M的内存。而启动时 -f 因子的大小会决定slab数量的大小，所以也会影响到最后有多少数据会超出容量。
相关推荐
郗瑞强 2020-08-16
85590296 2020-07-22
jkzyx 2020-06-29
luotuo 2020-06-26
LinuxJob 2020-06-26
ol0 2020-06-26
清溪算法君老号 2020-06-25
86251043 2020-06-13
CSDN0BLOG 2020-06-09
ol0 2020-05-26
andyhuabing 2020-05-22
程序员俱乐部 2020-05-06
83530391 2020-05-05
ol0 2020-05-02
83530391 2020-04-09
85590296 2020-03-25
carolAnn 2020-03-07
大脸猫脸大 2020-03-03
ol0 2020-02-18