安科网

HDFS的基本概念

FireFox

FireFox

2017-10-15

关注 关注
一 块（Block）
HDFS的文件被分成块进行存储，HDFS块的默认大小64MB，块是文件存储处理的逻辑单元
 
二 NameNode
NameNode是管理节点，存放文件元数据
文件与数据块的映射表
数据块与数据节点的映射表

HDFS的基本概念
 
 
三 DataNode
DataNode是HDFS的工作节点，存放数据块

<!--5f39ae17-8c62-4a45-bc43-b32064c9388a:W3siYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImFsaWduIjoibGVmdCIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwiYmxvY2tJZCI6IjMyeWdpczE1MDMyMDA5MzE2ODMiLCJyaWNoVGV4dCI6eyJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlLCJkYXRhIjpbeyJjaGFyIjoi5LiAIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiIgIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiLlnZciLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19LHsiY2hhciI6Iu+8iCIsInN0eWxlcyI6eyJib2xkIjp0cnVlfX0seyJjaGFyIjoiQiIsInN0eWxlcyI6eyJib2xkIjp0cnVlfX0seyJjaGFyIjoibCIsInN0eWxlcyI6eyJib2xkIjp0cnVlfX0seyJjaGFyIjoibyIsInN0eWxlcyI6eyJib2xkIjp0cnVlfX0seyJjaGFyIjoiYyIsInN0eWxlcyI6eyJib2xkIjp0cnVlfX0seyJjaGFyIjoiayIsInN0eWxlcyI6eyJib2xkIjp0cnVlfX0seyJjaGFyIjoi77yJIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fV19fSx7ImJsb2NrVHlwZSI6InBhcmFncmFwaCIsInN0eWxlcyI6eyJhbGlnbiI6ImxlZnQiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowLCJsaW5lLWhlaWdodCI6MS43NX0sImJsb2NrSWQiOiIxMWdzc2ExNTAzMjAxMDU1NDc0IiwicmljaFRleHQiOnsiaXNSaWNoVGV4dCI6dHJ1ZSwia2VlcExpbmVCcmVhayI6dHJ1ZSwiZGF0YSI6W3siY2hhciI6IkgifSx7ImNoYXIiOiJEIn0seyJjaGFyIjoiRiJ9LHsiY2hhciI6IlMifSx7ImNoYXIiOiLnmoQifSx7ImNoYXIiOiLmlocifSx7ImNoYXIiOiLku7YifSx7ImNoYXIiOiLooqsifSx7ImNoYXIiOiLliIYifSx7ImNoYXIiOiLmiJAifSx7ImNoYXIiOiLlnZcifSx7ImNoYXIiOiLov5sifSx7ImNoYXIiOiLooYwifSx7ImNoYXIiOiLlrZgifSx7ImNoYXIiOiLlgqgifSx7ImNoYXIiOiLvvIwifSx7ImNoYXIiOiJIIn0seyJjaGFyIjoiRCJ9LHsiY2hhciI6IkYifSx7ImNoYXIiOiJTIn0seyJjaGFyIjoi5Z2XIn0seyJjaGFyIjoi55qEIn0seyJjaGFyIjoi6buYIn0seyJjaGFyIjoi6K6kIn0seyJjaGFyIjoi5aSnIn0seyJjaGFyIjoi5bCPIn0seyJjaGFyIjoiNiJ9LHsiY2hhciI6IjQifSx7ImNoYXIiOiJNIn0seyJjaGFyIjoiQiJ9LHsiY2hhciI6Iu+8jCJ9LHsiY2hhciI6IuWdlyJ9LHsiY2hhciI6IuaYryJ9LHsiY2hhciI6IuaWhyJ9LHsiY2hhciI6IuS7tiJ9LHsiY2hhciI6IuWtmCJ9LHsiY2hhciI6IuWCqCJ9LHsiY2hhciI6IuWkhCJ9LHsiY2hhciI6IueQhiJ9LHsiY2hhciI6IueahCJ9LHsiY2hhciI6IumAuyJ9LHsiY2hhciI6Iui+kSJ9LHsiY2hhciI6IuWNlSJ9LHsiY2hhciI6IuWFgyJ9XX19LHsiYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImFsaWduIjoibGVmdCIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwiYmxvY2tJZCI6IjM3YWlqaTE1MDMyMDA5NzgwMTIiLCJyaWNoVGV4dCI6eyJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlLCJkYXRhIjpbXX19LHsiYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImFsaWduIjoibGVmdCIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwiYmxvY2tJZCI6IjYxa25hdDE1MDMyMDE5NDA2NzAiLCJyaWNoVGV4dCI6eyJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlLCJkYXRhIjpbeyJjaGFyIjoi5LqMIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiIgIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiJOIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiJhIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiJtIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiJlIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiJOIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiJvIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiJkIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fSx7ImNoYXIiOiJlIiwic3R5bGVzIjp7ImJvbGQiOnRydWV9fV19fSx7ImJsb2NrVHlwZSI6InBhcmFncmFwaCIsInN0eWxlcyI6eyJhbGlnbiI6ImxlZnQiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowLCJsaW5lLWhlaWdodCI6MS43NX0sImJsb2NrSWQiOiI5NWZmcnExNTAzMjAxMTc2MDU5IiwicmljaFRleHQiOnsiaXNSaWNoVGV4dCI6dHJ1ZSwia2VlcExpbmVCcmVhayI6dHJ1ZSwiZGF0YSI6W3siY2hhciI6Ik4ifSx7ImNoYXIiOiJhIn0seyJjaGFyIjoibSJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiJOIn0seyJjaGFyIjoibyJ9LHsiY2hhciI6ImQifSx7ImNoYXIiOiJlIn0seyJjaGFyIjoi5pivIn0seyJjaGFyIjoi566hIn0seyJjaGFyIjoi55CGIn0seyJjaGFyIjoi6IqCIn0seyJjaGFyIjoi54K5In0seyJjaGFyIjoi77yMIn0seyJjaGFyIjoi5a2YIn0seyJjaGFyIjoi5pS+In0seyJjaGFyIjoi5paHIn0seyJjaGFyIjoi5Lu2In0seyJjaGFyIjoi5YWDIn0seyJjaGFyIjoi5pWwIn0seyJjaGFyIjoi5o2uIn1dfX0seyJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsiYWxpZ24iOiJsZWZ0IiwiaW5kZW50IjowLCJ0ZXh0LWluZGVudCI6MCwibGluZS1oZWlnaHQiOjEuNzV9LCJibG9ja0lkIjoiNTRhbnd6MTUwMzIwMTIxNDcxOCIsInJpY2hUZXh0Ijp7ImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWUsImRhdGEiOlt7ImNoYXIiOiLmlocifSx7ImNoYXIiOiLku7YifSx7ImNoYXIiOiLkuI4ifSx7ImNoYXIiOiLmlbAifSx7ImNoYXIiOiLmja4ifSx7ImNoYXIiOiLlnZcifSx7ImNoYXIiOiLnmoQifSx7ImNoYXIiOiLmmKAifSx7ImNoYXIiOiLlsIQifSx7ImNoYXIiOiLooagifV19fSx7ImJsb2NrVHlwZSI6InBhcmFncmFwaCIsInN0eWxlcyI6eyJhbGlnbiI6ImxlZnQiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowLCJsaW5lLWhlaWdodCI6MS43NX0sImJsb2NrSWQiOiI3NXVzY28xNTAzMjAxMjQzNDgzIiwicmljaFRleHQiOnsiaXNSaWNoVGV4dCI6dHJ1ZSwia2VlcExpbmVCcmVhayI6dHJ1ZSwiZGF0YSI6W3siY2hhciI6IuaVsCJ9LHsiY2hhciI6IuaNriJ9LHsiY2hhciI6IuWdlyJ9LHsiY2hhciI6IuS4jiJ9LHsiY2hhciI6IuaVsCJ9LHsiY2hhciI6IuaNriJ9LHsiY2hhciI6IuiKgiJ9LHsiY2hhciI6IueCuSJ9LHsiY2hhciI6IueahCJ9LHsiY2hhciI6IuaYoCJ9LHsiY2hhciI6IuWwhCJ9LHsiY2hhciI6IuihqCJ9XX19LHsiYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImFsaWduIjoibGVmdCIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwiYmxvY2tJZCI6IjY1cHF1cTE1MDMyMDEzNzY2NzEiLCJyaWNoVGV4dCI6eyJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlLCJkYXRhIjpbeyJjaGFyIjoiMSIsInN0eWxlcyI6eyJjb2xvciI6IiNERjQwMkEifX0seyJjaGFyIjoiNSIsInN0eWxlcyI6eyJjb2xvciI6IiNERjQwMkEifX0seyJjaGFyIjoiLiIsInN0eWxlcyI6eyJjb2xvciI6IiNERjQwMkEifX0seyJjaGFyIjoicCIsInN0eWxlcyI6eyJjb2xvciI6IiNERjQwMkEifX0seyJjaGFyIjoibiIsInN0eWxlcyI6eyJjb2xvciI6IiNERjQwMkEifX0seyJjaGFyIjoiZyIsInN0eWxlcyI6eyJjb2xvciI6IiNERjQwMkEifX1dfX0seyJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsiYWxpZ24iOiJsZWZ0IiwiaW5kZW50IjowLCJ0ZXh0LWluZGVudCI6MCwibGluZS1oZWlnaHQiOjEuNzV9LCJibG9ja0lkIjoiMjVpaGhmMTUwMzIwMTk2MjM2MyIsInJpY2hUZXh0Ijp7ImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWUsImRhdGEiOltdfX0seyJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsiYWxpZ24iOiJsZWZ0IiwiaW5kZW50IjowLCJ0ZXh0LWluZGVudCI6MCwibGluZS1oZWlnaHQiOjEuNzV9LCJibG9ja0lkIjoiODBrY3lrMTUwMzIwMDk4MzU5NiIsInJpY2hUZXh0Ijp7ImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWUsImRhdGEiOlt7ImNoYXIiOiLkuIkiLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19LHsiY2hhciI6IiAiLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19LHsiY2hhciI6IkQiLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19LHsiY2hhciI6ImEiLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19LHsiY2hhciI6InQiLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19LHsiY2hhciI6ImEiLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19LHsiY2hhciI6Ik4iLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19LHsiY2hhciI6Im8iLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19LHsiY2hhciI6ImQiLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19LHsiY2hhciI6ImUiLCJzdHlsZXMiOnsiYm9sZCI6dHJ1ZX19XX19LHsiYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImFsaWduIjoibGVmdCIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwiYmxvY2tJZCI6IjEzeXpiaTE1MDMyMDA5NzMyMzAiLCJyaWNoVGV4dCI6eyJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlLCJkYXRhIjpbeyJjaGFyIjoiRCJ9LHsiY2hhciI6ImEifSx7ImNoYXIiOiJ0In0seyJjaGFyIjoiYSJ9LHsiY2hhciI6Ik4ifSx7ImNoYXIiOiJvIn0seyJjaGFyIjoiZCJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiLmmK8ifSx7ImNoYXIiOiJIIn0seyJjaGFyIjoiRCJ9LHsiY2hhciI6IkYifSx7ImNoYXIiOiJTIn0seyJjaGFyIjoi55qEIn0seyJjaGFyIjoi5belIn0seyJjaGFyIjoi5L2cIn0seyJjaGFyIjoi6IqCIn0seyJjaGFyIjoi54K5In0seyJjaGFyIjoi77yMIn0seyJjaGFyIjoi5a2YIn0seyJjaGFyIjoi5pS+In0seyJjaGFyIjoi5pWwIn0seyJjaGFyIjoi5o2uIn0seyJjaGFyIjoi5Z2XIn1dfX0seyJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsiYWxpZ24iOiJsZWZ0IiwiaW5kZW50IjowLCJ0ZXh0LWluZGVudCI6MCwibGluZS1oZWlnaHQiOjEuNzV9LCJibG9ja0lkIjoiNXNibWwxNTAzMjAxODk5NDc1IiwicmljaFRleHQiOnsiaXNSaWNoVGV4dCI6dHJ1ZSwia2VlcExpbmVCcmVhayI6dHJ1ZSwiZGF0YSI6W3siY2hhciI6IjEiLCJzdHlsZXMiOnsiY29sb3IiOiIjREY0MDJBIn19LHsiY2hhciI6IjYiLCJzdHlsZXMiOnsiY29sb3IiOiIjREY0MDJBIn19LHsiY2hhciI6Ii4iLCJzdHlsZXMiOnsiY29sb3IiOiIjREY0MDJBIn19LHsiY2hhciI6InAiLCJzdHlsZXMiOnsiY29sb3IiOiIjREY0MDJBIn19LHsiY2hhciI6Im4iLCJzdHlsZXMiOnsiY29sb3IiOiIjREY0MDJBIn19LHsiY2hhciI6ImciLCJzdHlsZXMiOnsiY29sb3IiOiIjREY0MDJBIn19XX19XQ==-->


HDFS的基本概念
 

hdfs line-height white-space

FireFox

FireFox

0 关注 0 粉丝 0 动态

关注 关注

相关推荐

hadoop伪分布式环境搭建

core-site.xml文件主要配置了访问Hadoop集群的主要信息，其中master代表主机名称，也可以使用IP替换，9000代表端口。外部通过配置的hdfs：//master：9000信息，就可以找到Hadoop集群。hdfs-site.xml配置文

WeiHHH 2020-09-23

HDFS分布式存储中NameNode 和DataNode 有什么区别？

随着互联网不断得突飞猛进，数据就逐渐演变为科技和经济发展的核心。更是对于互联网时代的人类和企业来说，是至关重要的，可能对于普通人来说没有太大影响，但是对于国家和大型企业来说，数据就是其命脉，人工智能就是对数据海量化的最好证明之一。所以，数据存储的稳定在一定

憧憬 2020-08-21

2020年首个存储挖矿项目HDFS是什么？

2020年首个存储挖矿项目HDFS是什么？通过去中心化金融DEFI已经积累了数十亿美元的市值，另一边以Fecoin为代表的存储挖矿项目却一直处于雷声大雨点小的状态，因为Fecoin已经连续跳票数次，对于矿工而言则颇为扫兴，好在HDFS及时出现补上了Feco

ViMan0 2020-08-14

hadoop框架三大组件hdfs、mapreduce、yarn 内容

1）分布式的运算程序往往需要分成至少2个阶段。2）第一个阶段的MapTask并发实例，完全并行运行，互不相干。3）第二个阶段的ReduceTask并发实例互不相干，但是他们的数据依赖于上一个阶段的所有MapTask并发实例的输出。4）MapReduce编程

tomli 2020-07-26

hdfs、hive、hbase的搭建总结

-- 完全分布式文件系统的名称 ：schema ip port -->. -- 分布式文件系统的其他路径的所依赖的一个基础路径，完全分布式不能使用默认值，因为临路径不安全，linux系统在重启时，可能会删除此目录下的内容-->. --

eternityzzy 2020-07-19

hadoop集群的启动与停止

漫长的启动时间…………思考：每次都一个一个节点启动，如果节点数增加到1000个怎么办？早上来了开始一个一个节点启动，到晚上下班刚好完成，下班？这些名称是我的三台机器的主机名，各位请改成自己的主机名！如果集群是第一次启动，需要格式化NameNode，这里使用

飞鸿踏雪0 2020-07-09

Spark RDD

②通过Spark读取外部存储文件，将文件数据转变为RDD。可以从本地磁盘读取，也可以从HDFS读取

adayan0 2020-07-05

HDFS

hdfs dfs -put [-f] [-p] [-l] <本地路径> <目标路径>hdfs dfs -put /opt/a.txt /usr/root//txtdir. 本人已经设置了环境变量，则可以在任意的路径下可直接使用hdf

zzjmay 2020-07-04

hadoop创建目录

//1.vm arguments中添加后面的参数来修改用户 -DHADOOP_USER_NAME=hadoop

硅步至千里 2020-06-25

[AWS][大数据][Hadoop] 使用EMR做大数据分析

创建一个存储桶比如hadoop202006…这里我解释一下Hadoop集群中的一些组件，了解大数据的同学直接忽略就好。Apache Hadoop：在分布式服务器集群上存储海量数据并运行分布式分析应用的开源框架，其核心部件是HDFS与MapReduce。Ap

swazerz 2020-06-22

Hadoop

8.在2008年，Doug加入了Yahoo，开源了Hadoop，在Yahoo期间设计实现了Pig、HBase等框架

HJWZYY 2020-06-21

大数据期末复习重点

系统上安装 jdk环境.基于/usr/local/ 目录下解压安装.注: 可检查目录大小: df-h/usr/local/　　。解压压缩包tar -zxvf jdk-7u80-linux.tar.gz 改名: mv jdk-7u80-linux jdk7　

ViMan0 2020-06-21

HDFS【概述、数据流】

能处理PB级别数据、能处理百万的文件数据量。1）客户端通过Distributed FileSystem模块向NameNode请求上传文件，NameNode检查目标文件是否已存在，父目录是否存在。4）NameNode返回3个DataNode节点，分别为dn1

sujins 2020-06-14

HDFS【shell操作hdfs命令】

-moveFromLocal：从本地剪切粘贴到HDFS. -copyFromLocal：从本地文件系统中拷贝文件到HDFS路径去。-appendToFile：追加一个文件到已经存在的文件末尾。2）-get：等同于copyToLocal，就是从HDFS下载文

sujins 2020-06-14

hdfs读数据流程

1，客户端通过Distributed FileSystem 向namenode请求下载文件，namenode 通过查找元数据，返回文件块所在datanode的地址。2，客户端挑选一台datanode服务器，建立连接，请求读取数据；如果dn异常，则从第二优先

Cloudeep 2020-06-14

hdfs写数据流程

1，客户端向nn请求上传文件，nn检查该文件和父目录是否存在。4，nn根据副本原则，返回给客户端块上传的dn节点信息。5，客户端和dn1建立连接，请求上传数据，dn1接着和dn2建立连接，dn2和dn3建立连接，通信管道就建立完成。6，dn1，dn2，dn

ViMan0 2020-06-14

HDFS【Java API操作】

* conf > 工程下的xxx-site.xml > 集群中xxx-site.xml > 集群中xxx-default.xml. //配置副本数 - 该配置只针对本次的操作有效。* boolean overwrite --如果

sujins 2020-06-13

15.HA高可用

HA即高可用，实现高可用最关键的策略就是消除单点故障。HA严格地来说应该分成各个组件的HA机制，HDFS的HA和YARN的HA。Hadoop2.0之前，在HDFS集群中NameNode存在单点故障。HDFS HA功能通过配置Active/Standby两个

sujins 2020-06-10

Hadoop之hadoop fs和hdfs dfs、hdfs fs三者区别

tackoverflow的解释Hadoop fs：使用面最广，可以操作任何文件系统。hadoop dfs与hdfs dfs：只能操作HDFS文件系统相关，前者已经Deprecated，一般使用后者。Following are the three comma

zzjmay 2020-06-08

flume

Flume是一个分布式、可靠、和高可用的海量日志采集、聚合和传输的系统。a)Source：采集组件，用于跟数据源对接，以获取数据。c)Channel：传输通道组件，用于从source将数据传递到sink. #因为要存到hdfs上，所以下沉组件位hdfs.

zzjmay 2020-06-07
FireFox

FireFox

W3CSchool教程
HTML 教程
CSS 教程
Bootstrap 教程
Javascript 教程
jQuery 教程
后端教程
C 教程
Java 教程
PHP 教程
Python 教程
Go 教程
移动开发
Android 教程
Swift 教程
Kotlin 教程
jQuery Mobile 教程
ionic 教程
关于我们
新闻动态
联系方式
招聘英才
安科实验室
帮助与反馈

安科网(Ancii)，中国第一极客网

Copyright © 2013 - 2019 Ancii.com

京ICP备18063983号 京公网安备11010802014868号