hadoop 初学者在eclipse中开发的问题
那最简单的IO说吧 在本地文件系统和HDFS文件系统中的IO操作
下面这个例子是合并本地文件系统的目录下的文件 到HDFS文件系统
public static void main(String[] args) throws IOException, URISyntaxException {
//Hadoop resource configuration class
Configuration conf=new Configuration();
//get FileSystem's instance
//call factory method FileSystem.get(Configuration conf)
FileSystem hdfs=FileSystem.get(new URI("hdfs://127.0.0.1:9000"),conf);
//local FileSystem's instance
FileSystem local =FileSystem.getLocal(conf);
//Path Object encode file and direcotory
Path inputDir=new Path("/home/hadoop/Documents"); //(args[0]);
Path hdfsFile=new Path("/mytest_example2.txt");
//use the FileSystem's listStatus()method() to get a list of files in a directory
FileStatus[] inputFiles=local.listStatus(inputDir);
//create FSDataOutputStream
FSDataOutputStream out=hdfs.create(hdfsFile);
//loop local files and merge a new file
for(int i=0;i<inputFiles.length;i++)
{
//user the Path to request an FSDataInputStream object for reading in the file
System.err.println(inputFiles[i].getPath().getName()+"-----------------");
FSDataInputStream in =local.open(inputFiles[i].getPath());
byte [] buffer=new byte[256];
int bytesRead=0;
while ((bytesRead=in.read(buffer))>0)
{
out.write(buffer, 0, bytesRead);
}
in.close();
}
out.close();
}
以上的代码是可行的没有什么错误。但是如果是一下这样，就会报错前提呢在你的项目的classpath下也就是bin目录下没有hadoop的HDFS配置文件（core-site.xml和hdfs-site.xml）
FileSystem hdfs=FileSystem.get(/*new URI("hdfs://127.0.0.1:9000"),*/conf);
Path hdfsFile=new Path("hdfs://127.0.0.1:9000/mytest_example2.txt");
错误提示：
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://127.0.0.1:9000/, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:513)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:499)
at org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:594)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:448)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776)
at com.liweichao.PutMerge.main(PutMerge.java:40)
解决途径一个是在初始化Filesystem的时候
1 ： Configuration conf=new Configuration();
//get FileSystem's instance
//call factory method FileSystem.get(Configuration conf)
//就会自动加载到hadoop的资源配置文件
FileSystem hdfs=FileSystem.get(new URI("hdfs://127.0.0.1:9000"),conf);
在有Path对象定位的具体目录
Path hdfsFile=new Path("/mytest_example2.txt");
2：在你的项目的classpath下也就是bin目录下没有hadoop的HDFS配置文件（core-site.xml和hdfs-site.xml）