sqooba / hdfs Goto Github PK

Why the error ignored here? If I catch the error, I got it as the title above.

For example executing a command on a file without the correct permission returns the following (cryptic) error message:

./hdfs cat /tmp/donothavepermissions
getBlockLocations call failed with ERROR_APPLICATION (org.apache.hadoop.security.AccessControlException)

It would be nice to have more indicative error messages like:

cat: Permission denied: user=myuser, access=READ, inode="/tmp/donothavepermissions":otheruser:othergroup:-rwxrwx---

log.Fatal might be a bit rough in some cases.

Ok as long as this is only used from CLI, but can be improved.

It would be nice to have a flag to get a structure, parsable output for the ls command, ie: json.

HDFS HA uses what they call nameservice, i.e. a logical name resolving in multiple Namenodes inside the configuration. As of today the library supports HA by providing manually the Namenodes, but this could be handled directly within the configuration

The ls command outputs plain text, which has to be parsed if used as part as another script. This could be avoided (or eased) if the command would, optionally, return a formatted output, such as in JSON.

pipes for the win

Cache TGT's and service tickets in a credential cache.

Note: not sure if gokrb5 supports writing such files, but should not be hard to implement.

When I use this library to operate hdfs, the program is always error, the error is
2018/06/20 10:16:09 create /user/tnt/abc.txt: getFileInfo call failed with FATAL_INVALID_RPC_HEADER (org.apache.hadoop.ipc.RpcServerException)

the code is as follow

func main() {
        os.Setenv("HADOOP_NAMENODE","10.202.45.82:8020")
	os.Setenv("HADOOP_KRB_CONF","krb5.conf")
	os.Setenv("HADOOP_KEYTAB","xue.keytab")
	os.Setenv("HADOOP_SNAME","ds")
	client :=getClient()
	fw,err:=client.Create("/user/tnt/abc.txt")
	if err != nil{
		log.Fatal(err.Error())
	}
	fw.Write([]byte("hehehe"))
	fw.Close()
}
func getClient() *Client {
	nn := os.Getenv("HADOOP_NAMENODE")
	if nn == "" {
		log.Fatal("HADOOP_NAMENODE not set")
	}

	hadoopCfg := LoadHadoopConf("")
	krbClient := GetKrbClientIfRequired(hadoopCfg)

	options := ClientOptions{}

	options.Addresses = []string{nn}
	options.KerberosClient = krbClient
	options.ServicePrincipalName = GetServiceName()

	c, err := NewClient(options)
	if err != nil {
		log.Fatal(err)
	}

	return c
}

Not sure of what it would take, but there is a hidden complete method that might be usable for this.

the dfs.nameservice property is inserted in dfs.namenode.rpc-address.<here>.nn1 in the hdfs-site.xml file.

This makes the code run into no namenode address in configuration

combined with the use of credential cache, this should allow us to run go-hdfs without setting any additional env variable.