DevTech101

DevTech101

hadoop

How to configure beeline for hive on remote client

http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html cd /usr/java/latest/jre/lib/security/ ;unzip UnlimitedJCEPolicyJDK7.zip;mv UnlimitedJCEPolicy/* . cd /opt;wget http://archive.cloudera.com/cdh5/cdh/5/hive-0.12.0-cdh5.0.0.tar.gz tar zxf hive-0.12.0-cdh5.0.0.tar.gz;mv hive-0.12.0-cdh5.0.0 hive export HADOOP_HOME=/opt/hadoop-2.6.0-cdh5.4.0 wget wget -O hive-conf.zip http://host.domain.com:7180/cmf/services/5/client-config unzip hive-conf.zip export HIVE_HOME=/export/home/cognos/hive-0.12.0-cdh5.0.0 export HIVE_CONF_DIR=/export/home/cognos/hive-conf alias hive=$HIVE_HOME/bin/hive export HADOOP_USER_CLASSPATH_FIRST=true export PATH=/usr/gnu/bin:$PATH:/export/home/cognos/hive-0.12.0-cdh5.0.0/bin:/opt/hadoop-2.6.0-cdh5.4.0/bin Source https://docs.oracle.com/cd/E63064_01/doc.42/e63062/users.htm#BIGUG328

Hadoop hdfs (BDA) setting extended ACL’s

How to set extended ACL’s hdfs dfs -setfacl -R -m default:user:hive:rwx /data/report_data Check extended ACL’s hadoop fs -getfacl /data/report_data Set the mask to get the full default inherent access hadoop fs -setfacl [-R] -m mask::rwx /data/report_data Reference https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html

Testing HDFS Over WebHDFS By Using CURL

A quick update on hdfs put/get/etc… Simplest solution to upload/download/delete/etc… Use webhdfs. samples are below. ————— Simple Example in one step. Note: In this example, replace node name with any of the 6 nodes [n01-n06] curl -L -i -X PUT -T source_file_name.pdf “http://n03.domain.com:50075/webhdfs/v1/user/usera/dest_file_name.pdf?user.name=usera&op=CREATE&user.name=usera&namenoderpcaddress=bda-cluster1-ns&overwrite=false” # Example You have a source file: solaris_openstack.pdf user.name: usera file dest: …

Testing HDFS Over WebHDFS By Using CURL Read More »