Oracle BDA iPython, Notebook and Jupytar Configuration

How to install iPython/Notebook on an Oracle BDA Install the below packages pip install ibackports.ssl_match_hostname- ipython-1.2.1.tar.gz tornado-3.2.1.tar.gz add parcels in CDH GUI parcel address Remote Parcel Repository URLs donwload, distrubite, activate System startup script /usr/local/notebook/bin/ #!/bin/bash export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/rh/python27/root/usr/lib64 export PATH=/opt/rh/python27/root/usr/bin:$PATH # Notebook export PYSPARK_DRIVER_PYTHON=ipython export PYSPARK_DRIVER_PYTHON_OPTS=”notebook –NotebookApp.open_browser=False –NotebookApp.ip=’*’ –NotebookApp.port=8880″ # Jupyter #export PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/bin/jupyter …

Oracle BDA iPython, Notebook and Jupytar Configuration Read More »

To increase SPARK kyroserializer.buffer.max

To address the errors – like the one below 6 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 2.0 (TID 16, org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 3. To avoid this, increase spark.kryoserializer.buffer.max value. Set In CDH under SPARK, look for spark-defaults.conf, add the below. One of the two values below shuld …

To increase SPARK kyroserializer.buffer.max Read More »

How to configure beeline for hive on remote client cd /usr/java/latest/jre/lib/security/ ;unzip;mv UnlimitedJCEPolicy/* . cd /opt;wget tar zxf hive-0.12.0-cdh5.0.0.tar.gz;mv hive-0.12.0-cdh5.0.0 hive export HADOOP_HOME=/opt/hadoop-2.6.0-cdh5.4.0 wget wget -O unzip export HIVE_HOME=/export/home/cognos/hive-0.12.0-cdh5.0.0 export HIVE_CONF_DIR=/export/home/cognos/hive-conf alias hive=$HIVE_HOME/bin/hive export HADOOP_USER_CLASSPATH_FIRST=true export PATH=/usr/gnu/bin:$PATH:/export/home/cognos/hive-0.12.0-cdh5.0.0/bin:/opt/hadoop-2.6.0-cdh5.4.0/bin Source

How to load/add a jar file in Hive – Installing JSONSerDe in a CDH environment

How to install, configure Json capability in CDH distribution The Cloudera(CDH) distribution is not coming with json capability, in order to use that you will need to add/install your own There are 3 options to add/load a jar file if you use the cloudera(CDH) distribution add yourjar.jar Create an .hiverc, an example in the link …

How to load/add a jar file in Hive – Installing JSONSerDe in a CDH environment Read More »

Hadoop hdfs (BDA) setting extended ACL’s

How to set extended ACL’s hdfs dfs -setfacl -R -m default:user:hive:rwx /data/report_data Check extended ACL’s hadoop fs -getfacl /data/report_data Set the mask to get the full default inherent access hadoop fs -setfacl [-R] -m mask::rwx /data/report_data Reference

How to configure LDAP on Oracle big data appliance(BDA)

First configure yum and install SSSD # Configure yum in the /etc/yum.repos.d/ Get it from Install SSSD yum install sssd sssd-client # yum tip yum clean all yum makecache our SSSD config – /etc/sssd/sssd.conf [sssd] services = nss, pam, sudo, autofs config_file_version = 2 domains = default [domain/default] ldap_id_use_start_tls = False cache_credentials = True …

How to configure LDAP on Oracle big data appliance(BDA) Read More »