Where I left things was a manual process to push the log files from my home automation server into my Hadoop cluster where I created a few hive tables to allow me to quickly query specific types of data (e.g. electricity readings or temperature logs).
This was starting to get a little boring and was taking up valuable time that could be better spent investigating the data rather than preparing it.
Ultimately I want to look at some form of incremental loading however as my data is still relatively small am happy settle for a full load every night.
Enter the next piece of kit in the Hadoop toolbox. Oozie
The process would need to copy the files from the HA Server, load them into the HDFS and then run the scripts to prepare my Hive tables ready for analysis.
First task was to get the file onto my edge node (master01). For details of the manual process I set up click here (doesn work yet!). I had internal push/pull debate in my head.
Push – Create a cron job on my HA server to automatically push the files onto the edge node and then have some form of file watcher that kicks off the load and refresh when the files appear
Pull – Use oozie to completely control it from the Hadoop environment.
I chose pull.. err mainly because I wanted to learn about oozie but also it seemed a simpler option for the full load every night.
1. Set up the oozie account(s)
2. Enable passwordless login
3. Create the script to copy and load the files
4. Create a oozie workflow to call the script
5. Top and tail with Hive queries to create the tables
6. Create the oozie coordinator
7. Job done!
The details
1. Set up the Oozie account(s)
As part of the initial set up a service account was set up to run oozie. By default this didn’t have a home directory or ssh keys generated. If I wanted to get this account to have passwordless login to my HA Server I need to fix this.
master01 as sudo
mkdir /home/oozie
chown oozie /home/oozie
su oozie
ssh-keygen -t rsa -P “”
I didn’t want to use the admin account on my HA server as I had been for the manual refresh so I created an oozie user there as well
haserver as sudo
adduser oozie
mkdir /home/oozie
choown oozie /home/oozie
su oozie
mkdir /home/oozie/.ssh
2. Now need to get the public key of oozie@master01 and add it to the authorized_keys of oozie@haserver
master01 as sudo
cp /home/oozie/.ssh/id_rsa.pub /home/pi/oozie_key.pub
haserver as oozie (in home directory)
scp pi@192.168.1.210:~/oozie_key.pub oozie_key.pub
cat oozie_key.pub >> .ssh/authorized_keys
A quick test on master01 all looks good.
Create copy.sh in the oozie home directory on master0
scp oozie@haserver:/home/zcaprb7/SensorLogs/* /edge/
HADOOP_USER_NAME=hduser /opt/hadoop/bin/hadoop fs -put /edge/*loopenergy.json /user/zcaprb7/loop_jsonfiles/;
HADOOP_USER_NAME=hduser /opt/hadoop/bin/hadoop fs -put /edge/*domoticz.log /user/zcaprb7/domoticz_logfiles/;
rm /edge/*loopenergy.json
rm /edge/*domoticz.log
We can now call this from the oozie workflow by creating a new workflow and adding a ssh task