The Kite Examples project provides examples of how to use the Kite SDK.
Each example is a standalone Maven module with associated documentation.
datasetis the closest to a HelloWorld example of Kite. It shows how to create datasets and perform streaming writes and reads over them.dataset-hbaseshows how to store entities in HBase using theRandomAccessDatasetAPI.dataset-stagingshows how to use two datasets to manage Parquet-formatted dataloggingis an example of logging events from a command-line programs to Hadoop via Flume, using log4j as the logging API.logging-webappis likelogging, but the logging source is a webapp.
demois a full end-to-end example of a webapp that logs events using Flume and performs session analysis using Crunch and Hive.
The easiest way to run the examples is on the Cloudera QuickStart VM, which has all the necessary Hadoop services pre-installed, configured, and running locally. See the notes below for any initial setup steps you should take.
The current examples run on version 5.1.0 of the QuickStart VM.
Checkout the latest branch of this repository in the VM:
git clone git://github.com/kite-sdk/kite-examples.git
cd kite-examplesThen choose the example you want to try and refer to the README in the relevant subdirectory.
There are two ways to run the examples with the QuickStart VM:
- Logged in to the VM guest (username and password are both
cloudera). - From your host computer.
The advantage of the first approach is that you don't need to install anything extra on your host computer, such as Java or Maven, so there are no fewer set up steps.
For either approach, you need to make the following changes while logged into the VM:
- Sync the system clock For some of the examples it's important that the host and
guest times are in sync. To synchronize the guest, login and type
sudo ntpdate pool.ntp.org. - Configure the NameNode to listen on all interfaces In order to access the cluster from
the host computer, the NameNode must be configured to listen on all network interfaces. This
is done by setting the
dfs.namenode.rpc-bind-hostproperty in/etc/hadoop/conf/hdfs-site.xml:
<property>
<name>dfs.namenode.rpc-bind-host</name>
<value>0.0.0.0</value>
</property>- Configure the History Server to listen on all interfaces In order to access the
cluster from the host computer, the History Server must be configured to listen on all
network interfaces. This is done by setting the
mapreduce.jobhistory.addressproperty in/etc/hadoop/conf/mapred-site.xml:
<property>
<name>mapreduce.jobhistory.address</name>
<value>0.0.0.0:10020</value>
</property>- Configure HBase to listen on all interfaces In order to access the cluster from
the host computer, HBase must be configured to listen on all network interfaces. This
is done by setting the
hbase.master.ipc.addressandhbase.regionserver.ipc.addressproperties in/etc/hbase/conf/hbase-site.xml:
<property>
<name>hbase.master.ipc.address</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hbase.regionserver.ipc.address</name>
<value>0.0.0.0</value>
</property>- Restart the vm Restart the VM with
sudo shutdown -r now
The second approach is preferable when you want to use tools from your own development environment (browser, IDE, command line). However, there are a few extra steps you need to take to configure the QuickStart VM, listed below:
- Add a host entry for quickstart.cloudera Add or edit a line like the following
in
/etc/hostson the host machine
127.0.0.1 localhost.localdomain localhost quickstart.cloudera
- Enable port forwarding Most of the ports that need to be forward are pre-configured
on the QuickStart VM, but there are few that we need to add. For VirtualBox, open
the Settings dialog for the VM, select the Network tab, and click the Port Forwarding
button. Map the following ports - in each case the host port and the guest port
should be the same. Also, your VM should not be running when you are making these changes.
- 8032 (YARN ResourceManager)
- 10020 (MapReduce JobHistoryServer)
If you have VBoxManage installed on your host machine, you can do this via command line as well. In bash, this would look something like:
# Set VM_NAME to the name of your VM as it appears in VirtualBox
VM_NAME="QuickStart VM"
PORTS="8032 10020"
for port in $PORTS; do
VBoxManage modifyvm "$VM_NAME" --natpf1 "Rule $port,tcp,,$port,,$port"
doneSome of the examples include integration tests. You can run them all with the following command:
for module in $(ls -d -- */); do
(cd $module; mvn clean verify; if [ $? -ne 0 ]; then break; fi)
done-
What are the usernames/passwords for the VM?
- Cloudera manager: cloudera/cloudera
- HUE: cloudera/cloudera
- Login: cloudera/cloudera
-
I can't find the file in VirtualBox (or VMWare)!
- You probably need to unpack it: In Windows, install 7zip, and extract the
VM files from the
.7zfile. In linux or mac,cdto where you copied the file and run7zr e cloudera-quickstart-vm-4.3.0-kite-vbox-4.4.0.7z - You should be able to import the extracted files to VirtualBox or VMWare
- You probably need to unpack it: In Windows, install 7zip, and extract the
VM files from the
-
How do I open a
.ovffile?- Install and open VirtualBox on your computer
- Under the menu "File", select "Import..."
- Navigate to where you unpacked the
.ovffile and select it
-
What is a
.vmdkfile?- The
.vmdkfile is the virtual machine disk image that accompanies a.ovffile, which is a portable VM description.
- The
-
How do I open a
.vboxfile?- Install and open VirtualBox on your computer
- Under the menu "Machine", select "Add..."
- Navigate to where you unpacked the
.vboxfile and select it
-
How do I fix "VTx" errors?
- Reboot your computer and enter BIOS
- Find the "Virtualization" settings, usually under "Security" and enable all of the virtualization options
-
How do I get my mouse back?
- If your mouse/keyboard is stuck in the VM (captured), you can usually
release it by pressing the right
CTRLkey. If you don't have one (or that didn't work), then the release key will be in the lower-right of the VirtualBox window
- If your mouse/keyboard is stuck in the VM (captured), you can usually
release it by pressing the right
-
Other problems
- Using VirtualBox? Try using VMWare.
- Using VMWare? Try using VirtualBox.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent