Build Loader

In order to build the loader, run following command.

> sbt "project loader" "clean" "assembly"

This will give you s2graph-loader-assembly-X.X.X-SNAPSHOT.jar under loader/target/scala-2.xx/

Source Data Storage Options

For bulk loading, source data can be either in HDFS or a Kafka queue.

For source data in HDFS

provide example run of following step under loader/loader.py

1. subscriber.TransferToHFile

tranfer edge format in TSV into HFile directly.

following is paramter for this job.

parameter index note example
0 input path in hdfs
1 output path in hdfs
2 zkQuorum for target hbase
3 tableName will be used in target hbase
4 dbUrl for s2graph core
5 maxHFilePerRegionServer

2. distcp hfile into production hbase cluster(optional)

3. chmod hfile

4. complete load

hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles #{pathToHFile} #{hbase table name}

results matching ""

    No results matching ""