机器学习-knowledgecubes-PROSAGA-码农传奇

About

A Knowledge Cube, or KC for short, is a semantically-guided data management architecture, where data semantics influences the data management architecture rather than a predefined scheme. KC relies on semantics to define how the data is fetched, organized, stored, optimized, and queried. Knowledge cubes use RDF to store data. This allows knowledge cubes to store Linked Data from the Web of Data. Knowledge cubes envisions breaking down the centralized architecture into multiple specialized cubes, each having its own index and data store.

Quick Start Guide

Create Encode Data

java -cp uber-knowledgecubes-0.1.0.jar:scala-library-2.11.0.jar edu.purdue.knowledgecubes.DictionaryEncoderCLI -i src/main/resources/datasets/original/sample.nt -o /home/amadkour/kclocal/encoded.nt -l /home/amadkour/kclocal -s space

The kclocal will contain the created dictionaries, the initial data structure used by the store

Create Store

spark-submit --master local[*] --class edu.purdue.knowledgecubes.StoreCLI target/uber-knowledgecubes-0.1.0.jar -i /home/amadkour/kclocal/encoded.nt -l /home/amadkour/kclocal -f 0.01 -t roaring -d /home/amadkour/kcdb

The database kcdb directory contains the actual data and reductions for the input NT file. The following is the directory structure of the local store:

Run Query Workload

spark-submit --master local[*] --class edu.purdue.knowledgecubes.BenchmarkReductionsCLI target/uber-knowledgecubes-0.1.0.jar -l /home/amadkour/kclocal -f 0.01 -t roaring -d /home/amadkour/kcdb -q src/main/resources/queries/original

The command generates the workload reductions under the kcdb/reductions/join directory. The partitions are saved using parquet format.

Local Store Overview

$ ls
amadkour@amadkour:~/kclocal$ ls
GEFI  dbinfo.yaml  dictionary  encoded.nt  join-reductions.yaml  
results-20200625115017.txt  tables.yaml

GEFI: directory represents the generalized filters created for the input datasets.
dbinfo.yaml: file lists meta-data about the store datasets.
dictionary: directory containts the string to id mappings created by the dictionary module.
join-reductions.yaml: directory contains metadata about the generated reductions.
results-20200625115017.txt: is the output file containing the query performance output when running the benchmarking modules.
tables.yaml: file lists the meta-data about the tables.

Database Directory Overview

amadkour@amadkour:~/kcdb$ ls
data  reductions

The database contains parquet formatted files that represents the original data and reductions:

data: contains the original data created based on the input NT files.
reductions: contains the workload-driven reductions created after running a query workload (e.g. after running the Benchmark CLI tool mentioned below).

Program such as spark-shell can be used to view the parquet file content:

scala> var data = spark.read.parquet("/home/amadkour/kcdb/reductions/join/13_TRPO_JOIN_13_TRPS")
data: org.apache.spark.sql.DataFrame = [s: int, p: int ... 1 more field]
scala> data.show()
+---+---+---+
|  s|  p|  o|
+---+---+---+
| 11| 13|  3|
| 11| 13|  4|
| 12| 13|  5|
|  8| 13|  1|
+---+---+---+

WORQ: Workload-Driven RDF Query Processing

KC uses a workload-driven RDF query processing technique, or WORQ for short, for filtering non-matching entries during join evaluation as early as possible to reduce the communication and computation overhead. WORQ generates a reduced sets of triples (or reductions, for short) to represent join pattern(s) of query workloads. WORQ can materialize the reductions on disk or in memory and reuses the reductions that share the same join pattern(s) to answer queries. Furthermore, these reductions are not computed beforehand, but are rather computed in an online fashion. KC also answer complex analytical queries that involve unbound properties. Based on a realization of KC on top of Spark, extensive experimentation demonstrates an order of magnitude enhancement in terms of preprocessing, storage, and query performance compared to the state-of-the-art cloud-based solutions.

Features

A spark-based API for SPARQL querying
Efficient execution of frequent workload join patterns
Materialze workload join patterns in memory or on disk
Efficiently answer unbound property queries

Usage

KC provide spark-based API for issuing RDF related operations. There are three steps necessary for running the system:

Dictionary Encoding
Store Creation
Querying

Dictionary Encoding

KC requires that the dataset be dictionary encoded. The dictionary encoding allows adding resources (subjects or objects) as integers to the filters.

java -cp target/uber-knowledgecubes-0.1.0.jar edu.purdue.knowledgecubes.DictionaryEncoderCLI -i [NT File] -o [Output File] -l [Local Path for the new store] -s space

The command generates a dictionary encoded version of the dataset. This encoded NT file is used for creating the store. KC automatically encodes and decodes SPARQL queries and the corresponding results.

Store Creation

KC provide the Store class for creation of an RDF store. The input to the store is a spark session, database path where the RDF dataset will be stored, and a local configuration path.

import org.apache.spark.sql.SparkSession
import edu.purdue.knowledgecubes.GEFI.GEFIType
import edu.purdue.knowledgecubes.GEFI.join.GEFIJoinCreator
import edu.purdue.knowledgecubes.storage.persistent.Store
val localPath = "/path/to/local/path"
val dbPath = "/path/to/db/path"
val ntPath = "/path/to/rdf/file"
val spark = SparkSession.builder
            .appName(s"KnowledgeCubes Store Creator")
            .getOrCreate()
val store = Store(spark, dbPath, localPath)
store.create(ntPath)

SPARQL Querying

KC provides a SPARQL query processor that takes as input the spark session, database path of where the RDF dataset was created, local configuration file path, a filter type, and a false postivie rate (if any).

import org.apache.spark.sql.SparkSession
import edu.purdue.knowledgecubes.queryprocessor.QueryProcessor
import edu.purdue.knowledgecubes.GEFI.GEFIType
val spark = SparkSession.builder
            .appName(s"Knowledge Cubes Query")
            .getOrCreate()
val localPath = "/path/to/local/path"
val dbPath = "/path/to/db/path"
val filterType = GEFIType.ROARING // Roaring bitmap
val falsePositiveRate = 0
val queryProcessor = QueryProcessor(spark, dbPath, localPath, filterType, falsePositiveRate)
val query =
  """
    SELECT ?GivenName ?FamilyName WHERE{
        ?p <http://yago-knowledge.org/resource/hasGivenName> ?GivenName . 
        ?p <http://yago-knowledge.org/resource/hasFamilyName> ?FamilyName . 
        ?p <http://yago-knowledge.org/resource/wasBornIn> ?city . 
        ?p <http://yago-knowledge.org/resource/hasAcademicAdvisor> ?a .
        ?a <http://yago-knowledge.org/resource/wasBornIn> ?city .
    }
  """.stripMargin
// Returns a Spark DataFrame containing the results
val r = queryProcessor.sparql(query)

Constructing Filters

Additionaly, KC provides an API for creating additional filters. KC provides exact and approximate structures for filtering data. Currently KC supports GEFIType.BLOOM, GEFIType.ROARING, and GEFIType.BITSET.

import org.apache.spark.sql.SparkSession
import edu.purdue.knowledgecubes.GEFI.GEFIType
import edu.purdue.knowledgecubes.GEFI.join.GEFIJoinCreator
import edu.purdue.knowledgecubes.utils.Timer
val spark = SparkSession.builder
            .appName(s"KnowledgeCubes Filter Creator")
            .getOrCreate()
var localPath = "/path/to/db/path"
var dbPath = "/path/to/local/path"
var filterType = GEFIType.ROARING
var fp = 0
val filter = new GEFIJoinCreator(spark, dbPath, localPath)
filter.create(filterType, fp)

Query Execution Benchmarking

KC provides a set of benchmarking classes

BenchmarkFilteringCLI: For benchmarking the query execution when using filters
BenchamrkReductionsCLI: For benchmarking the query execution when using reductions only

Publications

Amgad Madkour, Ahmed M. Ali, Walid G. Aref, “WORQ: Workload-driven RDF Query Processing”, ISWC 2018 [Paper][Slides]
Amgad Madkour, Walid G. Aref, Ahmed M. Aly, “SPARTI: Scalable RDF Data Management Using Query-Centric Semantic Partitioning”, Semantic Big Data (SBD18) [Paper][Slides]
Amgad Madkour, Walid G. Aref, Sunil Prabhakar, Mohamed Ali, Siarhei Bykau, “TrueWeb: A Proposal for Scalable Semantically-Guided Data Management and Truth Finding in Heterogeneous Web Sources”, Semantic Big Data (SBD18) [Paper][Slides]
Amgad Madkour, Walid G. Aref, Saleh Basalamah, “Knowledge Cubes - A Proposal for Scalable and Semantically-Guided Management of Big Data”, IEEE BigData 2013 [Paper][Slides]

Contact

If you have any problems running KC please feel free to send an email.

Amgad Madkour amgad@alumni.purdue.edu