hive>> impyla>> 返回
项目作者： cloudera

项目描述 ：
Impala和Hive的Python DB API 2.0客户端（HiveServer2协议）
高级语言： Python
项目地址: git://github.com/cloudera/impyla.git
创建时间： 2014-04-14T23:52:07Z
项目社区：https://github.com/cloudera/impyla
开源协议：Apache License 2.0
下载


impyla

Python client for HiveServer2 implementations (e.g., Impala, Hive) for
distributed query engines.

For higher-level Impala functionality, including a Pandas-like interface over
distributed data sets, see the Ibis project.

Features

  • HiveServer2 compliant; works with Impala and Hive, including nested data

  • Fully DB API 2.0 (PEP 249)-compliant Python client (similar to
    sqlite or MySQL clients) supporting Python 2.6+ and Python 3.3+.

  • Works with Kerberos, LDAP, SSL

  • SQLAlchemy connector

  • Converter to pandas DataFrame, allowing easy integration into the
    Python data stack (including scikit-learn and
    matplotlib); but see the Ibis project for a richer
    experience

Dependencies

Required:

  • Python 2.7+ or 3.5+

  • six, bitarray

  • thrift==0.16.0

  • thrift_sasl==0.4.3

Optional:

  • kerberos>=1.3.0 for Kerberos over HTTP support. This also requires Kerberos libraries
    to be installed on your system - see System Kerberos

    • On Windows operating systems an alternative is ‘winkerberos’ which can be installed with pip

  • pandas for conversion to DataFrame objects; but see the Ibis project instead

  • sqlalchemy for the SQLAlchemy engine

  • pytest and requests for running tests; unittest2 for testing on Python 2.6

System Kerberos

Different systems require different packages to be installed to enable Kerberos support in
Impyla. Some examples of how to install the packages on different distributions follow.

Ubuntu:

  1. apt-get install libkrb5-dev krb5-user

RHEL/CentOS:

  1. yum install krb5-libs krb5-devel krb5-server krb5-workstation

Installation

Install the latest release with pip:

  1. pip install impyla

For the latest (dev) version, install directly from the repo:

  1. pip install git+https://github.com/cloudera/impyla.git

or clone the repo:

  1. git clone https://github.com/cloudera/impyla.git
  2. cd impyla
  3. python setup.py install

Running the tests

impyla uses the pytest toolchain, and depends on the following
environment variables:

  1. export IMPYLA_TEST_HOST=your.impalad.com
  2. export IMPYLA_TEST_PORT=21050
  3. export IMPYLA_TEST_AUTH_MECH=NOSASL

To run the maximal set of tests, run

  1. cd path/to/impyla
  2. py.test --connect impala

Leave out the --connect option to skip tests for DB API compliance.

To test impyla with different Python versions tox can be used.
The commands below will run all impyla tests with all supported and
installed Python versions:

  1. cd path/to/impyla
  2. tox

To filter environments / tests use -e and pytest arguments after --:

  1. tox -e py310 -- -ktest_utf8_strings

Usage

Impyla implements the Python DB API v2.0 (PEP 249) database interface
(refer to it for API details):

  1. from impala.dbapi import connect
  2. conn = connect(host='my.host.com', port=21050) # auth_mechanism='PLAIN' for unsecured Hive connection, see function doc
  3. cursor = conn.cursor()
  4. cursor.execute('SELECT * FROM mytable LIMIT 100')
  5. print cursor.description  # prints the result set's schema
  6. results = cursor.fetchall()

The Cursor object also exposes the iterator interface, which is buffered
(controlled by cursor.arraysize):

  1. cursor.execute('SELECT * FROM mytable LIMIT 100')
  2. for row in cursor:
  3.     print(row)

Furthermore the Cursor object returns you information about the columns
returned in the query. This is useful to export your data as a csv file.

  1. import csv
  3. cursor.execute('SELECT * FROM mytable LIMIT 100')
  4. columns = [datum[0] for datum in cursor.description]
  5. targetfile = '/tmp/foo.csv'
  7. with open(targetfile, 'w', newline='') as outcsv:
  8.     writer = csv.writer(outcsv, delimiter=',', quotechar='"', quoting=csv.QUOTE_ALL, lineterminator='\n')
  9.     writer.writerow(columns)
  10.     for row in cursor:
  11.         writer.writerow(row)

You can also get back a pandas DataFrame object

  1. from impala.util import as_pandas
  2. df = as_pandas(cur)
  3. # carry df through scikit-learn, for example

How do I contribute code?

You need to first sign and return an
ICLA
and
CCLA
before we can accept and redistribute your contribution. Once these are submitted you are
free to start contributing to impyla. Submit these to CLA@cloudera.com.

Find

We use Github issues to track bugs for this project. Find an issue that you would like to
work on (or file one if you have discovered a new issue!). If no-one is working on it,
assign it to yourself only if you intend to work on it shortly.

It’s a good idea to discuss your intended approach on the issue. You are much more
likely to have your patch reviewed and committed if you’ve already got buy-in from the
impyla community before you start.

Fix

Now start coding! As you are writing your patch, please keep the following things in mind:

First, please include tests with your patch. If your patch adds a feature or fixes a bug
and does not include tests, it will generally not be accepted. If you are unsure how to
write tests for a particular component, please ask on the issue for guidance.

Second, please keep your patch narrowly targeted to the problem described by the issue.
It’s better for everyone if we maintain discipline about the scope of each patch. In
general, if you find a bug while working on a specific feature, file a issue for the bug,
check if you can assign it to yourself and fix it independently of the feature. This helps
us to differentiate between bug fixes and features and allows us to build stable
maintenance releases.

Finally, please write a good, clear commit message, with a short, descriptive title and
a message that is exactly long enough to explain what the problem was, and how it was
fixed.

Please create a pull request on github with your patch.