IDataAPI>> databricks-platform-administration-framework>> 返回
项目作者： kennydataml

项目描述 ：
Framework for end to end databricks platform administration using YAML. Includes cluster management, ACL permissions, and wrapper for SCIM & Permissions REST API.
高级语言： Python
项目地址: git://github.com/kennydataml/databricks-platform-administration-framework.git
创建时间： 2021-02-11T15:50:50Z
项目社区：https://github.com/kennydataml/databricks-platform-administration-framework
开源协议：GNU General Public License v3.0
下载


Databricks DevOps: Platform Administration Framework

Framework for end to end platform administration using YAML configuration files. This includes cluster management and ACL permissions.
The Python code is a wrapper for SCIM and Permissions REST API.

Tech Debt

  • SCIM API is still in-preview. Should switch to AAD integration once it is GA.
  • Permissions API is still in-preview. Update the end point in api.py once it is GA.

    High-Level End to End Deployment

  1. Create Workspace
  2. Create Secret Scopes for workspace (manual)
    • programatically needs AAD token which requires AAD application.
    • also needs KV resource ID, KV DNS name.
  3. create feature branch and configure ACL.yaml per environment
    • requires Service Principal IDs, Secret Scope Names
  4. commit and PR to deployment branch (develop is default)
    • azure pipeline yaml will run cluster.py, then acl.py in that order
    • if pipeline doesn’t exist, Set up build from develop branch using azure pipeline yaml

      Python sys.path

      NOTE : remember to add the databricks folder to python’s sys.path
  • local development: conda develop .
  • azure pipelines: there are 2 options
    1. # option 1 (recommended)
    2. variables:
    3.   - name: PYTHONPATH
    4.     value: "%PYTHONPATH%;$(extra_path)"
    5. # option 2
    6. - script: python script.py
    7.   env:
    8.     PYTHONPATH : "%PYTHONPATH%;$(extra_path)"
  1. .
  2. │   README.md
  3. │   requirements.txt            # required python libs
  4. │   __init__.py
  5. │
  6. ├───databricks_api
  7. │   │   acl.py                  # ACL main script. uses ACL*.yaml
  8. │   │   api.py                  # custom API classes for SCIM and Permissions API
  9. │   │   base.py                 # base super classes
  10. │   │   cluster.py              # cluster management main script. uses clusterconf*.yaml and clusterlib*.yaml
  11. │   │   utils.py                # common utilities
  12. │   │   __init__.py
  13. │   │
  14. │   └───configuration           # can duplicate as necessary
  15. │           ACL.yaml            # ACL configuration
  16. │           clusterconf.yaml    # cluster configuration
  17. │           clusterlib.yaml     # cluster library configuration for all clusters in workspace
  18. │
  19. └───test                        # pytest
  20.         ACL_expected.yaml
  21.         ACL_template.yaml
  22.         test_utils.py
  23.         __init__.py

cluster.py usage

  1. python databricks_api\cluster.py -h
  2. usage: cluster.py [-h] -pat PERSONAL_ACCESS_TOKEN -wu WORKSPACE_URL
  3.                   [-ccf CLUSTER_CONFIG_FILE] [-clf CLUSTER_LIBRARY_FILE]
  5. Databricks Workspace ACL Configuration
  7. optional arguments:
  8.   -h, --help            show this help message and exit
  9.   -pat PERSONAL_ACCESS_TOKEN, --personal_access_token PERSONAL_ACCESS_TOKEN
  10.                         Personal Access Token from Admin Console
  11.   -wu WORKSPACE_URL, --workspace_url WORKSPACE_URL
  12.                         Workspace URL
  13.   -ccf CLUSTER_CONFIG_FILE, --cluster_config_file CLUSTER_CONFIG_FILE
  14.                         Default is clusterconf.yaml
  15.   -clf CLUSTER_LIBRARY_FILE, --cluster_library_file CLUSTER_LIBRARY_FILE
  16.                         Default is clusterlib.yaml

clusterconf.yaml

for ML clusters, specify similar to spark_version: 8.1.x-cpu-ml-scala2.12.
for GPU ML: spark_version: 8.1.x-gpu-ml-scala2.12. Note that GPU spark version does not support credential passthrough.

acl.py usage

  1. python databricks_api\acl.py -h
  2. usage: acl.py [-h] -pat PERSONAL_ACCESS_TOKEN -wu WORKSPACE_URL [-af ACL_FILE]
  4. Databricks Workspace ACL Configuration
  6. optional arguments:
  7.   -h, --help            show this help message and exit
  8.   -pat PERSONAL_ACCESS_TOKEN, --personal_access_token PERSONAL_ACCESS_TOKEN
  9.                         Personal Access Token from Admin Console
  10.   -wu WORKSPACE_URL, --workspace_url WORKSPACE_URL
  11.                         Workspace URL
  12.   -af ACL_FILE, --acl_file ACL_FILE
  13.                         Default is ACL.yaml

Resources