databricks Cookbook

Configures a Databricks workspace, using the Databricks REST API 2.0.

Requirements

Rubygems

The following gems are installed automatically by your Chef client when using this cookbook, as part of its main dependencies:

• databricks to access the Databricks REST API.

Platforms

The following platforms are supported and tested with Test Kitchen:

TODO: Compile the exact list when Test Kitchen will be fully configured.

Chef

• Chef 15+

Resources

The following resources are available in this cookbook:

• [databricks_file](documentation/databricks_file.md)
• [databricks_job](documentation/databricks_job.md)
• [databricks_instance_pool](documentation/databricks_instance_pool.md)

Usage

This cookbook is resource-based. Use the resources described above in wrapper cookbooks or recipes.

Each resource uses attributes host and token to know the Databricks host to target, and the token used in the REST API queries.

If you want to group all resources accessed under the same host and token, you can use the on_databricks helper that provides a scope to define such resources.
Then resources defined in such a scope will use the given host and token as default values.

Example:

on_databricks('https://my-databricks.net', 'my-token') do

  databricks_file '/FileStore/txt/test.txt' do
    source 'test.txt'
  end

  databricks_job 'Test job' do
    settings(
      new_cluster: {
        spark_version: '7.4.x-scala2.12',
        node_type_id: 'Standard_DS3_v2',
        enable_elastic_disk: true,
        num_workers: 2
      },
      timeout_seconds: 0,
      max_concurrent_runs: 1
    )
  end

end

Development

Development workflow

A standard PR-based development workflow is applied to this repository, using a single master release branch and linear strategy.

If you want to contribute to the code, please create a fork of the repository in your Github's user space, then push a branch with your modifications on your fork and create a PR from it to the master branch on the main repository.

Testing

Lint testing can be done using Chef's delivery tool:

/opt/chef-workstation/bin/delivery local lint

Unit testing can be done using Chefspec.
Several dependent Rubygems are required for unit testing to run:

• webmock
• All cookbook dependencies described in metadata.rb

You can install those dependencies for tests using chef gem install:

/opt/chef-workstation/bin/chef gem install webmock diffy databricks

Then unit tests can be run like this:

/opt/chef-workstation/bin/chef exec rspec

Publishing

Publishing is done using knife supermarket share and requires a publisher's key to work.
It is also needed to have the cookbook directory named databricks for publishing to occur with knife:

/opt/chef-workstation/bin/knife supermarket share databricks --cookbook-path ..