Menu

Chef Supermarket

Adoptable Cookbooks List

Looking for a cookbook to adopt? You can now see a list of cookbooks available for adoption!
List of Adoptable Cookbooks

Supermarket Belongs to the Community

Supermarket belongs to the community. While Chef has the responsibility to keep it running and be stewards of its functionality, what it does and how it works is driven by the community. The chef/supermarket repository will continue to be where development of the Supermarket application takes place. Come be part of shaping the direction of Supermarket by opening issues and pull requests or by joining us on the Chef Mailing List.

View our collection of guides, documentation, and articles

Cookbooks
Advanced Options

Select Badges

Select Supported Platforms

Select Status

RSS

spark-jobserver (1) Versions 0.1.0

Installs/Configures spark-jobserver

Policyfile
Berkshelf
Knife 
cookbook 'spark-jobserver', '~> 0.1.0', :supermarket
cookbook 'spark-jobserver', '~> 0.1.0'
knife supermarket install spark-jobserver
knife supermarket download spark-jobserver
README
Dependencies
Changelog
Quality 17%

spark-jobserver cookbook

Installs the [Spark Jobserver](../spark-jobserver/spark-jobserver).

Dev quickstart:

# minimal required configuration; these values have no defaults and must be provided
chef.json = {
  spark: {
    jobserver: {
      # path to your spark install
      spark_home: '/usr/local/spark',
      spark_conf_dir: '/usr/local/spark/conf',
      # you need to provide the jobserver jar; this gets downloaded, but can be a `file://` uri
      jar_url: 'https://domain.com/some/path/spark-job-server.jar'
    }
  }
}

include_recipe 'spark-jobserver::default'

To use the provided [Vagrantfile](Vagrantfile) to spin up a demo instance:

  1. Copy the Spark distribution binary into the repo root. By default this is assumed to be spark-1.1.0-bin-hadoop2.4.tgz. See below if you wish to use a different Spark distro.
  2. Copy the jobserver jar into the repo root. By default this is assumed to be spark-job-server.jar. Building the jobserver itself is easy: pushd .../spark-jobserver; sbt clean assembly.
  3. Spin up the Vagrant instance: vagrant up
    • If you want to use a different Spark distro, set the SPARK_DIST environment variable (without the extension): SPARK_DIST=spark-1.1.0-bin-cdh4 vagrant up
  4. Now you can browse the Jobserver UI at http://33.33.33.123:8090 and go through the jobserver examples.

Note that this repo assumes RVM; YMMV with other installations such as ChefDK.
The Vagrant demo requires following plugins:

Supported Platforms

Chefspec [tested on](spec/default_helper.rb):

{
  'ubuntu' => [ '12.04', '13.04', '13.10', '14.04' ],
  'debian' => [ '6.0.5' ],
  'centos' => [ '5.8', '6.0', '6.3' ],
  'redhat' => [ '5.8', '6.3' ],
}

Confirmed to work on:

  • Ubuntu 12.04
  • Ubuntu 14.04

With Chef 11.8+

Attributes

See [attributes](attributes/default.rb) for all options.

Usage

The only cookbook dependency is runit; java is assumed to be installed somehow.

Follow the appropriate guides for Spark
standalone or
Mesos deployment.
The relevant attributes are:

  • node.spark.jobserver.master_type
  • node.spark.jobserver.master_url

spark-jobserver::default

Include spark-jobserver::default in your node's run_list:

{
  "run_list": [
    "recipe[spark-jobserver::default]"
  ]
}

Contributing

  1. Fork the repository on Github
  2. Create a named feature branch (i.e. add-new-recipe)
  3. Write your change
  4. Write tests for your change (if applicable)
    • At least a Chefspec proving the fix; Test Kitchen coming later
  5. Run the tests, ensuring they all pass
  6. Submit a Pull Request

License and Authors

Author:: Matt Chu (matt.chu@gmail.com)

Dependent cookbooks

runit >= 0.0.0

Contingent cookbooks

There are no cookbooks that are contingent upon this one.

0.1.0

Initial release of spark-jobserver

Collaborator Number Metric 
            
0.1.0 failed this metric
Failure: Cookbook has 0 collaborators. A cookbook must have at least 2 collaborators to pass this metric.

Contributing File Metric 
            
0.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a CONTRIBUTING.md file

Foodcritic Metric 
            
0.1.0 failed this metric
FC064: Ensure issues_url is set in metadata: spark-jobserver/metadata.rb:1
FC065: Ensure source_url is set in metadata: spark-jobserver/metadata.rb:1
FC066: Ensure chef_version is set in metadata: spark-jobserver/metadata.rb:1
FC067: Ensure at least one platform supported in metadata: spark-jobserver/metadata.rb:1
FC069: Ensure standardized license defined in metadata: spark-jobserver/metadata.rb:1
Run with Foodcritic Version 16.3.0 with tags metadata,correctness ~FC031 ~FC045 and failure tags any

No Binaries Metric 
            
0.1.0 passed this metric

Testing File Metric 
            
0.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a TESTING.md file

Version Tag Metric 
            
0.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must include a tag that matches this cookbook version number

Matt Chu Matt Chu

Details

Updated October 9, 2014 Created on

Supported Platforms

    None Specified

License

Apache v2.0

Download Cookbook