Adoptable Cookbooks List

Looking for a cookbook to adopt? You can now see a list of cookbooks available for adoption!
List of Adoptable Cookbooks

Supermarket Belongs to the Community

Supermarket belongs to the community. While Chef has the responsibility to keep it running and be stewards of its functionality, what it does and how it works is driven by the community. The chef/supermarket repository will continue to be where development of the Supermarket application takes place. Come be part of shaping the direction of Supermarket by opening issues and pull requests or by joining us on the Chef Mailing List.

Select Badges

Select Supported Platforms

Select Status

RSS

flume (3) Versions 3.0.4

Flume: reliable decoupled shipment of logs and data.

Policyfile
Berkshelf
Knife
cookbook 'flume', '~> 3.0.4', :supermarket
cookbook 'flume', '~> 3.0.4'
knife supermarket install flume
knife supermarket download flume
README
Dependencies
Quality 0%

flume chef cookbook

Flume: reliable decoupled shipment of logs and data.

Overview

Cookbook to install flume on a cluster.

Use flume::master to set up a master node. Use flume::node to set up a
physical node. Currently only one physical node per machines.

Configure logical nodes with the logical_node resource - see the test_flow.rb
recipe for an example. This is still somewhat experimental, and some features
will not work as well as they should until chef version 0.9.14 and others until
the next release of flume.

Coming soon flume::xxx_plugin.

Notes

This recipe relies on cluster_discovery_services to determine which nodes
across the cluster act as flume masters, and which nodes provide zookeeper
servers.

Recipes

  • config - Finalizes the config, writes out the config files
  • default - Base configuration for flume
  • hbase_sink_plugin - Hbase Sink Plugin
  • jruby_plugin - Jruby Plugin
  • master - Configures Flume Master, installs and starts service
  • node - Configures Flume Node, installs and starts service
  • test_flow - Test Flow

Integration

Supports platforms: debian and ubuntu

Cookbook dependencies:
* java
* apt
* runit
* volumes
* metachef
* hadoop_cluster

Attributes

  • [:flume][:aws_access_key] - AWS access key used for writing to s3 buckets
  • [:flume][:aws_secret_key] - AWS secret key used for writing to s3 buckets
  • [:flume][:cluster_name] - (default: "cluster_name")
    • The name of the cluster to participate with (masters and zookeepers...)
  • [:flume][:plugins] - Hash for plugin configuration
    • If you have a particular plugin to configure, you can also configure the classpath and the classes to include in the configuration file with attributes in the following forms: node[:flume][:plugin][{plugin_name}][:classes] node[:flume][:plugin][{plugin_name}][:classpath] node[:flume][:plugin][{plugin_name}][:java_opts]
  • [:flume][:classes] -
    • classes to include as plugins
  • [:flume][:classpath] - list of directories and jars to add to the FLUME_CLASSPATH
  • [:flume][:java_opts] - list of command line parameters to add to the jvm
  • [:flume][:collector] - Format of node's logs

    • output_format -- Controls what format the node writes logs (using collectorSink):
      • avro - Avro Native file format. Default currently is uncompressed.
      • avrodata - Binary encoded data written in the avro binary format.
      • avrojson - JSON encoded data generated by avro.
      • default - a debugging format.
      • json - JSON encoded data.
      • log4j - a log4j pattern similar to that used by CDH output pattern.
      • raw - Event body only. This is most similar to copying a file but does not preserve any uniqifying metadata like host/timestamp/nanos.
      • syslog - a syslog like text output format.

    codec -- Controls what kind of compression the collector will use when writing a file.
    whether or not collected logs are gzipped before writing
    them to their final resting place (using collectorSink)
    * GZipCodec
    * BZip2Codec

  • [:flume][:data_dir] - Directory for local in-transit files (default: "/data/db/flume")

  • [:flume][:home_dir] - (default: "/usr/lib/flume")

  • [:flume][:conf_dir] - (default: "/etc/flume/conf")

    • default[:flume][:tmp_dir] = '/mnt/flume/tmp'
  • [:flume][:log_dir] - (default: "/var/log/flume")

  • [:flume][:pid_dir] - (default: "/var/run/flume")

  • [:flume][:master][:external_zookeeper] - false to use flume's zookeeper. True to attach to an external zookeeper. (default: "false")

    • By default, flume installs its own zookeeper instance. With :external_zookeeper to "true", the recipe will work out which machines are in the zookeeper quorum based on cluster membership; modify node[:discovers][:zookeeper_server] to have it use an external cluster
  • [:flume][:master][:zookeeper_port] - port to talk to zookeeper on (for external zookeeper) (default: "2181")

  • [:flume][:master][:run_state] - (default: "stop")

  • [:flume][:node][:run_state] - (default: "start")

License and Author

Author:: Chris Howe - Infochimps, Inc (coders@infochimps.com)
Copyright:: 2011, Chris Howe - Infochimps, Inc

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

readme generated by cluster_chef's cookbook_munger

Dependent cookbooks

java >= 0.0.0
apt >= 0.0.0
runit >= 0.0.0
volumes >= 0.0.0
metachef >= 0.0.0
hadoop_cluster >= 0.0.0

Contingent cookbooks

There are no cookbooks that are contingent upon this one.

Collaborator Number Metric
            

3.0.4 failed this metric

Failure: Cookbook has 0 collaborators. A cookbook must have at least 2 collaborators to pass this metric.

Contributing File Metric
            

3.0.4 failed this metric

Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a CONTRIBUTING.md file

Foodcritic Metric
            

3.0.4 failed this metric

FC016: LWRP does not declare a default action: flume/resources/logical_node.rb:1
FC034: Unused template variables: flume/templates/default/flume-site.xml.erb:1
FC043: Prefer new notification syntax: flume/recipes/master.rb:27
FC043: Prefer new notification syntax: flume/recipes/node.rb:27
FC047: Attribute assignment does not specify precedence: flume/attributes/default.rb:20
FC047: Attribute assignment does not specify precedence: flume/attributes/default.rb:21
FC047: Attribute assignment does not specify precedence: flume/recipes/hbase_sink_plugin.rb:32
FC047: Attribute assignment does not specify precedence: flume/recipes/hbase_sink_plugin.rb:36
FC047: Attribute assignment does not specify precedence: flume/recipes/hbase_sink_plugin.rb:38
FC047: Attribute assignment does not specify precedence: flume/recipes/jruby_plugin.rb:34
FC047: Attribute assignment does not specify precedence: flume/recipes/jruby_plugin.rb:37
FC047: Attribute assignment does not specify precedence: flume/recipes/jruby_plugin.rb:38
FC064: Ensure issues_url is set in metadata: flume/metadata.rb:1
FC065: Ensure source_url is set in metadata: flume/metadata.rb:1
FC066: Ensure chef_version is set in metadata: flume/metadata.rb:1
FC069: Ensure standardized license defined in metadata: flume/metadata.rb:1
FC072: Metadata should not contain "attribute" keyword: flume/metadata.rb:1
FC085: Resource using new_resource.updated_by_last_action to converge resource: flume/providers/logical_node.rb:36
FC085: Resource using new_resource.updated_by_last_action to converge resource: flume/providers/logical_node.rb:49
FC085: Resource using new_resource.updated_by_last_action to converge resource: flume/providers/logical_node.rb:57
FC085: Resource using new_resource.updated_by_last_action to converge resource: flume/providers/logical_node.rb:68
Run with Foodcritic Version 16.3.0 with tags metadata,correctness ~FC031 ~FC045 and failure tags any

No Binaries Metric
            

3.0.4 failed this metric

Failure: Cookbook should not contain binaries. Found:
flume/files/default/jruby-flume.jar
flume/files/default/commons-httpclient-3.0.1.jar
flume/files/default/jets3t-0.6.1.jar
flume/files/default/commons-codec-1.4.jar
flume/files/default/hbase-sink.jar

Testing File Metric
            

3.0.4 failed this metric

Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must contain a TESTING.md file

Version Tag Metric
            

3.0.4 failed this metric

Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of https://github.com/user/repo, and your repo must include a tag that matches this cookbook version number