Adoptable Cookbooks List

Looking for a cookbook to adopt? You can now see a list of cookbooks available for adoption!
List of Adoptable Cookbooks

Supermarket Belongs to the Community

Supermarket belongs to the community. While Chef has the responsibility to keep it running and be stewards of its functionality, what it does and how it works is driven by the community. The chef/supermarket repository will continue to be where development of the Supermarket application takes place. Come be part of shaping the direction of Supermarket by opening issues and pull requests or by joining us on the Chef Mailing List.

Select Badges

Select Supported Platforms

Select Status

RSS

confluent-platform (18) Versions 1.2.0

Install and configure Confluent platform (Kafka)

Policyfile
Berkshelf
Knife
cookbook 'confluent-platform', '= 1.2.0', :supermarket
cookbook 'confluent-platform', '= 1.2.0'
knife supermarket install confluent-platform
knife supermarket download confluent-platform
README
Dependencies
Changelog
Quality 100%

Confluent Platform

Description

Apache Kafka, an open source technology created and maintained by the founders
of Confluent, acts as a real-time, fault tolerant, highly scalable messaging
system. It is widely adopted for use cases ranging from collecting user
activity data, logs, application metrics, stock ticker data, and device
instrumentation. Its key strength is its ability to make high volume data
available as a real-time stream for consumption in systems with very different
requirements—from batch systems like Hadoop, to real-time systems that require
low-latency access, to stream processing engines that transform the data
streams as they arrive.

This infrastructure lets you build around a single central nervous system
transmitting messages to all the different systems and applications within your
company. Learn more on http://confluent.io.

This cookbook focuses on deploying Confluent Platform elements on your clusters
via Chef on systemd managed distributions. At the moment, this includes
Kafka, Schema Registry and Kafka Rest.

Requirements

Cookbooks and gems

Declared in [metadata.rb](metadata.rb) and in [Gemfile](Gemfile).

Platforms

A systemd managed distribution:
- RHEL Family 7, tested on Centos

Note: it should work fine on Debian 8 but the official docker image does not
allow systemd to work easily, so it could not be tested.

Usage

Easy Setup

Default recipe does nothing. Each service Kafka, Schema Registry or
Kafka Rest will be installed by calling respectively recipe
[install-kafka](recipes/install-kafka.rb),
[install-registry](recipes/install-registry.rb) and
[install-rest](recipes/install-rest.rb).

By default, this cookbook installs openjdk from the official repositories
(openjdk 8 on centos 7) in services recipe, just before
launching the service. You can deactivate this behavior by setting
node['confluent-platform']['java'] to "", or choose your package by setting
the package name in node['confluent-platform']['java'][node[:platform]].

The recommended way to use this cookbook is through the creation of a different
role per cluster, that is a role for Kafka, Schema Registry and
Kafka Rest. This enables the search by role feature, allowing a simple
service discovery.

See [roles](test/integration/roles) for some examples and Cluster Search
documentation for more information.

Test

This cookbook is fully tested through the installation of the full platform
in docker hosts. This uses kitchen, docker and some monkey-patching.

If you run kitchen list, you will see 7 suites:
- dnsdock-centos-7
- zookeeper-centos-7
- kafka-01-centos-7
- kafka-02-centos-7
- kafka-03-centos-7
- registry-01-centos-7
- rest-01-centos-7

Each corresponds to a different node in the cluster.

For more information, see [.kitchen.yml](.kitchen.yml) and [test](test)
directory.

Local cluster

Of course, the cluster you install by running kitchen converge is fully
working so you can use it as a local cluster to test your development (like a
new Kafka client). Moreover, compared to a single node cluster usually
installed on workstations, you can detected partition/timing/fault-tolerance
issues you could not because of the simplicity of a single-node system.

You can access it by adding the dnsdock used in the cluster as your main DNS
resolver: add
docker inspect --format '{{.NetworkSettings.IPAddress}}' dnsdock-kafka
in /etc/resolv.conf.

Then to produce some messages:

kafka-console-producer.sh \
  --broker-list kafka-kitchen-01.kitchen:9092 \
  --topic my_topic

And to read them:

kafka-console-consumer.sh \
  --zookeeper zookeeper-kafka.kitchen/kafka-kitchen \
  --topic my_topic \
  --from-beginning

Or you can use Rest API with http://rest-kitchen-01.kitchen:8082 and full
Schema Registry support, located at http://registry-kitchen-01.kitchen:8081.

Attributes

Configuration is done by overriding default attributes. All configuration keys
have a default defined in [attributes/default.rb](attributes/default.rb).
Please read it to have a comprehensive view of what and how you can configure
this cookbook behavior.

Recipes

default

Does nothing.

repository

Configure confluent repository.

install-service

Install and fully configure a given service by running repository and its
4 dedicated recipes: package, user, config and service, in that order.

service-package

Install given service from confluent repository.

service-user

Create given service system user and group.

service-config

Generate service configuration. May search for dependencies (like Zookeeper
or other nodes of the same cluster) with the help of cluster-search cookbook.

service-service

Install systemd unit for the given service, then enable and start it.

Note: install java package (OpenJDK 8 on centos 7) by default, can be
disabled by setting node['confluent-platform']['java'] to "". A platform
specific configuration for the package to install is also possible.

Resources/Providers

None.

Changelog

Available in [CHANGELOG](CHANGELOG).

Contributing

Please read carefully [CONTRIBUTING.md](CONTRIBUTING.md) before making a merge
request.

License and Author

Copyright:: 2015, Sam4Mobile

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Dependent cookbooks

cluster-search >= 0.0.0
yum >= 0.0.0

Contingent cookbooks

There are no cookbooks that are contingent upon this one.

Changelog ========= 1.2.0 ----- Main: - Clarify and fix JVM options for services - Use to_hash instead of dup to work on node values - Improve readibility of default system user names Fixes: - Fix and clean the creation of Kafka work directories - Fix zookeeper.connect chroot path Test: - Rationalize docker provision to limit images - Fix typo in roles/rest-kitchen.json name - Wait 15s after registry start to strengthen tests Packaging: - Reorganize README: + Move changelog from README to CHANGELOG + Move contribution guide to CONTRIBUTING.md + Reorder README, fix Gemfile missing - Add Apache 2 license file - Add missing chefignore - Fix long lines in rest and registry templates 1.1.0 ----- - Cleaning, use only dependencies from supermarket 1.0.1 ----- - Set java-1.8.0-openjdk-headless as default java package 1.0.0 ----- - Initial version with Centos 7 support

Foodcritic Metric
            

1.2.0 passed this metric