Configuring Kafka SSL and SASL Authentication

Kafka is widely used by countless companies for event streaming, data integration and so on.

The installation of Kafka is quite simple following the documentation you can have you first topic created in less than 30 minutes.

However, if you need to use Kafka in a secure way you will need more than 30 minutes, if you don’t have experience with Kafka. My goal with this article is to help people like me that don’t have tons of experience on Kafka and need to configure SSL or SASL either connect Kafka to with other applications or to have a secure connection.

Let’s get started.

Keystore, Truststore, CA …

The KeyStore is used by the adapter for client authentication, while the TrustStore is used to authenticate a server in SSL authentication. [1]

A KeyStore consists of a database containing a private key and an associated certificate, or an associated certificate chain. The certificate chain consists of the client certificate and one or more certification authority (CA) certificates. [1]

A TrustStore contains only the certificates trusted by the client (a “trust” store). These certificates are CA root certificates, that is, self-signed certificates. [1]

Configuring SSL and SASL

After some researching and troubleshooting, I could configure Kafka using SSL and SASL.

You need to run most of the commands with Keytool. To find Keytool run this command:

sudo find / -name jreYou should find something like this path:cd /usr/java/jdk1.8.0_271-amd64/jre/bin/

Let’s create server keystore. In the SAN=DNS use the FQDN. In this case I’m using a compute instance on Oracle Cloud named developerinstance. Replace by your hostname.

./keytool -keystore /home/opc/kafka/server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA -ext SAN=DNS:developerinstance.subnet.vcn.oraclevcn.com#It will ask you to introduce the following parameters, very important to put the same hostname in first and last nameEnter keystore password:
Re-enter new password:
What is your first and last name?
[Unknown]: developerinstance.subnet.vcn.oraclevcn.com
What is the name of your organizational unit?
[Unknown]: Example
What is the name of your organization?
[Unknown]: Example
What is the name of your City or Locality?
[Unknown]: Example
What is the name of your State or Province?
[Unknown]: Example
What is the two-letter country code for this unit?
[Unknown]: PT
Is CN=developerinstance.subnet.vcn.oraclevcn.com, OU=Example, O=Example, L=Example, ST=Example, C=PT correct?
[no]: yes
Enter key password for <localhost>
(RETURN if same as keystore password):
Re-enter new password:
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /home/opc/kafka/server.keystore.jks -destkeystore /home/opc/kafka/server.keystore.jks -deststoretype pkcs12".
#Step 2./keytool -keystore /home/opc/kafka/kafka.server.keystore.jks -alias localhost -keyalg RSA -genkey#again, it will ask you to fill in the different parameters, very important to put the same hostname
Enter keystore password:
Re-enter new password:
What is your first and last name?
[Unknown]: developerinstance.subnet.vcn.oraclevcn.com
What is the name of your organizational unit?
[Unknown]: Example
What is the name of your organization?
[Unknown]: Example
What is the name of your City or Locality?
[Unknown]: Example
What is the name of your State or Province?
[Unknown]: Example
What is the two-letter country code for this unit?
[Unknown]: PT
Is CN=developerinstance.subnet.vcn.oraclevcn.com, OU=Example, O=Example, L=Example, ST=Example, C=PT correct?
[no]: yes
Enter key password for <localhost>
(RETURN if same as keystore password):
Re-enter new password:
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /home/opc/kafka/kafka.server.keystore.jks -destkeystore /home/opc/kafka/kafka.server.keystore.jks -deststoretype pkcs12".
#Step 3openssl req -new -x509 -keyout ca-key -out ca-cert -days 365#Introduce pass phrase and the same parameters as before.Generating a RSA private key
................+++++
......................................+++++
writing new private key to 'ca-key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:PT
State or Province Name (full name) []:Example
Locality Name (eg, city) [Default City]:Example
Organization Name (eg, company) [Default Company Ltd]:Example
Organizational Unit Name (eg, section) []:Example
Common Name (eg, your name or your server's hostname) []:developerinstance.subnet.vcn.oraclevcn.com
Email Address []:eloi.lopes@example.com
#Step 4./keytool -keystore /home/opc/kafka/kafka.client.truststore.jks -alias CARoot -importcert -file /home/opc/kafka/ca-cert#put the password and say yes#Step 5
./keytool -keystore /home/opc/kafka/kafka.server.truststore.jks -alias CARoot -importcert -file /home/opc/kafka/ca-cert
#put the password and say yes#Step 5./keytool -keystore /home/opc/kafka/kafka.server.keystore.jks -alias localhost -certreq -file /home/opc/kafka/cert-file#Step 6openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:<YOUR_PASS PHRASE>#Step 7./keytool -keystore /home/opc/kafka/kafka.server.keystore.jks -alias CARoot -importcert -file /home/opc/kafka/ca-cert
#put the password and say yes
#Step 8./keytool -keystore /home/opc/kafka/kafka.server.keystore.jks -alias localhost -importcert -file /home/opc/kafka/cert-signed
#put the password and say yes

Create a file named zookeeper_jaas.conf . Create this File in <Kafka home>/config put this code inside:

Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_super="admin-secret"
user_kafka="kafka-secret";
};

Start zookepeeper using these commands:

export KAFKA_OPTS="-Djava.security.auth.login.config=/home/opc/kafka/kafka_2.13-3.0.0/config/zookeeper_jaas.conf"bin/zookeeper-server-start.sh config/zookeeper.properties

Let’s configure Kafka brokers. Edit the <kafka home>/config/server.properties and add these lines:

listeners=PLAINTEXT://:9092,SSL://:9093,SASL_SSL://:9094security.inter.broker.protocol=SSL
ssl.client.auth=required
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.truststore.location=/home/opc/kafka/kafka.server.truststore.jks
ssl.truststore.password=<password>
ssl.keystore.location=/home/opc/kafka/kafka.server.keystore.jks
ssl.keystore.password=<password>
ssl.key.password=<password>
ssl.endpoint.identification.algorithm=
sasl.enabled.mechanisms=PLAIN
allow.everyone.if.no.acl.found=true
authorizer.class.name=kafka.security.authorizer.AclAuthorizer

Now, create a new file named kafka_server_jaas.conf. Create this File in <Kafka home>/config and add this code:

KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafkabroker"
password="kafkabroker-secret"
user_kafkabroker="kafkabroker-secret"
user_kafka-broker-metric-reporter="kafkabroker-metric-reporter-secret"
user_client="client-secret";
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="kafka"
password="kafka-secret";
};

In order to have access to the resource let’s add a super user on server.properties. This superuser is your distinguished name that you gave when you created the certificate. In my case is:

super.users=User:CN=developerinstance.subnet.vcn.oraclevcn.com,OU=Example,O=Example,L=Example,ST=Example,C=PT

Now, start kafka server:

export KAFKA_OPTS=-Djava.security.auth.login.config=/home/opc/kafka/kafka_2.13-3.0.0/config/kafka_server_jaas.confbin/kafka-server-start.sh config/server.properties

You can check if the SSL is configured correctly running this command:

openssl s_client -debug -connect developerinstance.subnet.vcn.oraclevcn.com:9093 -tls1

Configuring Clients

Let’s configure the client certificate for SSL:

./keytool -keystore /home/opc/kafka/client-certificate/client.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA -ext SAN=DNS:developerinstance.subnet.vcn.oraclevcn.com#again, fill in with the same parameters./keytool -keystore /home/opc/kafka/client-certificate/client.keystore.jks -alias localhost -certreq -file /home/opc/kafka/client-certificate/client.unsigned.certopenssl x509 -req -CA /home/opc/kafka/ca-cert -CAkey /home/opc/kafka/ca-key -in /home/opc/kafka/client-certificate/client.unsigned.cert -out /home/opc/kafka/client-certificate/client.signed.cert -days 365 -CAcreateserial#Attention is the pass phrase./keytool -keystore /home/opc/kafka/client-certificate/client.keystore.jks -alias CARoot -import -file /home/opc/kafka/ca-cert./keytool -keystore /home/opc/kafka/client-certificate/client.keystore.jks -alias localhost -import -file /home/opc/kafka/client-certificate/client.signed.cert

Now, create a file, named client_ssl.properties in “<kafka_home>/config” and add these lines:

security.protocol=SSL
ssl.truststore.location=/home/opc/kafka/kafka.client.truststore.jks
ssl.truststore.password=<password>
ssl.keystore.location=/home/opc/kafka/client-certificate/client.keystore.jks
ssl.keystore.password=<password>
ssl.key.password=<password>
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.client.auth=required

Let’s test our configuration for SSL, creating a topic:

bin/kafka-topics.sh --create --topic osa-topic-ssl --partitions 1 --replication-factor 1 --bootstrap-server developerinstance.subnet.vcn.oraclevcn.com:9093 --command-config /home/opc/kafka/kafka_2.13-3.0.0/config/client_ssl.properties#In a new tab run the producer:bin/kafka-console-producer.sh --broker-list developerinstance.subnet.vcn.oraclevcn.com:9093 --topic osa-topic-ssl --producer.config /home/opc/kafka/kafka_2.13-3.0.0/config/client_ssl.properties#Open a new tab and run the consumer:bin/kafka-console-consumer.sh --bootstrap-server developerinstance.subnet.vcn.oraclevcn.com:9093 --topic osa-topic-ssl --consumer.config /home/opc/kafka/kafka_2.13-3.0.0/config/client_ssl.propertiesNow, try to write something on producer tab and should see instantaneously on consumer tab.

For SASL we just need to create a file (client_sasl.properties)with this configuration:

security.protocol=SASL_SSL
ssl.truststore.location=/home/opc/kafka/kafka.client.truststore.jks
ssl.truststore.password=<password>
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="client" \
password="client-secret";

Creating kafka topic:

bin/kafka-topics.sh --create --topic osa-topic-sasl --partitions 1 --replication-factor 1 --bootstrap-server developerinstance.subnet.vcn.oraclevcn.com:9094 --command-config /home/opc/kafka/kafka_2.13-3.0.0/config/client_sasl.propertiesbin/kafka-console-producer.sh --broker-list developerinstance.subnet.vcn.oraclevcn.com:9094 --topic osa-topic-sasl --producer.config /home/opc/kafka/kafka_2.13-3.0.0/config/client_sasl.propertiesbin/kafka-console-consumer.sh --bootstrap-server developerinstance.subnet.vcn.oraclevcn.com:9094 --topic osa-topic-sasl --consumer.config /home/opc/kafka/kafka_2.13-3.0.0/config/client_sasl.properties

You can create a user:

bin/kafka-configs.sh --bootstrap-server developerinstance.subnet.vcn.oraclevcn.com:9094 --alter --add-config 'SCRAM-SHA-512=[password='kafkabroker-secret']' --entity-type users --entity-name kafkabroker --command-config /home/opc/kafka/kafka_2.13-3.0.0/config/client_sasl.properties

I hope that helps beginners in Kafka SSL/SASL. On bottom you have the references that I followed.

I hope this can help you. If you have any doubt, please reach out to me through LinkedIn or Medium.

References

[1] — https://docs.oracle.com/cd/E19509-01/820-3503/6nf1il6ek/index.html

Confluent — https://docs.confluent.io/platform/current/security/security_tutorial.html#configure-clients

Vertica — https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/KafkaIntegrationGuide/TLS-SSL/KafkaTLS-SSLExamplePart3ConfigureKafka.htm?tocpath=Integrating%20with%20Apache%20Kafka%7CUsing%20TLS%2FSSL%20Encryption%20with%20Kafka%7C_____7#4

--

--

Opinions expressed are solely my own and do not express the views or opinions of my employer Oracle. https://www.linkedin.com/in/eloilopes/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Eloi Lopes

Opinions expressed are solely my own and do not express the views or opinions of my employer Oracle. https://www.linkedin.com/in/eloilopes/