GoldenGate for Big Data — Replication to Oracle Object Storage

Oracle Object Storage allow customers to store any type of files in any format. It can be used as a central source of data for Data Science, ETLs and Data Warehouses projects.

For this article we want to store data from one source (MySQL) into Object Storage. To have data in real time we need to deploy 2 GoldenGate Instances. One for GoldenGate for non-oracle (MySQL) and other one GoldenGate for Big Data.

I’m using GoldenGate Marketplace images. If you don’t have access to Oracle Cloud Infrastructure (OCI) you can create free account and explore the different services.

If your source is not MySQL, it’s not a problem you can follow this tutorial and do exactly the same configuration.


  • GoldenGate for Big Data up and running

Preparing GoldenGate for Big Data

  1. Credentials to OCI
  2. Install Java SDK

In case you have already OCI credentials created, you just need to copy and paste on Config file and skip this step. If you don’t have, keep reading.

First we need to generate an API signing key. On GoldenGate Big Data instance:

mkdir .oci#Generate SSH key
openssl genrsa -out ~/.oci/oci_api_key.pem
chmod go-rwx ~/.oci/oci_api_key.pemopenssl rsa -pubout -in ~/.oci/oci_api_key.pem -out ~/.oci/oci_api_key_public.pemcat ~/.oci/oci_api_key_public.pem
#copy the public key

Go to your OCI account and on top right corner click on profile image and after on your user:

Click on Add API Keys and paste the public key, when you add it will show a text like this one:

key_file=#TO DO

Copy the text and go back to GoldenGate for Big Data and add it to a new file (Config):

-bash-4.2$ pwd
vi config

Next step is to install Java SDK:

sudo yum-config-manager --enable ol7_developer
sudo yum install java-oci-sdk


Sending Data from MySQL to Object Storage

GoldenGate Big Data Certificates

Download these 2 certificates:

Go to GG Big Data instance and copy the certificate from /etc/nginx/ogg.pem.

Still on GG Big Data instance, create a new user for the distribution path.

Configuration GoldenGate for MySQL

#you can copy and past from GG Big Data instance hosts file:<private ip> ogg21cbigdata

Go to you GG MySQL instance and click on Certificate Management:

Add a new Client certificate:

Client Certificates

  1. Certificate PEM — Certificate ending in “chain”
  2. Private-key — ogg.pem
  3. CA Certificates — without chain

You must upload all.

For CA Certificate, add the one ending in “chain”.

Now, we need to add the credential on GG MySQL to be used by distribution path. Add the same username and password as you did on GG Big Data.

Go to Distribution Service and add a new distribution path like this one:

Now, it should be created and running:

Creating the Replicat to Object Storage

On parameters file, pick the schema from your source and as a target it can be whatever you. It will create a folder on Object Storage with that name:

On properties add this configuration:

# Properties file for Replicat OBJ
#File Writer Handler Template
#The OCI Event handler
#TODO: Set the OCI properties
gg.eventhandler.oci.namespace=<namespace of Object Storage>
gg.eventhandler.oci.region=eu-frankfurt-1 #your region
gg.eventhandler.oci.compartmentID=<compartmen ID where of Bucket>
gg.eventhandler.oci.bucketMappingTemplate=<The bucket name>
#TODO: Uncomment and set the proxy settings if required.
#TODO: Set the OCI client libraries in the classpath.
#VERY IMPORTANT - include the JAR files of the Oracle Cloud #Infrastructure Java SDK
jvm.bootoptions=-Xmx512m -Xms32m

The parameters “gg.handler.filewriter.inactivityRollInterval” it’s very important since it will create a new file 10 seconds after the last record inserted. You can find more information on this blog post written by Deniz Sendil.

As you can see the files are being created on Object Storage:

I hope this can help you. If you have any doubt, please reach out to me through LinkedIn or Medium.



Opinions expressed are solely my own and do not express the views or opinions of my employer Oracle.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Eloi Lopes

Opinions expressed are solely my own and do not express the views or opinions of my employer Oracle.