Loading data from FTP using OCI Data Integration and OCI Functions
OCI Data Integration is a serverless ETL tool fully managed by Oracle. Right now, doesn’t have an FTP connector out of the box. In this article, I’m going to show you how to move data from FTP using OCI Functions.
We are gonna use a python script to connect to an FTP server, pick a file from there and Upload to OCI Object Storage Bucket, copy the same file to other bucket and delete the file from the first bucket.
If you are not familiarized with OCI Functions, you can follow this quick start. Also, you can find tutorials to get you started with OCI Data Integration.
In order not to have a huge article, I will not post here all policies for both services, they are all well documented. However, you can find it in a README file on my Github.
Deploying Python function on OCI Functions
In OCI console -> Menu -> Developer Services ->Functions you can create your application. Click on the application you created and go to getting started. In this example, I’m using Cloud Shell to deploy my function:
In getting started you will find all commands you need to run. Example:
fn use context <your context>
fn update context oracle.compartment-id ocid1.compartment.oc1.....In case you haven't created a repository, please do it before this step.
fn update context registry fra.ocir.io/<tenant>/repo-eloidocker login -u 'sehubemeaprod/oracleidentitycloudservice/eloi.lopes@oracle.com' fra.ocir.ioIf you don't have a token, please generate one. It's the password for docker. Paste the token.
Login Succeededfn init --runtime python MyFTPFunction
cd MyFTPFunction/
rm func.py
vi func.py<<Paste the code that is available on my github.>>vi requirements.txtPaste both:
fdk>=0.1.36
oci>=2.2.18fn -v deploy --app <your app name>Output:
Deploying myftpfunction to app: Test
Bumped to version 0.0.2
Building image fra.ocir.io/tenancy/repo-eloi/myftpfunction:0.0.2
FN_REGISTRY: fra.ocir.io/tenancy/repo-eloi
Current Context: eu-frankfurt-1
Sending build context to Docker daemon 7.68kB
...
...fn invoke Test myftpfunction{"state": "Uploading new object '<file_name>.csv' in bucket 'bucket-target'"}
Now, we have our function created and available on OCI Function.
Integrating OCI function on OCI Data Integration and Events
Create a dataflow and assign to a task:
My DataFlow:
Before you assign your dataflow to a task and run it. Please, make sure you have these two policies:
allow any-user to use fn-invocation in compartment <compartment> where ALL {request.principal.type=’disworkspace’, request.principal.id=’ocid1.disworkspace.oc1.eu-frankfurt-1…..’}
allow any-user to read fn-function in compartment <compartment>where ALL {request.principal.type=’disworkspace’, request.principal.id=’ocid1.disworkspace.oc1.eu-frankfurt-1….’}
Go to Observability & Management and click on Events. Create a rule like this one:
Click save.
On Attributes, you can specify the task name:
In this case the function is only called when the task name matches with “Task_OCI_Function”.
Please, assign the dataflow to a task and run it.
I hope this can help you. If you have any doubt, please reach out to me through LinkedIn or Medium.
Opinions expressed are solely my own and do not express the views or opinions of my employer Oracle.
