Easily Upload Large Files to Oracle Object Storage with Swift REST

This post is in a series of blog posts on migrating schemas from an Oracle DB (Cloud DBaaS) to Autonomous Data Warehouse (ADW). This post is focused on uploading the large Data Dump files from the DBaaS compute machine onto Cloud Object Storage using Swift REST.

Quick links to other posts in this series:
Summary Blog Post

1. Export Data into Dump Files
   1. Option 1: Export DB Schemas Using SQL Developer
   2. Option 2: Export DB Schemas Using Data Pump Command
2. Transfer Data Dump Files over to Oracle Cloud Object Storage
   1. Option 1: Swift REST Interface to Upload Files to Oracle Object Storage (this post)
   2. Option 2: OCI CLI Utility to Upload Files to Oracle Object Storage
3. Import Data Dump Files into ADW Using SQL Developer

The IT Leader's Guide to Multicloud Readiness

This guide provides practical key insights and important factors to consider to make informed decisions in your multicloud journey.

Download the Guide

The other approach I explain in the other post uses the OCI CLI utility. This post looks at another approach, which is using Swift REST and is my preferred approach to transfer files from Oracle Compute (for DBaaS) to Oracle Cloud Object Store.

So an alternative to having to install, configure and use OCI CLI is to use the Rest API with a Swift Password (Auth Token). Using a REST curl command, we can upload files to Object Storage. On windows this requires installing curl, however, on Oracle Cloud Compute Linux machines, it is already available. So this approach works well if you want to SSH into an Oracle DBaaS Compute and upload the DB Dump files directly to Cloud Object Storage, without having to download the large DMP files locally. Because with REST we can upload directly from Oracle Cloud Compute to Oracle Object Storage, it is much faster than running OCI CLI from a local machine. While this approach doesn’t require the hassle of installing CLI, doing sophisticated file management (such as breakdown and parallel upload) will require scripting with REST, something that is not necessary with the CLI utility.

To do a file upload using REST:

1. Create a Bucket on Oracle Object Storage
   This bucket belongs to a compartment and serves to hold the files that will be uploaded to Object Storage. When uploading the DB DMP files, you will specify the Namespace and Bucket name for the Object Storage to upload into. So take note of these to use them later.
2. Create an Authorization Token (also referred to as Swift Token) for your user. Following the instructions “To create an auth token”. https://docs.cloud.oracle.com/iaas/Content/Identity/Tasks/managingcredentials.htm#create_swift_password
   Take note of the token string to use in the curl command.
3. Create a pre-Authenticate Request on Object Storage.
   Navigate to the Bucket created in step 1.
   Click on the Create Pre-Authenticated Request Button.
   
   Select Object and enter the name of the Dump file as you want it to be on Object Storage. (If you are uploading multiple files together, select the target as Bucket instead of Object.)
   Select permit Reads and Writes and click Create.
   Take note of the PRE-AUTHENTICATED REQUEST URL to use in the curl command in the next step as the destination filename of the PUT command.
4. Open a command line and run the following curl command:

curl -v -X PUT -u '<user>:<token string from step 2>' --upload-file <local file location> <Pre-authenticated Request URL from step 3>

For example: 
curl -v -X PUT -u 'mazen.manasseh@perficient.com:reFdjuFa)krQ6{<{q' --upload-file /u01/app/oracle/admin/ORCL/dpdump/from_onprem/ITPASQTSTYLX_02.dmp https://objectstorage.us-ashburn-1.oraclecloud.com/p/4sS3kULX33GK9kL0fOwvHYcdYYmHUKE/n/idjuyvnx1qqz/b/MazenADWComp/o/ITPASQTSTYLX_02.dmp

Once done, remember to delete the pre-authenticated Request as it is no longer needed.