Easily Upload Large Files to Oracle Object Storage with Swift REST
Blog
  • Topics
  • Industries
  • Partners

Explore

Topics

Industries

Partners

Easily Upload Large Files to Oracle Object Storage with Swift REST

This post is in a series of blog posts on migrating schemas from an Oracle DB (Cloud DBaaS) to Autonomous Data Warehouse (ADW). This post is focused on uploading the large Data Dump files from the DBaaS compute machine onto Cloud Object Storage using Swift REST.

Quick links to other posts in this series:
Summary Blog Post

  1. Export Data into Dump Files
    1. Option 1: Export DB Schemas Using SQL Developer
    2. Option 2: Export DB Schemas Using Data Pump Command
  2. Transfer Data Dump Files over to Oracle Cloud Object Storage
    1. Option 1: Swift REST Interface to Upload Files to Oracle Object Storage (this post)
    2. Option 2: OCI CLI Utility to Upload Files to Oracle Object Storage
  3. Import Data Dump Files into ADW Using SQL Developer

The other approach I explain in the other post uses the OCI CLI utility. This post looks at another approach, which is using Swift REST and is my preferred approach to transfer files from Oracle Compute (for DBaaS) to Oracle Cloud Object Store.

So an alternative to having to install, configure and use OCI CLI is to use the Rest API with a Swift Password (Auth Token). Using a REST curl command, we can upload files to Object Storage. On windows this requires installing curl, however, on Oracle Cloud Compute Linux machines, it is already available. So this approach works well if you want to SSH into an Oracle DBaaS Compute and upload the DB Dump files directly to Cloud Object Storage, without having to download the large DMP files locally. Because with REST we can upload directly from Oracle Cloud Compute to Oracle Object Storage, it is much faster than running OCI CLI from a local machine. While this approach doesn’t require the hassle of installing CLI, doing sophisticated file management (such as breakdown and parallel upload) will require scripting with REST, something that is not necessary with the CLI utility.

To do a file upload using REST:

  1. Create a Bucket on Oracle Object Storage
    This bucket belongs to a compartment and serves to hold the files that will be uploaded to Object Storage. When uploading the DB DMP files, you will specify the Namespace and Bucket name for the Object Storage to upload into. So take note of these to use them later.
  2. Create an Authorization Token (also referred to as Swift Token) for your user. Following the instructions “To create an auth token”. https://docs.cloud.oracle.com/iaas/Content/Identity/Tasks/managingcredentials.htm#create_swift_password
    Take note of the token string to use in the curl command.
  3. Create a pre-Authenticate Request on Object Storage.
    Navigate to the Bucket created in step 1.Swift 1
    Click on the Create Pre-Authenticated Request Button.
    Swift 2
    Select Object and enter the name of the Dump file as you want it to be on Object Storage. (If you are uploading multiple files together, select the target as Bucket instead of Object.)
    Select permit Reads and Writes and click Create.
    Take note of the PRE-AUTHENTICATED REQUEST URL to use in the curl command in the next step as the destination filename of the PUT command.
  4. Open a command line and run the following curl command:

curl -v -X PUT -u ‘<user>:<token string from step 2>’ –upload-file <local file location> <Pre-authenticated Request URL from step 3>

For example:
curl -v -X PUT -u ‘mazen.manasseh@perficient.com:reFdjuFa)krQ6{<{q’ –upload-file /u01/app/oracle/admin/ORCL/dpdump/from_onprem/ITPASQTSTYLX_02.dmp https://objectstorage.us-ashburn-1.oraclecloud.com/p/4sS3kULX33GK9kL0fOwvHYcdYYmHUKE/n/idjuyvnx1qqz/b/MazenADWComp/o/ITPASQTSTYLX_02.dmp

Once done, remember to delete the pre-authenticated Request as it is no longer needed.

 

2 thoughts on “Easily Upload Large Files to Oracle Object Storage with Swift REST

  1. Juan David Chica Londoño

    Is there a file size limit to upload using REST? i got this error, my dump file size is 380 GB

    {“code”:”RequestEntityTooLarge”,”message”:”The Content-Length must be less than 53687091200 bytes (it was ‘397539438592’)”}* Closing connection 0
    * schannel: shutting down SSL/TLS connection with objectstorage.us-ashburn-1.oraclecloud.com port 443
    * schannel: clear security context handle

  2. Mazen Manasseh Post author

    That’s a good point. The limit on OOS is 50GiB for each object. So if your file is larger than 50GiB, what you want to do is break it down into smaller segments so that each segment is not more than 50GiB. The maximum total file size of all the related segments that make up one file can be up to 10TiB. You want to use a multi-part upload approach. Following has more info on this: https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/usingmultipartuploads.htm

    In your case, either regenerate the DMP file and make sure that while doing the export, specify that each dump file has a max size of 50GB. This way the export will automatically generate the segments in separate smaller files. If you don’t want to regenerate the data dump export, there are ways to segment the already existing 380GB file into smaller segments. You may want to check the linux split command if that’s what you prefer to do, instead of re-exporting the data dump.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to the Weekly Blog Digest:

Sign Up