Mention the word “PST” around an Exchange admin and you’ll likely receive the same gloomy expression as if you mentioned “Public Folders”. PSTs, however, cause headaches for many others outside of the Exchange administration world. They’re even a pain for the users that create them as they can be wiped away with the reimaging of a workstation or failure of a hard drive. From a legal eDiscovery standpoint, they’re a nightmare and they make data security, well, not so secure.
During the course of a mail migration, it’s not uncommon to find that users have accumulated gigabytes or possibly terabytes of PST data. In some cases they’ve been forced to as we’ve enacted artificially small mailbox quotas. So the user takes their emails off the mail server and places them into a PST on the file server instead.
Few will argue that PSTs should not be used but what do we do with the PSTs that we have?
Microsoft has a new offering in Office 365 that will hopefully provide an answer.
Update: 08/15/2016
If you’re interested in using PowerShell to import the PSTs, check out my latest post on this topic: Office 365 – How To Import PSTs From Azure Storage Via PowerShell
Up Until Now
Previously there were a number of third-party tools available or even a tool from Microsoft. I always found the old Microsoft “PST Capture Tool” pretty painful to use and quite buggy. If there were small quantities of PSTs that needed to be imported, you could probably make it work but it wasn’t great for anything large scale. Another sort of knock against most of the previous options is that you were uploading the PSTs over the Internet which could be time consuming depending on the amount of data or your network speed.
Office 365 Import Service
A feature that has showed up in “first release” tenants is the new “Office 365 Import Service”. This feature has the following description on the Office 365 Roadmap:
This feature is one that I think will really become useful for organizations with large quantities of PSTs.
Testing It Out
While the import options are to either ship a drive to Microsoft or upload via the Internet, I only used the upload option in my test. Once the data is in Azure, the process should be relatively similar.
At a high level, the process is as follows:
- Consolidate PSTs into a network share
- Download and install the “Microsoft Azure Storage Tools”
- Obtain your unique upload URL and key
- Upload the PST data using “azcopy.exe” from the “Microsoft Azure Storage Tools”
- Create a mapping file using the supplied sample
- Import the data into mailboxes
The first step is to locate your PSTs and know who they belong to, this is where some of the third-party tools might have an advantage. For my test, I dumped three PSTs into a share called “PST”.
The “Microsoft Azure Storage Tools” can be found as a link within the import service or downloaded at: http://aka.ms/downloadazcopy
You will see the Office 365 Import Service as a link off the main admin page in the portal. If you don’t have this link then the feature has not been rolled out to your tenant just yet. Keep in mind that the tenant I’m using has “first release” enabled which generally receives new features a couple weeks in advance.
When selecting the “import” link, you will see options for creating a new job, obtaining your URL and key and viewing the shipping address if you are shipping in drives.
Obtaining Unique URL and Key
Selecting the graphic of the key will provide you with the unique URL and key that will be used during the upload.
Uploading the PST Data
Now we can upload the data using the previously downloaded “Microsoft Azure Storage Tools”. The syntax used will be similar to the example below:
AzCopy.exe /source:\\SERVER\SHARE /dest:{your unique URL}/SERVER/SHARE /destkey:{your unique key} /S /V:C:\Logs\upload.log
Note that the “AzCopy.exe” utility is located in “C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy” or you can find it from the start menu under the name “Microsoft Azure Storage Command Line”.
Based on my test, it appears that there is some multi-threading occurring in the upload as my three test files all completed their upload around the same time. The upload rate maxed out my available 10 Mbps upload bandwidth so it took a little over 1.5 hours to transfer 6 GB of data; I would assume if you have more capacity, you would see better throughput. It’s not clear if there is any throttling or cap on the Microsoft side. If bandwidth is a concern, this is where the drive shipping option is attractive.
Matching File Creation
Once the files are uploaded, you will need to create a “matching file”. Microsoft supplies a sample template but essentially it’s a CSV that identifies the source PST file name, target mailbox and the folder where you want the data placed. It looks similar to below.
Importing the Data into Mailboxes
Once the files have been uploaded, we can now begin the import process. You will need to have the Exchange “Mailbox Import Export” role just as you would when doing similar operations on-premises.
Selecting “New Job” and then “Upload Data Over The Network” will take you into the import wizard. After selecting that you’ve already uploaded the data and created your matching file, you will be prompted for a job name. Interestingly, the job name can only be in lowercase.
Once you’ve entered the job name, you’ll be prompted for the location of the matching file (local to your machine) and the import process begins. There is a link to a status page during the import job and a summary page provided once the import job completes.
At this point you just let it run. After an hour I had imported about 3 GB of data across three mailboxes but it’s hard to say what the expected rate is. The process also appears to be multi-threaded where it’s importing multiple PSTs concurrently.
Final Thoughts
For a first release of the feature, it does a good job. I would expect additional functionality to be added in the future in regards to status and notifications. Overall I think this service will be a great offering for organizations that have PSTs sitting around post-migration. In particular, the drive shipping option will be helpful for organizations that have terabytes of PSTs. With 50 GB mailboxes and unlimited archives in Office 365, there’s no reason to allow users to keep this data sitting in files on your servers.
One consideration with importing large quantities of data is that it could result in large amounts of caching by your end users; leveraging the online archive (which is not cached) may make sense. Also, once PSTs have been expunged from your environment, it would be recommended to block the creation of new PSTs via Group Policy to stop users from creating new ones. You can find additional tips in the FAQ for this feature.
Update: 09/01/2015
This feature was in “Public Preview” through August 2015 and it looks like it will become a paid feature in General Release based on the amount of data (/GB) being imported into the service.
Update: 04/01/2016
The syntax of this process has changed slightly. I would suggest checking out the latest documentation at: Import PST files or SharePoint data to Office 365
Did you find this article helpful?
Leave a comment below or follow me on Twitter (@JoePalarchio) for additional posts and information on Office 365.
Looking to do some more reading on Office 365?
Catch up on my past articles here: Joe Palarchio.
PST Capture was awful. @Schnoll said they bought it but didn’t do anything to improve it.
The MessageOps “Office 365 Exchange Migration Tool” is great for the price.
I look forward to trying this new feature out. Hopefully it won’t be subjected to throttling by MS like importing via Outlook is (0.5 GB/hr).
I think this option will quickly become the tool of choice unless you need something to scour the network for PSTs and identify them in which case some of the third-party options will still have a place.
The option of shipping 4 TB of PSTs at a time will be ideal for companies with large quantities.
I’ve been playing with this for a few hours. Upload times are great for me at 9GB in 3/4 minutes. Any idea how to specify the data if uploaded to the same location in the mailbox as the PST?
Setting “IsArchive” to false will put the data in the actual mailbox however keep in mind the implications to the OST when doing this. Depending on the date of the data and version of Outlook, it’s possible that Outlook will then try to cache all that data.
Do you need to have an Azure subscription to use this feature?
No, I am not aware of any requirement for an Azure subscription.
Are there any powershell Cmdlets to modify/control import jobs, have been trying out process but fails when it hits a bad item limit of 2. Cannot find any technical documentation anywhere and O365 support have just referred me back to the PST Capture tool
Darren-
I believe that behind the scenes, the tool is using the “New-MailboxImportRequest” cmdlet but the “BadItemLimit” does not seem to be exposed. Like many Office 365 features, I would look for this to improve in the future but there does not appear to be an option today.
Thanks
Joe
What a great method for importing PSTs. Soooooo much better than PST Capture. Also so much faster, uploads 8GB in 1min 53sec.
8GB in 1min 53sec? How’s 2025 treating you?
Some points:
-I hope this stays free beyond August, when the preview is scheduled to end, but I have my doubts.
-Compression would be a natural for this, so why it doesn’t require that you zip the files first is beyond me, since it would save time (for those not in 2025) and ensure the integrity of the upload.
-Azcopy gives you no clue whatsoever that anything is happening until after it’s happened. It just sits on a message about the transfer speed being 0.
-Some control over timeouts would be nice, as I get a fatal error “The client could not finish the operation within specified timeout” for anything beyond small PSTs. There IS no specified timeout, and running the command again, which retries, ends the same way. My UL bandwidth isn’t so bad that it should cause this.
Rick-
My transfer rate was 6 GB in about 1 hour 43 min which was about the maximum transfer rate I could get out of on my 10 Mbps home cable modem.
The import program is currently in “preview” so I’m sure it will only get better and hopefully address some of your items. We’ve already seen it recently expanded to include SharePoint Online data.
To follow up, I borrowed someone’s 5Mb/s connection, and the upload went fine. Conclusion: if you have a slow connection that’s otherwise stable and predictable for uploading, forget about it with Azcopy, which for whatever reason is more demanding.
Worth mentioning also is the time that it takes to process the PST once uploaded. You might think that since it’s in Azure that it would be relatively quick, but a multi-GB PST will take hours, possibly longer than the UL.
Yes, the SP migration looks interesting, though just a tad involved. Also, even in preview it requires a an Azure subscription and Azure Storage account, which I’m assuming are not of the variety that come included with O365 but an actual paid subscription on top of it. I haven’t seen any articles yet putting this into practice, so I’m not sure.
https://technet.microsoft.com/en-us/library/mt203955.aspx
Hello all,
I’ve tested several times today, just a 40Mb pst file and right after 15 min I’m always getting the same timeout error.
The following lines are the messages I’m getting in the log:
[2015/07/14 12:26:16.820-03:00][VERBOSE] Transfer FAILED: Outlook.test.pst.
[2015/07/14 12:26:16.823-03:00][ERROR] Outlook.test.pst: The client could not finish the operation within specified timeout.
The client could not finish the operation within specified timeout.
Could anyone please give me a hand with this?
There is not too much information about this, Could it be that internet upload speed is not enough? I mean, I’m just trying to upload a 40mb file and my internet connection is not that bad.
Thanks,
Fabricio.
Fabricio-
I would try using verbose mode (/V) and then looking to see if there is any additional info in “AzCopyVerbose.log” located in the “%LocalAppData%\Microsoft\Azure\AzCopy” directory.
I’m gonna try using a faster connection.
But I suppossed that 1Mb/s would’ve been enough for a 40mb file.
@joe I was already running in verbose mode. There are not too useful info there.
[2015/07/14 12:11:12.606-03:00][VERBOSE] Start transfer: Outlook.test.pst
[2015/07/14 12:26:16.820-03:00][VERBOSE] Transfer FAILED: Outlook.test.pst.
[2015/07/14 12:26:16.823-03:00][ERROR] Outlook.test.pst: The client could not finish the operation within specified timeout.
The client could not finish the operation within specified timeout.
[2015/07/14 12:26:16.831-03:00] Transfer summary:
—————–
Total files transferred: 1
Transfer successfully: 0
Transfer skipped: 0
Transfer failed: 1
Elapsed time: 00.00:15:10
I’d guess, as Rick said, it’s something regarding my internet connection speed. I’ll try in a couple of hours and would let you know if that was the problem.
Thanks.
Fabricio.
Hello,
Thanks for the post, it helped a lot; however, I do have a question, I set the is archive option to false but still didn’t import the data to the inbox or kept the old folder structure, instead everything was created in a new folder called Imported.
Is there a way no to create the Imported folder and instead import the mails into the Inbox and create the original folder structure?
Didier-
What did you have “TargetRootFolder” set to in your CSV file? It should be set to “/” if you want to go into the root of the mailbox and keep the folder structure.
Thanks!
Joe
I heard that microsoft office 365 upload speed was 1mb/h.
If is true, my work wil not finish because I have 50 users and each pst files capacity is change to 20 gb from 5 gb.
What can I do?
Serkan-
I transferred 6 GB in about 1.75 hours which was about the max my Internet connection could do.
You should get much better throughput than 1 MB/hr.
Hello,
Is it possible to have the pst file on local machine (not on a shared drive)?
Thanks,
AbdelRahman
AbdelRahman-
I have not tried but it’s possible you could reference the file via the local administrative share (\\localhost\c$).
after transferring the data to azure storage, we then had 1 of the 2 transfer’s fail.
Fatal error MailboxReplicationPermanentException has occurred.
The other archive was successfully imported. I can’t see any option to retry the transfer, and it would appear to try again I have to repeat the whole process of uploading the pst given a new key will be generated. Is there a way to just do the import data into mailbox step?
Steve-
You can likely use “Resume-MailboxImportRequest” along with “Set-MailboxImportRequest” to modify some of the import options such as the BadItemLimit. It’s possible that if the PST is corrupted, you’ll need to try and repair it offline with something like “scanpst.exe” and then upload and import again.
Thanks
Joe
Hi, thank you for the article. Can we import PST to Shared Mailbox using this method?
Rodney-
I haven’t tested it yet but I would expect shared mailboxes to work fine.
Is there a way to delete jobs from the Import panel after they have completed in O365? I am also looking for a way to manage the jobs in an organized fashion (alphabetically or in order started) preferably by date.
I try this, but all the test i do a same error, they cannot find pst file,
the pst file is in my pc, like d:\mypst.pst
in azure tool, what I have to insert in /source:\\SERVER\SHARE ?
I try create mapping h:\\myhosts\d\ and in azure tool /source:\\MyHostNAME\d
again…
Could not find source file TEMPT1-WIN7X64/e/My-PST.pst. Please correct the FilePath column in the mapping file and create a new job with the updated mapping file
Full source path
TEMPT1-WIN7X64/e/My-PST.pst
But, the file are uploaded, log file:
[2016/02/01 22:06:42.751-02:00] >>>>>>>>>>>>>>>>
[2016/02/01 22:06:42.761-02:00][VERBOSE] 5.0.0 : AzCopy /Source:\\TEMPT1-WIN7X64\e /Dest:https://*********************************.blob.core.windows.net/ingestiondata/ /Destkey:****** /S /V:e:\Uploadlog.log
[2016/02/01 22:06:45.621-02:00][VERBOSE] Start transfer: My-PST.pst => My-PST.pst
[2016/02/01 22:06:45.981-02:00][VERBOSE] Start transfer: Uploadlog.log => Uploadlog.log
[2016/02/01 22:06:46.151-02:00][VERBOSE] Transfer FAILED: Uploadlog.log => Uploadlog.log.
[2016/02/01 22:06:46.161-02:00][ERROR] Uploadlog.log: Failed to open file \\TEMPT1-WIN7X64\e\Uploadlog.log: The process cannot access the file ‘\\TEMPT1-WIN7X64\e\Uploadlog.log’ because it is being used by another process..
[2016/02/01 22:26:32.147-02:00][VERBOSE] Finished transfer: My-PST.pst => My-PST.pst
[2016/02/01 22:26:32.197-02:00] Transfer summary:
—————–
Total files transferred: 2
Transfer successfully: 1
Transfer skipped: 0
Transfer failed: 1
Elapsed time: 00.00:19:49
WHY THE PST ARE NOT FOUND?
in csv. file
Exchange,TEMPT1-WIN7X64/e,My-PST.pst,my.email@mydomain.onmicrosoft.com,FALSE,,,,
please, help…
Yuri-
It looks like your file is in use. Make sure it’s not connected in Outlook or in use by some other hung process.
Tiffany-
I haven’t tested but I suspect Remove-MailboxImportRequest should work via PowerShell.
To find my SOURCE file, I had to leave the FilePath blank in my CSV file. Once I did that, it worked fine. Now I just have to figure out how to remove all the test imports I did.
Thanks for the article…..i have Pst’s directly on my PC i am trying to upload….so i have my link and then \\pcname\c$\PST however i am getting:
the filename,directory name or volume label syntax is inccorect.
Any thoughts?
Joe-
I believe I may have had problems using the administrative share. Try and setup a named share for C:\PST and it should work.
Thanks!
Joe
Hi Joe,
Microsoft recently updated the command to use /DestSAS instead of /DestKey.
https://technet.microsoft.com/en-us/library/mt644809.aspx
10GB limit on PST files still exist, when will MSFT realize average PST is 30-30-40 GB !
I have uploaded a 30GB to Azure using the process have not tried mapping it in yet, but they have advised it will not import, anyone done more than a 10GB PST using this process
Scott-
I know we’ve imported some PSTs close to 10 GB but I don’t know that I’ve done any larger than that.
What was your experience? Did it work?
Hi,
Thanks for the informative blog.
Another easy, reliable & efficient method to import PST to Office 365 is using SysTool Office 365 Restore.
This tool helps to import the PST file to Office 365. The best part of this tool is that it has NO SIZE LIMITATION; a user can import any sized PST file with ease. It has many other amazing features too.
To know more about the tool, visit the link below:
Efficiently Migrate PST to Office 365 in Few Seconds
Hi,
I’ve been trying to upload a pst to a shared mailbox but i can’t seem to make it work. The job stays on 0% and eventually fails. When i change the destination in the PstImportMappingFile to a user folder it succeeds.
Have you succeeded in uploading to a shared mailbox?
I have been battling with this for days, tried it over 20 times with different permutations of the “TargetRootFolder” to be \ or / and created a new folder called “Import” in the users’ mailboxes and tried specifying the TargetRootFolder as /Import. I also checked that the user I am logged in with has the Mailbox Import Export permission (go to Admin Roles in the Exchange Admin Center, create a new role). I also checked that everything was in the correct case, but that made no difference.
The import will sit on “In Progress” with 0 items imported, and then after an hour, report “Completed with Errors” and do nothing. Even weirder is that it worked for two mailboxes.
I have found the solution – which may be related to me deleting and recreating the users;
In the mapping CSV, for the Mailbox/Account/Username parameter, I put in the GUID for the user from AzureAD.
Go to Manage.WindowsAzure.com, then go down to Active Directory, select your domain, select a user you are trying to import the mail for, and look for their “Object ID” – it looks like 24ca673d-6e83-4c1e-4ab0-befad77b6262
Specify their Object ID instead of their mailbox or username.
It worked for me!
Workload,FilePath,Name,Mailbox,IsArchive,TargetRootFolder,SPFileContainer,SPManifestContainer,SPSiteUrl
Exchange,,PSTNAME.pst,ObjectID,FALSE,/,,,
YES!!!!!!!!!!! Thank you!!! That was it. I’m finally able to import the last pst’s that kept failing. That was the final peace of the puzzle, i’m finished with my migration.
Respect!
Christian you deserve a platinum medal for that post. Wasted 3 hours of my life trying to fix imports while the issue was in old accounts still hanging around in the recycle bin. GUIDs solved everything.
Everyone, this feature is horrible and a waste of time.
Buy a messageops license and easily migrate all your data for 99 dollars.
Or better yet, use Migration Wiz.
I can’t believe there still isn’t a simple upload pst option, where you can just browse to an PST, select a target, and click upload. Seriously how does this still not exist.
How is it useless? For most SMBs it’s an easy self-service method to migrate. I took 10 employee mailboxes totalling over 40GB to Office365 in about 8~10 hours total, with not a single service interruption . No idea how I would’ve done it otherwise. HTTP uploads aren’t realistic for more than a few 100 megabytes, that would cover about 2% of the migration scenarios while the Azure upload is just a few lines of copy/paste.
Heck, I would’ve been done in 6 or 7 hours, of which 95% watching movies, if I hadn’t run into the GUID curiosity.
The performance of the Exchange Online PST import is abysmal. It took me only 2 hours to upload a 37GB PST file to Microsoft Azure (on a 100/40 DSL line). But the import has been running for 7 1/4 hours and has only imported 6GB of the PST!
Even worse, the import does not prioritize important data. If they had coded it properly it would at least import contacts and calendar first, then Inbox and then the rest of the folder structure. But instead its wasting hours importing irrelevant archival data while the user has an empty contact folder, no calendar and no inbox.
I really can not recommend using this feature. Importing on the client PC using Outlook is slow too, but it is not slower than the PST import and at least you have the data fully available on the one client you are importing from! And you can even manually prioritize by importing contacts and calendar first.
Can we import the large items (more than 150MB) ?
PST Capture tool was a waste of time. Even now there are user facing issues
The logging of skipped items is non exsistent. Is there a way to identify the exact items that were skipped
I had tested the above scenario but after importing PST files I didn’t see any mails in target mailbox folder but when I search with any name it’s showing all mails with that name. Please help how I can see in all mails in target mailbox folder. Thanks in advance.