forcedotcom / data-migration-tool Goto Github PK

License: BSD 3-Clause "New" or "Revised" License

Java 100.00%

data-migration-tool's Introduction

Data-Migration-Tool (Datafall)

Data-Migration-Tool (Datafall) is a tool that migrates the data from one Salesforce Org to another Salesforce Org.

Migrating data between orgs can be a daunting ask, this is specially true if you have a complex data model in the form of parent-child, hierarchical, lookups, record types etc relationships.

Doing so manually is error prone, time consuming, frustrating and not an ideal approach. We wrote a Java based tool using Salesforce SOAP to migrate the data from one Salesforce Org to another Salesforce Org. Data-Migration-Tool tool lets you migrate the data from one Salesforce Org to another Salesforce Org. This is a standalone java tool that one can run from a local machine.

Build Data Migrator

Tool has two modules.

datamigrator-core: This module has tools core classes
datamigrator-module: This module has sample examples on how the tool can be used to migrate different kind of related objects from one Salesforce org to another Salesforce org. datamigrator-module contains sample JSON object relationship mappings.

Build

git clone [email protected]:forcedotcom/Data-Migration-Tool.git
cd Data-Migration-Tool
mvn clean install

Usage: Migrate Products, Pricebooks and PricebookEntries Sample

# Enter Source and Target org credentials in `/datamigrator-module/src/main/resources/build.properties`
mvn clean install
mvn exec:java -Dexec.mainClass="migrator.module.client.MigrateProducts" -pl datamigrator-module

Running in your IDE

Overview

You can load the project into your IDE and run the main class to kick-off the migration. We recommend using Eclipse. Because this tool is a standard Maven project, you can import it into your IDE using the root pom.xml file. In Eclipse, choose Open from the File menu -> Import -> Maven -> and select Existing Maven Projects -> select the the root folder.

After opening the project in Eclipse, you will see below two modules:

datamigrator-core: This module has tools core classes
datamigrator-module: Contains the mapping and main java classes to run the migration

Tool comes with many main classes as a way to show different kind of mappings and migrations:

Main Class: migrator.module.client.MigrateProducts
Main Class: migrator.module.client.MigrateAll
Main Class: migrator.module.client.MigrateAccounts
Main Class: migrator.module.client.MigrateAccountsHierarchical

and many more...

Running using command line (More Sample Migrations...)

To avoid creating duplicates in the target org, use external ids. This section has some examples using external ids.

Migrate a simple object's records (no relationships)

# Edit the object-mappings/Products.json with your object API name, save

mvn exec:java -Dexec.mainClass="migrator.module.client.MigrateOnlyProducts" -pl datamigrator-module

Migrate an object with lookups (Pricebook2 with lookup to Product) relationship

# Edit the object-mappings/Products.json, PricebookEntry_nonstandard.json, PricebookEntry_standard.json with your object API name, save.

mvn exec:java -Dexec.mainClass="migrator.module.client.MigrateProducts" -pl datamigrator-module

Migrate an object with lookups (Pricebook2 with lookup to Product) relationship with External Ids

# Edit the object-mappings/Products_extId.json, PricebookEntry_nonstandard_extId.json, PricebookEntry_standard_extId.json with your object API name, save.

mvn exec:java -Dexec.mainClass="migrator.module.client.MigrateProductsWithExternalId" -pl datamigrator-module

Migrate object's with masterdetail (Accounts/Assets/Oppties) relationships

# Edit the object-mappings/AccountwithAssetsAndOppties.json with your object API names, save

mvn exec:java -Dexec.mainClass="migrator.module.client.MigrateAccounts" -pl datamigrator-module

Migrate object's with hierarchical relationships

# Edit the object-mappings/AccountwithAssetsAndOppties_hierarchical.json with your object API names, save

mvn exec:java -Dexec.mainClass="migrator.module.client.MigrateAccountsHierarchical" -pl datamigrator-module

Migrate multiple object's in a single run

# Edit the object-mappings as referenced in MigrateAll.java with your object API names, save

mvn exec:java -Dexec.mainClass="migrator.module.client.MigrateAll" -pl
datamigrator-module

Migrate Accounts with Opportunities and Assets. Make sure you edit the JSON with your external ids from Accounts with Opportunities and Assets objects.

# Edit the object-mappings/AccountwithAssetsAndOppties_hierarchical.json with your object's external Id API name, save

mvn exec:java -Dexec.mainClass="migrator.module.client.MigrateAccountsHierarchical" -pl datamigrator-module

Usage (Sample Deletions from Target Org)

Delete a simple object's records

# Edit the object-mappings/Products.json with your object API name, save

mvn exec:java -Dexec.mainClass="migrator.module.client.DeleteProducts" -pl datamigrator-module

Delete multiple object's records in a single run

# Edit the object-mappings as referenced in DeleteAll.java with your object API names, save

mvn exec:java -Dexec.mainClass="migrator.module.client.DeleteAll" -pl datamigrator-module

Usage (Compare objects metadata in source and target org)

Compare objects Metadata in source and target org

mvn exec:java -Dexec.mainClass="migrator.module.client.MetadataCompareExample" -Dexec.args="object1,object2" -pl datamigrator-module

Usage (Given a set of objects, create a JSON mapping structure)

Given a set of objects, create a JSON mapping structure

mvn exec:java -Dexec.mainClass="migrator.module.client.GenerateMappingExample" -Dexec.args="object1,object2" -pl datamigrator-module

Usage (Counts the number of records in source and target org)

Once migration has completed, you may want to compare the number of records in source and target org

mvn exec:java -Dexec.mainClass="migrator.module.client.ValidateExample" -Dexec.args="object1,object2" -pl datamigrator-module

More Advanced Object Relationship JSON Mappings, Resources

For more information, see the wiki

Authors

Anoop Singh by [Anoop Singh](mailto:[email protected], [email protected])

data-migration-tool's People

Contributors

Stargazers

Watchers

Forkers

yangzhang00 alankipalle rahman123 jorge0136 abenassi87 arundecent applewave growthheroes rpayne0629 blackhat-87 karthikkornalies mcicheick jantaks dcyaste andrew2000 pradeepthoom developwitheddy bellanc samudralap yerrak asif-github yada91 nileshndethe saicharanreddyk topchoice mukundkoiraladh devarshi87 saichandernagamalla darthmanwe harshad7 pushkar-abhijeet yaty-dev servicemax-engineering omerfarukozdemir isabella232 dcheng666666 mikefigari

data-migration-tool's Issues

Login Failed

Hi there,

I'm attempting to run the sample 'Migrate Products, Pricebooks and PricebookEntries' - on the mvn exec:java -Dexec.mainClass="migrator.module.client.MigrateProducts" -pl datamigrator-module command i'm getting Login failed! error.

Any advice would be highly appreciated. I'm keen to be able to utilize this tool.

my build.properties file looks like this:

source.sfdc.login=<SOURCE_SF_USERNAME>
source.sfdc.password=<SOURCE_SF_PASSWORD>
source.env.endpoint="https://salesforce.com/services/Soap/u/44.0/"
# TARGET
target.sfdc.login=<TARGET_SF_USERNAME>
target.sfdc.password=<TARGET_SF_PASSWORD>
target.env.endpoint="https://test.salesforce.com/services/Soap/u/44.0/"

# # Optional proxy
# env.proxy.host=
# env.proxy.port=

target.threads.count=1

source.type=org

object.mapping.dir=/object-mappings/
data.mapping.dir=/data-mappings/ ```

Not able to upsert record from one org to another org Using externalId

Hi ,

I tried to upsert product record from my one dev org to another dev org using an external ID,
but it is throwing error.

2019-04-10 21:25:26 ERROR SfdcApiServiceImpl:1169 - ObjectName: Product2 ,originalIndex: 0 ,sourceId: 01t0o000007Ei9KAAS ,Error Message: External_ID__c not specified

I tried to access External Id I am able to get the name in class.

Can you please help me , I am stuck here.

Thanks
Saket gupta

Event and Task did not get WhatId and WhoId migrated

Hi,

I am trying to migrate Events and Tasks using the modified AccountwithAssetsAndOppties_hierarchical.json and following is my snippet. It did not migrate the Contact (WhoId) and Account (WhatId) in this case. Any idea why? In our org WhatId is always an Account and WhoId is always a Contact.

.... more objects before { "parentMappedField": "WhatId", "childObject": "Event", "ExternalIdField": "Account.External_Id__c", "where": " Account.External_Id__c = 'MYEXTID-JIBBER-JABBER-SELF' AND IsChild = FALSE", "unmappedFields": [ "AccountId" ], "sequence": 2 }, { "parentMappedField": "WhatId", "childObject": "Task", "ExternalIdField": "Account.External_Id__c", "where": " Account.External_Id__c = 'MYEXTID-JIBBER-JABBER-SELF'", "unmappedFields": [ "AccountId" ], "sequence": 3 }, .... more objects after

Record Types

Hi,

I am facing issues trying to migrate properly the recordtypes assigned to source org to a targert org.

I tried to add as lookups the record type, but still it jsut assign me a default record type instead of doing the upsert with the proper record type.

How I can approach this in roder to align properly record type for my NE__Matrix_Parameter_Row__c object ?

`[
{
"parent": "NE__Matrix_Parameter__c",
"externalIdField": "Source_External_Id__c",
"lookups": [
{
"lookupMappedField": "NE__Catalog_Header__c",
"lookupObject": "NE__Catalog_Header__c",
"keys": ["NE__Source_Catalog_Header_Id__c"]
}

],
"children": [
  {
    "parentMappedField": "NE__Matrix_Parameter__c",
    "childObject": "NE__Matrix_Parameter_Row__c",
    "externalIdField": "NE__Source_External_Id__c",
    "sequence": 1,
    "lookups": [
	        	{
	        		"lookupMappedField": "Commercial_Product__c",
	                "lookupObject": "NE__Product__C",
	                "keys": ["Name","Code__c"]
	        	},
	        	{
	        		"lookupMappedField": "FI_MPR_LKP_Discount_Product__c",
	                "lookupObject": "NE__Product__C",
	                "keys": ["Name","Code__c"]
	        	},
	        	{
	        		"lookupMappedField": "RecordTypeId",
	                "lookupObject": "RecordType",
	                "keys": ["Id","Name","DeveloperName"]
	        	
	        	
	        	}
	        ]
  }
]

}
]`

thanks in advance

Use with VSCODE

Can i use this with VSCODE IDE? If yes what are the step to install and use it with this IDE??

Docker Image File

Hi @anoop-76,

I was wondering if you use a docker image to run the Data Migration Tool. If so, can you make it public? This may be an easy way to mitigate the maven errors i'm receiving.

Best,

Brianna

Migrate users

Hi,
I'm wondering if there is any way to migrate users?

Best,
Przemek

Tool does not work when version higher than v46

It looks like something about the describe sobject response is breaking the tool when version > 46. The response is in the logs and it looks valid but after that response a java error occurs.

Here's the stack trace for v47. I used the delete account sample which works for v46.

com.sforce.ws.ConnectionException: unable to find end tag at: START_TAG seen ...... @1:64986
at com.sforce.ws.bind.TypeMapper.consumeEndTag(TypeMapper.java:437)
at com.sforce.soap.partner.DescribeSObjectResult.load(DescribeSObjectResult.java:1345)
at com.sforce.ws.bind.TypeMapper.readSingle(TypeMapper.java:674)
at com.sforce.ws.bind.TypeMapper.readObject(TypeMapper.java:556)
at com.sforce.soap.partner.DescribeSObjectResponse_element.setResult(DescribeSObjectResponse_element.java:48)
at com.sforce.soap.partner.DescribeSObjectResponse_element.loadFields1(DescribeSObjectResponse_element.java:107)
at com.sforce.soap.partner.DescribeSObjectResponse_element.loadFields(DescribeSObjectResponse_element.java:83)
at com.sforce.soap.partner.DescribeSObjectResponse_element.load(DescribeSObjectResponse_element.java:77)
at com.sforce.ws.bind.TypeMapper.readSingle(TypeMapper.java:674)
at com.sforce.ws.bind.TypeMapper.readObject(TypeMapper.java:556)
at com.sforce.ws.transport.SoapConnection.bind(SoapConnection.java:180)
at com.sforce.ws.transport.SoapConnection.receive(SoapConnection.java:154)
at com.sforce.ws.transport.SoapConnection.send(SoapConnection.java:99)
at com.sforce.soap.partner.PartnerConnection.describeSObject(PartnerConnection.java:1492)
at migrator.core.service.impl.MetadataServiceImpl.process(MetadataServiceImpl.java:95)
at migrator.core.service.impl.MetadataServiceImpl.process(MetadataServiceImpl.java:80)
at migrator.core.service.MetadataObjectHolder.init(MetadataObjectHolder.java:77)
at migrator.core.sobject.MigrableLookupObject.setup(MigrableLookupObject.java:111)
at migrator.core.sobject.MigrableMasterDetailObject.setup(MigrableMasterDetailObject.java:90)
at migrator.core.sobject.MigrableMasterDetailObject.process(MigrableMasterDetailObject.java:72)
at migrator.core.sobject.MigrableObjectFactory.migrate(MigrableObjectFactory.java:68)
at migrator.module.client.DeleteAccounts.process(DeleteAccounts.java:52)
at migrator.module.client.DeleteAccounts.main(DeleteAccounts.java:57)
com.sforce.ws.ConnectionException: unable to find end tag at: START_TAG seen ...... @1:70305
at com.sforce.ws.bind.TypeMapper.consumeEndTag(TypeMapper.java:437)
at com.sforce.soap.partner.DescribeSObjectResult.load(DescribeSObjectResult.java:1345)
at com.sforce.ws.bind.TypeMapper.readSingle(TypeMapper.java:674)
at com.sforce.ws.bind.TypeMapper.readObject(TypeMapper.java:556)
at com.sforce.soap.partner.DescribeSObjectResponse_element.setResult(DescribeSObjectResponse_element.java:48)
at com.sforce.soap.partner.DescribeSObjectResponse_element.loadFields1(DescribeSObjectResponse_element.java:107)
at com.sforce.soap.partner.DescribeSObjectResponse_element.loadFields(DescribeSObjectResponse_element.java:83)
at com.sforce.soap.partner.DescribeSObjectResponse_element.load(DescribeSObjectResponse_element.java:77)
at com.sforce.ws.bind.TypeMapper.readSingle(TypeMapper.java:674)
at com.sforce.ws.bind.TypeMapper.readObject(TypeMapper.java:556)
at com.sforce.ws.transport.SoapConnection.bind(SoapConnection.java:180)
at com.sforce.ws.transport.SoapConnection.receive(SoapConnection.java:154)
at com.sforce.ws.transport.SoapConnection.send(SoapConnection.java:99)
at com.sforce.soap.partner.PartnerConnection.describeSObject(PartnerConnection.java:1492)
at migrator.core.service.impl.MetadataServiceImpl.process(MetadataServiceImpl.java:93)
at migrator.core.service.MetadataObjectHolder.init(MetadataObjectHolder.java:87)
at migrator.core.service.impl.MetadataCompareService.findCommonFields(MetadataCompareService.java:107)
at migrator.core.service.impl.MetadataCompareService.findCommonFields(MetadataCompareService.java:95)
at migrator.core.sobject.MigrableLookupObject.setup(MigrableLookupObject.java:113)
at migrator.core.sobject.MigrableMasterDetailObject.setup(MigrableMasterDetailObject.java:90)
at migrator.core.sobject.MigrableMasterDetailObject.process(MigrableMasterDetailObject.java:72)
at migrator.core.sobject.MigrableObjectFactory.migrate(MigrableObjectFactory.java:68)
at migrator.module.client.DeleteAccounts.process(DeleteAccounts.java:52)
at migrator.module.client.DeleteAccounts.main(DeleteAccounts.java:57)
Exception in thread "main" java.lang.NullPointerException
at migrator.core.service.impl.MetadataCompareService.findCommonFields(MetadataCompareService.java:112)
at migrator.core.service.impl.MetadataCompareService.findCommonFields(MetadataCompareService.java:95)
at migrator.core.sobject.MigrableLookupObject.setup(MigrableLookupObject.java:113)
at migrator.core.sobject.MigrableMasterDetailObject.setup(MigrableMasterDetailObject.java:90)
at migrator.core.sobject.MigrableMasterDetailObject.process(MigrableMasterDetailObject.java:72)
at migrator.core.sobject.MigrableObjectFactory.migrate(MigrableObjectFactory.java:68)
at migrator.module.client.DeleteAccounts.process(DeleteAccounts.java:52)
at migrator.module.client.DeleteAccounts.main(DeleteAccounts.java:57)

Error when importing project into Eclipse

I am getting following error when importing project into Eclipse:

Please suggest how to resolve this.

BULK API Option

Do we have an Option to migrate data with Bulk API Mode.

memory usage improvements?

I was running into troubles with memory when attempting a migration very similar to the pricebookentry migration in the samples. To make sure it wasn't my error, I tried running MigrateProducts.java from the samples and got the same result

Below is the tail of what was in the console (for MigrateProducts) when it stopped running. I let it run for 3 hours or so. I also had turned on the traceLog file which had grown to about 2GB at that point. It looks like it queries all of my products (about 10k of them). And then tries again. I assume one query is for source and the other is for target though the logging isn't very clear about that. When I looked at the trace log the last entry is a retrieve more so it looks like at some point the amount of data retrieved is so much that it really crawls.

I'd guess the only solutions would be either some kind of database or a poor man's database using flat files and lots of streams.

BTW, for my own effort (before I switched to the sample) I was doing something very similar. I was migrating pricing data but with custom tables from a managed package. I only wanted to move pricing for a few products. But since the table equivalent to pricebookitem has a lookup to product2, the tool still wanted to download the whole product table (even though I really only needed 3 rows.

I was successful migrating a few hundred rows from a smaller table.

2020-12-02 18:55:16 DEBUG SfdcApiServiceImpl:214 - Lookup: Running SF query for [Product2]
2020-12-02 18:55:16 DEBUG SfdcApiServiceImpl:950 - sforceObject.query=SELECT [...] FROM Product2
2020-12-02 18:55:16 DEBUG SfdcApiServiceImpl:881 - query:SELECT [...] FROM Product2
2020-12-02 18:55:42 DEBUG SfdcApiServiceImpl:899 -
Logged-in user can see 10664 Product2 records.
2020-12-02 18:55:42 DEBUG SfdcApiServiceImpl:269 - SF query for type [Product2] took [25903]ms
2020-12-02 18:55:42 DEBUG SfdcApiServiceImpl:214 - Lookup: Running SF query for [Product2]
2020-12-02 18:55:42 DEBUG SfdcApiServiceImpl:950 - sforceObject.query=SELECT T[...] FROM Product2
2020-12-02 18:55:42 DEBUG SfdcApiServiceImpl:881 - query:SELECT [...] FROM Product2

Unknown lifecycle phase "pl"

I downloaded the tool, followed installation instructions, and finally tried to run one of the sample commands. I tried to run the Migrate Accounts with Opportunities and Assets with the command
mvn exec:java -Dexec.mainClass="migrator.module.client.MigrateAccountsHierarchical.java" pl datamigrator-module
but in return I got the following error:

[ERROR] Unknown lifecycle phase "pl". You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/LifecyclePhaseNotFoundException

Is there something I'm missing from the setup? I haven't changed anything on any file.
I tried to run
mvn clean install before the actual command and that didn't help either.

Any help is appreciated.
Thanks!

Dump data to CSV / Load data from CSV

It looks like the project only supports on-the-fly copy of data from one org to the next. Would be good to be able to load data from a set of CSV files, and to export data to CSV files for posterior use (e.g. backup data in a sandbox before refreshing).