apigeecs / apigee-migrate-tool Goto Github PK

Hi! Thanks for maintaining this tool!

I'm currently trying to use this tool to migrate data between some corporate Apigee installations, I've ran into an issue I've mostly been able to address which is that most of the tasks here will fire off asynchronous javascript requests as fast as possible without waiting for a callback. This was a problem for me because I didn't want to install grunt and NPM on my production servers, and was forced to use this tool over an SSH tunnel; which then gets overwhelmed with requests and dies whenever I try to migrate ~6000 developers or ~7000 something keys.

Now for the most part (and having no javascript experience), I was able to substitute out the request library in each of the tasks for the request-promise library which returns a promise for the callback of the async function, and then use the recent javascript 'async/await' functionality to make those tasks execute synchronously. (Awaiting on any request to callback before proceeding with the loop) However I've found the key.js task quite tricky to get to execute synchronously and is what I'm stuck on now.

Anyway, what I'm getting at is a rate limiting feature would be nice!

Hi,

When I am running exportAll, it is not downloading all KVMs. In my enironment, I have 19 but it's downloading only 16. What could be the reason?

Then I thought of running exportEnvKVM task individually, but it is just creating env folder under kvm folder of data, but there is nothing in the folder.

Not sure if I am missing something or there is really any issue.

Could someone please help?

Thanks in advance!!
Vivek Chikane

Hi,
I have used "grunt exportAll -v" all data is not getting imported to target environment. So is there any singe command to import all the exported data to local drive.?

Can all the data be imported using single command.?

Or
we need to run all the below commands manually one by one:
grunt importTargetServers
grunt importProxies
grunt importSharedFlows
grunt importDevs
grunt importProducts
grunt importApps
grunt importKeys
grunt importProxyKVM
grunt importEnvKVM
grunt importOrgKVM

Please provide unit tests, e.g. mocha, so that further commits can be vetted to ensure the continual success of the software.

Using this tool behind a corporate web-proxy requires uncovering undocumented configs. Needs to be addressed in docs at the least.

If the value of the target env differs from the value of source env the import of the developer app fails.

Workaround: If you set the target env manually in each DevApp export file

"environments" : ["A"],

the import works.

exportDevs is not exporting all data !

So that apigee-migrate-tool can relax strict requirement for ES modules, and thereby work with a wider range of Node.js environments.

delete key function is not getting executed;

https://github.com/apigeecs/apigee-migrate-tool/blob/master/tasks/app.js#L220

There are many areas where request fail for unknown reasons and log messages do not provide an indication as to what failed and why. Often simply logging null to console and not indicating what resource failed with a helpful message.

If you export the developers from an organization A and try to import them to organization B the target organization is not set during import but the organization of the source.

Workround: The import works if you manually change the value of the property

"organizationName" : "A",

to

"organizationName" : "B",

in every developer export file.

An org with many (> 10,000) developers fails to export all the developers and generates this error:
Fatal error: error is not defined
exit code $? is 3

The task "deployProxies" fails with these traces :

Running "deployProxies" task
Warning: proxy_url is not defined Use --force to continue.
Aborted due to warnings.

Hey, I'd like to present this to LLBean, do you think it's possible to make this repo public? LLBean understands these tools are all in beta, but they'd rather tweaking it and perhaps send fixes.

WDYT?

Hi,
There is an error in the file "tasks/proxy.js", line 91: the method " grunt.verbose.log(err); " does not exist. This should be replaced with something like " grunt.verbose.writeln(err); "

                   var req = request.post(url+name, function (err, resp, body) {
                      if (err) {
                        grunt.verbose.log(err);
                      } else {

1. The exportProxy feature downloads zero byte proxies. This was noted when we had to rerun the exports. We had to run the export using export all feature.
2. The tool does not clean up the artifacts properly with the delete options. We tried deleting apps and developers using the utility that ended up with orphaned relationships in the cassandra database.
3. The tool did not produce the same results when we cleaned up all the folders and re-ran the exports. This was noted when we were were debugging the zero byte proxy issue.
4. When you run the exportAll feature it downloads the proxies, products and devs only.

This tool doesn't migrate the apimodels resources and subresources. This is not mentionned in the doc.
These resources are required by the devPortals in order to work.

See here a real life use-case when migrating from one instance to another one : https://community.apigee.com/questions/54413/the-model-disappears-when-changing-the-management.html

exportEnvKVM is showing :
Exported 26 kvms
================== export ENV KVM DONE()

while only 15 are exported!

I'm just wondering if there is a possibility of adding the ability to migrate Companies using this tool in the near future? If not that's fine I'll just have to find another way of doing it.

i am trying to export all apps, but get an error. i looked at logs and i think last retrieved developer does have any applications registered..

--
Running "exportApps" task

Retrieved 6 developers
Retrieved 6 developers
Fatal error: Cannot read properties of undefined (reading 'length')
TypeError: Cannot read properties of undefined (reading 'length')
at Request._callback (/Users/santos/src/apigee-migrate-tool/tasks/app.js:55:27)
at Request.self.callback (/Users/santos/src/apigee-migrate-tool/node_modules/request/request.js:185:22)
at Request.emit (node:events:527:28)
at Request. (/Users/santos/src/apigee-migrate-tool/node_modules/request/request.js:1154:10)
at Request.emit (node:events:527:28)
at IncomingMessage. (/Users/santos/src/apigee-migrate-tool/node_modules/request/request.js:1076:12)
at Object.onceWrapper (node:events:641:28)
at IncomingMessage.emit (node:events:539:35)
at endReadableNT (node:internal/streams/readable:1345:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21)

apps are not exported , empty folders created instead

When running this tool I am getting below error.

Running tasks: exportAll

Running "exportAll" task

Running "exportProxies" task
========================= export Proxies ===========================
getting proxies...https://xxxxxxxxxxxxxxxx

Error: write EPROTO C4370000:error:0A000152:SSL routines:final_renegotiate:unsafe legacy renegotiation disabled:c:\ws\deps\openssl\openssl\ssl\statem\extensions.c:922:

Would appreciate help here?

Would you consider adding targets under the environmental configuration as well? You have all of the other state handled within Apigee already. Thanks!

While trying to exportAllSpecs I get an error
“.Cannot read properly ‘length’ of undefined.

Orgs with SAML federation or Orgs with Mandatory 2FA can not use this tool any longer.
Can we support Bearer tokens as an alternative to Basic Auth?

When I run grunt exportAll -v command, it executes only starting 4 tasks from register task. I tried this by changing the sequence.

grunt.registerTask('exportAll', ['exportDevs','exportProducts','exportApps', 'exportEnvKVM', 'exportProxies', 'exportOrgKVM', 'exportProxyKVM']);

Kindly suggest if I am going somewhere wrong.

when running "grunt exportProxies" from my prod environment , but its exporting from latest version the proxy not the proxies deployed to prod environment . My requirement is to export all the prod environment api and import to new org. Please suggest how to make this using this tool.

I am having a issue in the "import app". I have 140 Apps and exported all Apps in the data folder. While running "grunt importApps",

only 70 Apps imported . rest of the Apps not imported .
Also "Credential" section is empty for all import Apps. I want credential section also import.
EX- data/apps/[email protected]/AMEX-DEV-MANAGER-API-700_REGISTERED_NOT_VERIFIED this kind of not import . Any issue here ?
Please help me to fix these TWO issues.

Please indicate which Node.js engine the project requires:

• In package.json
• In .node-version
• In README.md or similar documentation

almost all scripts are parsing json responses incorrectly leading to the full tool to fail.
Nothing gets exported.

I fixed one or 2 scripts manually to confirm it was a parsing issue. appending a child element usually does the trick, but all scripts needs to be reviewed

Also export of revision is badly implemented. Last revision from json response is taken, rather than the revision with highest number.

Export proxy kvm not working not exporting any data

1. We are migrating from on premises version to public cloud. How to provide "to": version information for edge public cloud. This information need to be set into config.js
   https://github.com/apigeecs/apigee-migrate-tool/blob/master/config.js

2. When running this tool I am getting below error. I am guessing it is due to cert issue. We are using our company signed ssl certificate for management server. Please let us know workaround for this.

Running tasks: exportAll

Running "exportAll" task

Running "exportProxies" task
========================= export Proxies ===========================
getting proxies...https://:/

Error: unable to verify the first certificate

When you run the exportAll command it downloads the proxies, products and devs only.
Version from https://github.com/stevetraut/apigee-migrate-tool downloads all entities. But it is not up to date (for example flow hooks support is not added there.)
Please fix this problem in current repository.

1. The apps and dev tasks do not account for more than 1000 entries on retrieval from Apigee.
2. Apps import does not enforce desired status on import.

while tryingto export import sharedflow, i noticed sharedflow export is creating file size of 0 size and hence import gives the following error. Tried extracting the zip file of exported shared flow and that errors out as well.

"code" : "messaging.config.beans.InvalidBundle",
"message" : "Bundle is invalid. Empty bundle",
"contexts" : [ ]

[ sharedflows]# ls -rtl
total 0
-rw-r--r-- 1 root root 0 Jun 17 12:54 LoggingPreProxyFlowHookSharedFlow.zip

[ sharedflows]# unzip LoggingPreProxyFlowHookSharedFlow.zip
Archive: LoggingPreProxyFlowHookSharedFlow.zip
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of LoggingPreProxyFlowHookSharedFlow.zip or
LoggingPreProxyFlowHookSharedFlow.zip.zip, and cannot find LoggingPreProxyFlowHookSharedFlow.zip.ZIP, period.

Target Servers place results in ./data/targetservers regardless of env.
Should create sub-directory for each env (e.g. /data/targetservers/test, /data/targetservers/prod)

Flow Hooks place results in ./data/flowhooks regardless of env.
Should create sub-directory for each env (e.g. /data/flowhooks/test, /data/flowhooks/prod)

KVMs follow this pattern.

data
└── kvm
    └── env
        ├── prod
        │   └── GetLogValues
        └── test
            └── GetLogValues

Using importAll, if the data source is sparsely populated (e.g. missing targetservers) the tool exits prematurely. Happens for all entities.
Workaround is to use the individual commands in that case. See all_import_tasks.sh for the order.

with i am able to export only latest proxy and imported to new servers.
How i can export and import all the revisions of proxies and import them into now APIGEE system