change-table-schemas's Introduction

DynamoDB: Change Table Schemas

A tool to help migrate tables in DynamoDB

Steps to follow

Create a new table (let us call this NewTable), with the desired key structure, LSIs, GSIs.
Enable DynamoDB Streams on the original table
Associate a Lambda to the Stream, which pushes the record into NewTable. (This Lambda should trim off the migration flag in Step 5)
[Optional] Create a GSI on the original table to speed up scanning items. Ensure this GSI only has attributes: Primary Key, and Migrated (See Step 5).
Scan the GSI created in the previous step (or entire table) and use the following Filter:

FilterExpression = "attribute_not_exists(Migrated)"

Update each item in the table with a migrate flag (ie: “Migrated”: { “S”: “0” }, which sends it to the DynamoDB Streams (using UpdateItem API, to ensure no data loss occurs).

NOTE You may want to increase write capacity units on the table during the updates.
The Lambda will pick up all items, trim off the Migrated flag and push it into NewTable.
Once all items have been migrated, repoint the code to the new table
Remove original table, and Lambda function once happy all is good.

More information

https://www.abhayachauhan.com/2017/10/dynamodb-changing-table-schema/

change-table-schemas's People

Contributors

Stargazers

Watchers

change-table-schemas's Issues

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

How are you actually running the index.js file to avoid out of memory related errors? Our source table has around 15 million records. Seems like waiting for every single record to "load" and then launch the actual migration is not useful with such a large amount of records.

<--- Last few GCs --->

[6900:0x324add0]   158081 ms: Scavenge 1353.9 (1414.2) -> 1353.5 (1414.7) MB, 4.3 / 0.0 ms  (average mu = 0.180, current mu = 0.088) allocation failure 
[6900:0x324add0]   158088 ms: Scavenge 1354.2 (1414.7) -> 1353.8 (1415.7) MB, 3.1 / 0.0 ms  (average mu = 0.180, current mu = 0.088) allocation failure 
[6900:0x324add0]   158095 ms: Scavenge 1354.5 (1415.7) -> 1354.1 (1416.2) MB, 3.1 / 0.0 ms  (average mu = 0.180, current mu = 0.088) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x21bbf6d9e549 <JSObject>
    0: builtin exit frame: concat(this=0x0bcc7587df81 <JSArray[1157747]>,0x27b5b5dc6751 <JSArray[4785]>,0x0bcc7587df81 <JSArray[1157747]>)

    1: /* anonymous */ [0x387ded184991] [/home/ec2-user/migrate-dynamodb/scanTable.js:12] [bytecode=0x3557a19fdd61 offset=55](this=0x265da7b1aa09 <JSGlobal Object>,data=0x27b5b5dc6719 <Object map = 0xd8b743dd3d9>)
    2: StubFrame [pc: 0x31ead1941bf1]
    3:...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8c02c0 node::Abort() [node]
 2: 0x8c030c  [node]
 3: 0xad15de v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xad1814 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xebe752  [node]
 6: 0xebe858 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xeca982 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xecb2b4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xecdf21 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xe970b6 v8::internal::Factory::AllocateRawArray(int, v8::internal::PretenureFlag) [node]
11: 0xe9793a v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [node]
12: 0xea42f1 v8::internal::Factory::NewJSArrayStorage(v8::internal::Handle<v8::internal::JSArray>, int, int, v8::internal::ArrayStorageAllocationMode) [node]
13: 0xea45ec v8::internal::Factory::NewJSArray(v8::internal::ElementsKind, int, int, v8::internal::ArrayStorageAllocationMode, v8::internal::PretenureFlag) [node]
14: 0xe5db41 v8::internal::ElementsAccessor::Concat(v8::internal::Isolate*, v8::internal::Arguments*, unsigned int, unsigned int) [node]
15: 0xb5acc9  [node]
16: 0xb643c8 v8::internal::Builtin_ArrayConcat(int, v8::internal::Object**, v8::internal::Isolate*) [node]
17: 0x31ead195c17d 
Aborted
[ec2-user@ip-10-1-0-124 migrate-dynamodb]$ node --max-old-space-size=4096 index.js 
Received 11 items

Recommend Projects