moritanosuke / glacieruploader Goto Github PK

Hi,

Good tool,
I didnt read the API, is there a way to show the file name after uploaded (in list)?
Also, append the ability to delete archive in a selected vault

Ran

It would be awesome to have the Command part (very nice indeed) as part of a maven package so that it can be used in client code with ease.

Good job by the way!

Will you implement multipart uploads?

I found an archive id recently that started with a hyphen. From my reading the Jopt parser used in Glacieruploader picks this up as an arguement rather than an archive id. I dont know if there is a work around for this!?
-UMmj2Yz0eeZ4XQBqsgsARXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Exception in thread "main" joptsimple.UnrecognizedOptionException: 'U' is not a
recognized option
        at joptsimple.OptionException.unrecognizedOption(OptionException.java:88
)
        at joptsimple.OptionParser.validateOptionCharacters(OptionParser.java:47
2)
        at joptsimple.OptionParser.handleShortOptionCluster(OptionParser.java:42
1)
        at joptsimple.OptionParser.handleShortOptionToken(OptionParser.java:416
        at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:
56)
        at joptsimple.OptionParser.parse(OptionParser.java:379)
        at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:45)

To improve the handling of archives it would be nice if the glacieruploader remembers the uploaded archive ID and writes it into a file, maybe with additional metadata like Creation Date and the SHA256 TreeHash.

I'm constantly finding myself doing inventory listings just to get the archive IDs when I want to test downloads...

Just tried out the 0.1.1-beta1 version but that one does not allow a simple "-l".
Previously one called it without an argument to get a job-ID to be used to retried the inventory listing.

It seems that RequestArchivesListCommand#valid has got an added
options.has(optionParser.vault)

Adding a "-vault=backup" on the commandline and it works.

I do have vault=backup in my ~/.glacieruploaderrc but that seems to be ignored for
the "-l" option.

I removed "options.has(optionParser.vault) &&" in the valid()-method and
things worked again.

Is this a bug or have things changed how you now perform a "-l"?

I have a problem with deletion of archive when the archive id started from: -xxxxxxxxxxxxxx
I'm getting then INFO: Deleting archive null from vault xxxxx

Just wanted to leave a note here.

I tried listing my vaults and got the following error:

**$** java -jar glacieruploader-impl-0.1.1-SNAPSHOT-jar-with-dependencies.jar -s
INFO  Using end point: https://glacier.us-east-1.amazonaws.com
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/conn/socket/ConnectionSocketFactory
        at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.<init>(ApacheHttpClientFactory.java:42)
        at com.amazonaws.http.AmazonHttpClient.<clinit>(AmazonHttpClient.java:141)
        at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:162)
        at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:153)
        at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:138)
        at com.amazonaws.services.glacier.AmazonGlacierClient.<init>(AmazonGlacierClient.java:248)
        at com.amazonaws.services.glacier.AmazonGlacierClient.<init>(AmazonGlacierClient.java:227)
        at de.kopis.glacier.commands.AbstractCommand.<init>(AbstractCommand.java:51)
        at de.kopis.glacier.commands.CreateVaultCommand.<init>(CreateVaultCommand.java:41)
        at de.kopis.glacier.GlacierUploader.findAndExecCommand(GlacierUploader.java:105)
        at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:70)
Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.socket.ConnectionSocketFactory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 11 more

To resolve this I had to change the dependency of org.apache.httpcomponents.httpclient inside glacieruploader-impl's pom.xml to use version 4.5.2. This appears to already be done in the parent pom.

First of all, I want to thank you for your programm. It is really great and exactly what I was looking for! But I have a little problem with downloading files from Glacier. Upload work well, I have an archive ID and I see archive in vault. But when I try to download archive, I get an error:

INFO: Downloading archive LJNGbR8hSa... from vault MyVault...
Exception in thread "main" Status Code: 403, AWS Service: AmazonSNS, AWS Request
 ID: 478e0076-98a2-5c46-a7b5-4c8e42f20c8f, AWS Error Code: InvalidClientTokenId,
 AWS Error Message: The security token included in the request is invalid

Download command looks like this:

java -jar uploader-0.0.8-SNAPSHOT-jar-with-dependencies.jar --credentials aws.properties --endpoint https://glacier.eu-central-1.amazonaws.com/ --vault MyVault --download LJNGbR8hSaOQsyr9F5Sg6j... --target D:\9_PROJECTS_MY\glacieruploader-master\download.rar

File aws.properties contains valid accessKey and secretKey. Upload, multipart upload, create vault works great for me. What I'm doing wrong with downloading files?

The README.md contains two „Download archive“ sections with the exact same code sample (here and here).

Hello,

There is a way to define the path to .glacieruploaderrc ?

Regards,
Luis

The application is throwing this exception when launched:
java -jar uploader-0.0.8-SNAPSHOT-jar-with-dependencies.jar --credentials aws.properties

Exception in thread "main" java.lang.UnsupportedClassVersionError: de/kopis/glacier/GlacierUploader : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: de.kopis.glacier.GlacierUploader. Program will exit.

The next enhancement should be creation/deletion of vaults.

I want this tool to be available under GPL, but didn't add any licensing first.

http://docs.amazonwebservices.com/amazonglacier/latest/dev/uploading-an-archive-mpu-using-java.html says the following:

You can use the upload method to upload small or large archives. Depending on the archive size you are uploading,
this method determines whether to upload it in a single operation or use the multipart upload API to upload the
archive in parts.

Stuff like --credentials should be accessible via -c too. The preferences from #22 will be overridden by command line parameters

I was trying to use your uploader for testing Glacier. I was able to successfully build the uploader using maven but when I tried to get an inventory list (command is below) I received an error. Have you encountered this before? If so, any assistance pointing me in the proper direction is appreciated.

user1:/home/user1/glacierupload/glacieruploader/target# java -jar uploader-0.0.7-SNAPSHOT-jar-with-dependencies.jar --credentials /home/user1/glacierupload/backup_user_aws.properties -l
Nov 16, 2012 11:08:17 AM de.kopis.glacier.GlacierUploader main
INFO: Ooops, something is wrong with your setup.
Nov 16, 2012 11:08:17 AM de.kopis.glacier.GlacierUploader main
SEVERE: Something is wrong with the system configuration
java.net.MalformedURLException
at java.net.URL.(URL.java:619)
at java.net.URL.(URL.java:482)
at java.net.URL.(URL.java:431)
at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:65)
Caused by: java.lang.NullPointerException
at java.net.URL.(URL.java:524)

What is the minimum version of Java that I need to have to run this? I am getting this error:

Exception in thread "main" java.lang.UnsupportedClassVersionError: de/kopis/glacier/GlacierUploader : Unsupported major.minor version 51.0

This is on my Linux machine where I currently have the current version installed:

java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)

This is the machine where I currently have many other java programs and Tomcat running.

Maybe it's worth the effort to optionally create an MD5 checksum when uploading a new archive. The user can then store this and use the value to verify a download.

UPDATE: Regarding my original issue below, it appears that using the multipartupload option, glacieruploader does not use the configuration file for endpoint and vault defaults.

If I execute the following command below, it works (i.e using inline options -e and -v):

java -jar glacieruploader.jar -e https://glacier.us-east-1.amazonaws.com -v masterimages -m /backup/archives/archive_1_425.zip -p 134217728

------ ORIGINAL ISSUE ------
When I try to upload using multi-part with this command:

java -jar glacieruploader.jar --multipartupload /backup/archives/archive_1_425.zip

I get the following response:

Feb 1, 2013 1:06:52 PM de.kopis.glacier.GlacierUploader findAndExecCommand
INFO: Using end point: https://glacier.us-east-1.amazonaws.com
Feb 1, 2013 1:06:52 PM de.kopis.glacier.commands.HelpCommand exec
INFO: Ooops, can't determine what you want to do. Check your options.
Do not forget that --vault and --endpoint are mandatory for all commands.

This happens when I use both the -m and --mulipartupload options.

I've confirmed that the configuration and credentials are being read since I can execute the -l option:

java -jar glacieruploader.jar -l

Response:

Feb 1, 2013 1:08:39 PM de.kopis.glacier.GlacierUploader findAndExecCommand
INFO: Using end point: https://glacier.us-east-1.amazonaws.com
Feb 1, 2013 1:08:39 PM de.kopis.glacier.commands.RequestArchivesListCommand startInventoryListing
INFO: Starting inventory listing for vault masterimages...
Feb 1, 2013 1:08:40 PM de.kopis.glacier.commands.RequestArchivesListCommand startInventoryListing
SEVERE: Inventory retrieval jobs for vault arn:aws:glacier:us-east-1:187559545565:vaults/masterimages cannot be initiated yet, as Amazon Glacier has not yet generated an initial inventory for this vault.

Also, I've confirmed that the file path is valid, the archive exists and read permission is allowed for everyone.

This is using the JDK6 version you made available.

Thanks,
Manny

Questions:
1.) Does Glacier Uploader actually make certain that the checksum after upload matches the checksum that Amazon reports?

2.) Can Glacier Uploader resume properly after upload is interrupted?

If the answer to (1) is NO, what is the preferred way to check the checksum(manually?)

Thanks.

It would be nice to have a way to output the data in JSON to make it machine-parsable.

The use case here is to have a script that operates at a high-level while glacieruploader handles all of the operations/credentials.

Part of the changes that seems to be done include:

• adding an option: "--json-output" and "-j"
• redirecting the logging output to the error stream (to separate data and logs)
• passing a new boolean to all commands: json_output
• displaying in JSON when asked

Exception in thread "main" Status Code: 403, AWS Service: AmazonGlacier, AWS Request ID: XXX, AWS Error Code: InvalidSignatureException, AWS Error Message: Signature expired: 20120831T062211Z is now earlier than 20120831T063133Z (20120831T063633Z - 5 min.)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:583)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:317)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:167)
    at com.amazonaws.services.glacier.AmazonGlacierClient.invoke(AmazonGlacierClient.java:1949)
    at com.amazonaws.services.glacier.AmazonGlacierClient.uploadMultipartPart(AmazonGlacierClient.java:1581)
    at com.amazonaws.services.glacier.transfer.ArchiveTransferManager.uploadInMultipleParts(ArchiveTransferManager.java:441)
    at com.amazonaws.services.glacier.transfer.ArchiveTransferManager.upload(ArchiveTransferManager.java:249)
    at com.amazonaws.services.glacier.transfer.ArchiveTransferManager.upload(ArchiveTransferManager.java:210)
    at de.kopis.glacier.CommandLineGlacierUploader.upload(CommandLineGlacierUploader.java:46)
    at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:48)

Can this be handled gracefully?

INFO: Retrying request
14-03-2014 04:41:36 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Unable to execute HTTP request: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:375)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:363)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:801)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:773)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:119)
at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:169)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:119)
at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:102)
at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:140)
at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
at org.apache.http.impl.conn.AbstractClientConnAdapter.sendRequestEntity(AbstractClientConnAdapter.java:227)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:712)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:280)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:165)
at com.amazonaws.services.glacier.AmazonGlacierClient.invoke(AmazonGlacierClient.java:1949)
at com.amazonaws.services.glacier.AmazonGlacierClient.uploadMultipartPart(AmazonGlacierClient.java:1581)
at com.amazonaws.services.glacier.transfer.ArchiveTransferManager.uploadInMultipleParts(ArchiveTransferManager.java:445)
at com.amazonaws.services.glacier.transfer.ArchiveTransferManager.upload(ArchiveTransferManager.java:249)
at com.amazonaws.services.glacier.transfer.ArchiveTransferManager.upload(ArchiveTransferManager.java:210)
at de.kopis.glacier.commands.UploadArchiveCommand.upload(UploadArchiveCommand.java:49)
at de.kopis.glacier.commands.UploadArchiveCommand.exec(UploadArchiveCommand.java:64)
at de.kopis.glacier.GlacierUploader.findAndExecCommand(GlacierUploader.java:122)
at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:72)

Hi

I can't download uploader-0.0.3-SNAPSHOT-jar-with-dependencies.jar file.

I have a "files was not found" message.

regards

I'm trying to get a listing but I get everytime a OutOfMemoryError:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Unknown Source)
        at java.lang.String.<init>(Unknown Source)
        at java.lang.StringBuffer.toString(Unknown Source)
        at java.io.BufferedReader.readLine(Unknown Source)
        at java.io.BufferedReader.readLine(Unknown Source)
        at de.kopis.glacier.commands.ReceiveArchivesListCommand.retrieveInventoryListing(ReceiveArchivesListCommand.java:61)
        at de.kopis.glacier.commands.ReceiveArchivesListCommand.exec(ReceiveArchivesListCommand.java:81)
        at de.kopis.glacier.GlacierUploader.findAndExecCommand(GlacierUploader.java:119)
        at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:73)

I used this command line command:

X:\Tools>java -Xmx2048m -jar uploader-0.0.8-SNAPSHOT-jar-with-dependencies.jar --credentials x:\tools\aws.properties -v backup --endpoint https://glacier.eu-west-1.amazonaws.com --list-inventory xxxxxxxxxxxxx

Like the tool - it's straightforward to use.
Would it be feasible to include some mechanism whereby you could get a list of the available vaults?

It's possible view file list of archive? When view inventory , i see only inventory ID but not the file name...

My AWS account has Multi Factor Authentication activated it would be nice to add support in your credentials file for the Session token and ephemeral accesskey and secretkey required for MFA enabled API access

With the AWS SDK CLI it works like this:

aws sts get-session-token --serial-number "arn:aws:iam::[ACCOUNT_NUMBER]:mfa/[IAM_USERNAME]" --token-code [COE_FROM_MFA_DEVICE]

I put in my code from Google Authenticator tied to my AWS account and it spits out some JSON:

{
    "Credentials": {
        "SecretAccessKey": "[Secret Key]", 
        "SessionToken": "[Session Token]", 
        "Expiration": "2017-02-25T09:15:03Z", 
        "AccessKeyId": "[Access key ID]"
    }
}

Then I put these keys in my credential file

Otherwise to use your glacieruploader program I have to add another IAM user without the MFA enabled to use a static accesskey and secretkey

Hello

Is there a typo in the help file, the --file option was replaced with --upload

regards

roberto

The current version looks for a file named ´aws.properties´ in the classpath. That was the easiest way to implement it. I'll figure out how to use your own file in the next version.

Hello,

I don't seem to be able to download I keep getting a SNS error asking for it to be in the local region, both my archive and my SNS are both in EU-West.

Downloading archive 9a3b0eff1691c981bf9498348252c7d7ee7ee04ab871068cc435a127c5c82da1 from vault vm-test...
Exception in thread "main" Status Code: 400, AWS Service: AmazonGlacier, AWS Request ID: stAJ8r84tf0iP0hdcXlprOknnS-NCApAjKliJiN7Se5_CtM, AWS Error Code: InvalidParameterValueException, AWS Error Message: SNS topics must be in the local AWS region: arn:aws:sns:us-east-1:010459735181:glacier-archive-transfer-1346315137037
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:583)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:317)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:167)
at com.amazonaws.services.glacier.AmazonGlacierClient.invoke(AmazonGlacierClient.java:1949)
at com.amazonaws.services.glacier.AmazonGlacierClient.initiateJob(AmazonGlacierClient.java:1197)
at com.amazonaws.services.glacier.transfer.ArchiveTransferManager.download(ArchiveTransferManager.java:329)
at com.amazonaws.services.glacier.transfer.ArchiveTransferManager.download(ArchiveTransferManager.java:281)
at de.kopis.glacier.GlacierArchiveDownloader.download(GlacierArchiveDownloader.java:47)
at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:60)

Thanks

Kris.

I have found a problem with the HumanReadableSize class in that it does not
handle large files properly. It seems to happen with files > 10GB in size.
I have two examples (cut from a glacier -l command).

74397132 (70.96MB)
9807947179 (9.14GB)
19008123152 (7.00kB)
73476694570 (7.00kB)

When you call parse("73476694570") you will end up with a sizeAsDouble / 1024
as the value 7.175458454101562E7. Looking further into the sanitize() method
it seems that it cannot handle doubles in exponential format, e.g. 8.1234E8.

Of the following tests the first two fails on my machine.
(Put them in HumanReadableSizeTest.java)

@Test
public void terabyteLargeSizeFailed() {
    assertEquals("68TB", HumanReadableSize.parse("75240135239680"));
}

@Test
public void gigabyteLargeSizeFailed() {
    assertEquals("68GB", HumanReadableSize.parse("73476694570"));
}

@Test
public void megabyteLargeSizeFailed() {
    assertEquals("68.44MB", HumanReadableSize.parse("71754584"));
}

Please add support for file deletion.

I'm having troubles downloading archived files and viewing the vault's inventory.

Downloads just hang and no target file is created.

$ java -jar ./glacieruploader-impl-0.1.1-SNAPSHOT-jar-with-dependencies.jar --download LA_z3GZKbJdeJnaZV5Ur5UihRfuX8zWu-cm2n1ioRxDubpyVqWrylTklCrteec0VQKQrOb9GbIhOmDFpNjknnIw5FVXbe66XK4c2PkR1tDJ1PiKs-luab4GZiYvPQg04xiRX3DuPeg --target /home/dwiest/test
INFO Using end point: https://glacier.us-east-1.amazonaws.com
INFO Downloading archive LA_z3GZKbJdeJnaZV5Ur5UihRfuX8zWu-cm2n1ioRxDubpyVqWrylTklCrteec0VQKQrOb9GbIhOmDFpNjknnIw5FVXbe66XK4c2PkR1tDJ1PiKs-luab4GZiYvPQg04xiRX3DuPeg from vault Test...

I get a job id back when I try to list the vault's inventory, but I can't view the job's results.

$ java -jar ./glacieruploader-impl-0.1.1-SNAPSHOT-jar-with-dependencies.jar --list-inventory
INFO Using end point: https://glacier.us-east-1.amazonaws.com
INFO Starting inventory listing for vault Test...
INFO Inventory Job created with ID
Vk5Auz3lAPjikTtDy817MfZrSZohQYuAMa8h1NCe-y-Wh7m-7actsG7hFuKgl8xlZ5lwY4CgzqaFg2cr0VxkNuF1mpyN

[dwiest@prod-sftp-01 ~]$ java -jar ./glacieruploader-impl-0.1.1-SNAPSHOT-jar-with-dependencies.jar --list-inventory Vk5Auz3lAPjikTtDy817MfZrSZohQYuAMa8h1NCe-y-Wh7m-7actsG7hFuKgl8xlZ5lwY4CgzqaFg2cr0VxkNuF1mpyN
INFO Using end point: https://glacier.us-east-1.amazonaws.com
INFO Retrieving inventory for job id Vk5Auz3lAPjikTtDy817MfZrSZohQYuAMa8h1NCe-y-Wh7m-7actsG7hFuKgl8xlZ5lwY4CgzqaFg2cr0VxkNuF1mpyN...
ERROR The job is not currently available for download: Vk5Auz3lAPjikTtDy817MfZrSZohQYuAMa8h1NCe-y-Wh7m-7actsG7hFuKgl8xlZ5lwY4CgzqaFg2cr0VxkNuF1mpyN (Service: AmazonGlacier; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: cBdo5-aAYrNxvTwl4YGpMlwdBr4HusDpATFTB4e1whXs1ro)
com.amazonaws.services.glacier.model.InvalidParameterValueException: The job is not currently available for download: Vk5Auz3lAPjikTtDy817MfZrSZohQYuAMa8h1NCe-y-Wh7m-7actsG7hFuKgl8xlZ5lwY4CgzqaFg2cr0VxkNuF1mpyN (Service: AmazonGlacier; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: cBdo5-aAYrNxvTwl4YGpMlwdBr4HusDpATFTB4e1whXs1ro)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1386)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:939)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:714)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:465)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:427)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:376)
at com.amazonaws.services.glacier.AmazonGlacierClient.doInvoke(AmazonGlacierClient.java:3290)
at com.amazonaws.services.glacier.AmazonGlacierClient.invoke(AmazonGlacierClient.java:3259)
at com.amazonaws.services.glacier.AmazonGlacierClient.getJobOutput(AmazonGlacierClient.java:1536)
at de.kopis.glacier.commands.ReceiveArchivesListCommand.retrieveInventoryListing(ReceiveArchivesListCommand.java:52)
at de.kopis.glacier.commands.ReceiveArchivesListCommand.exec(ReceiveArchivesListCommand.java:74)
at de.kopis.glacier.GlacierUploader.findAndExecCommand(GlacierUploader.java:123)
at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:70)

I spent a bit of time banging my head up against this tonight. Multipart upload handles file names differently than --calculate and --upload.

java -jar /Applications/Utilities/CLI/glacieruploader.jar --endpoint https://glacier.us-west-2.amazonaws.com --vault MacBook_Offload --credentials /Users/nickb/Documents/Glacier\ Offload/.aws.credentials -a ./To\ Offload/2014-05-21\ 02.15.23.mov 
Oct 5, 2014 11:33:01 PM de.kopis.glacier.GlacierUploader findAndExecCommand
INFO: Using end point: https://glacier.us-west-2.amazonaws.com
Oct 5, 2014 11:33:07 PM de.kopis.glacier.commands.TreeHashArchiveCommand calculateTreeHash
INFO: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Calculating a Hash with -a is successful

java -jar /Applications/Utilities/CLI/glacieruploader.jar --endpoint https://glacier.us-west-2.amazonaws.com --vault MacBook_Offload --credentials /Users/nickb/Documents/Glacier\ Offload/.aws.credentials -u ./To\ Offload/2014-05-21\ 02.15.23.mov
Oct 5, 2014 11:33:50 PM de.kopis.glacier.GlacierUploader findAndExecCommand
INFO: Using end point: https://glacier.us-west-2.amazonaws.com
Oct 5, 2014 11:33:51 PM de.kopis.glacier.commands.UploadArchiveCommand upload
INFO: Starting to upload ./To Offload/2014-05-21 02.15.23.mov to vault MacBook_Offload...
Oct 5, 2014 11:49:16 PM de.kopis.glacier.commands.UploadArchiveCommand upload
INFO: Uploaded archive XXXXXXXXXXXXXXXXXXXX

Uploading a file with -u is also successful.

java -jar /Applications/Utilities/CLI/glacieruploader.jar --endpoint https://glacier.us-west-2.amazonaws.com --vault MacBook_Offload --credentials /Users/nickb/Documents/Glacier\ Offload/.aws.credentials -m ./To\ Offload/2014-05-21\ 02.15.23.mov
Oct 5, 2014 11:33:26 PM de.kopis.glacier.GlacierUploader findAndExecCommand
INFO: Using end point: https://glacier.us-west-2.amazonaws.com
Oct 5, 2014 11:33:27 PM de.kopis.glacier.commands.UploadMultipartArchiveCommand upload
INFO: Multipart uploading To (0.00B) to vault MacBook_Offload with part size 16777216 (16.00MB).
Oct 5, 2014 11:33:28 PM de.kopis.glacier.commands.UploadMultipartArchiveCommand initiateMultipartUpload
INFO: Upload ID (token): w2xx9k8X3xRjZrlruTrIyfHAOYc2TQzZlbVybX2S9v8tS8fSDf0JNFwPmshTzoms25vBs6_xSncBhpQqA7nu9eYDg7Yf
Oct 5, 2014 11:33:28 PM de.kopis.glacier.commands.UploadMultipartArchiveCommand upload
SEVERE: Something went wrong while multipart uploading ./To../To (No such file or directory)
java.io.FileNotFoundException: ./To (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at de.kopis.glacier.commands.UploadMultipartArchiveCommand.uploadParts(UploadMultipartArchiveCommand.java:117)
    at de.kopis.glacier.commands.UploadMultipartArchiveCommand.upload(UploadMultipartArchiveCommand.java:71)
    at de.kopis.glacier.commands.UploadMultipartArchiveCommand.exec(UploadMultipartArchiveCommand.java:175)
    at de.kopis.glacier.GlacierUploader.findAndExecCommand(GlacierUploader.java:119)
    at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:73)
Oct 5, 2014 11:33:28 PM de.kopis.glacier.commands.UploadMultipartArchiveCommand upload
INFO: Multipart uploading 2014-05-21 (0.00B) to vault MacBook_Offload with part size 16777216 (16.00MB).
Oct 5, 2014 11:33:29 PM de.kopis.glacier.commands.UploadMultipartArchiveCommand initiateMultipartUpload
INFO: Upload ID (token): c4RbEc1O2nUb-D96_HbKsKz9glta8k99a3jzXYY91Xn_FIESxWjNw0ZLm_qoV7Q0ijotTVwPx37L3wVzIHpuiZVMmcrF
Oct 5, 2014 11:33:29 PM de.kopis.glacier.commands.UploadMultipartArchiveCommand upload
SEVERE: Something went wrong while multipart uploading Offload/2014-05-21.Offload/2014-05-21 (No such file or directory)
java.io.FileNotFoundException: Offload/2014-05-21 (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at de.kopis.glacier.commands.UploadMultipartArchiveCommand.uploadParts(UploadMultipartArchiveCommand.java:117)
    at de.kopis.glacier.commands.UploadMultipartArchiveCommand.upload(UploadMultipartArchiveCommand.java:71)
    at de.kopis.glacier.commands.UploadMultipartArchiveCommand.exec(UploadMultipartArchiveCommand.java:175)
    at de.kopis.glacier.GlacierUploader.findAndExecCommand(GlacierUploader.java:119)
    at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:73)
Oct 5, 2014 11:33:29 PM de.kopis.glacier.commands.UploadMultipartArchiveCommand upload
INFO: Multipart uploading 02.15.23.mov (0.00B) to vault MacBook_Offload with part size 16777216 (16.00MB).
Oct 5, 2014 11:33:29 PM de.kopis.glacier.commands.UploadMultipartArchiveCommand initiateMultipartUpload
INFO: Upload ID (token): 1T73ZO8LCKDviu6uzqnG3zeCQHQWc-ezUKtoNnSGzACTz_G8wnHjaZMdWYSdPQnGj4AwyzdwLpi7ELUuBG_I7YaHeeYi
Oct 5, 2014 11:33:29 PM de.kopis.glacier.commands.UploadMultipartArchiveCommand upload
SEVERE: Something went wrong while multipart uploading 02.15.23.mov.02.15.23.mov (No such file or directory)
java.io.FileNotFoundException: 02.15.23.mov (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at de.kopis.glacier.commands.UploadMultipartArchiveCommand.uploadParts(UploadMultipartArchiveCommand.java:117)
    at de.kopis.glacier.commands.UploadMultipartArchiveCommand.upload(UploadMultipartArchiveCommand.java:71)
    at de.kopis.glacier.commands.UploadMultipartArchiveCommand.exec(UploadMultipartArchiveCommand.java:175)
    at de.kopis.glacier.GlacierUploader.findAndExecCommand(GlacierUploader.java:119)
    at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:73)

Uploading a file with -m does something odd trying to handle the spaces. I'm really not quite sure where its getting what it is getting.

Precious version
CreationDate: 2016-02-14T20:34:00Z
Size: 73476694570 (68.44GB)

0.1.1.-beta1
CreationDate: 2016-02-14T20:34:00Z
Size: 462250538 (440.84MB)

Converting the sizes to hex reveals a problem with the hi-order bytes.
For example, the sizes from above:
printf "%x\n" 73476694570
111b8d622a

printf "%x\n" 462250538
1b8d622a

Some more examples:
printf "%x\n" -1043299672
ffffffffc1d082a8
printf "%x\n" 24726504104
5c1d082a8

printf "%x\n" -745626463
ffffffffd38ea4a1
printf "%x\n" 12139275425
2d38ea4a1

printf "%x\n" 2096187049
7cf146a9
printf "%x\n" 70815663785
107cf146a9

When uploading the SHA256TreeHash could be computed on-the-fly to be able to check the archive integrity later using --list-inventory

Hi,
I made a multipart upload of a file and got some connections exceptions like this:

INFO  Part 378/393 (bytes 6325010432-6341787647/*) uploaded, checksum: 78f12cb8a20d5228937c881fb61635601aa6f31600857d54c575497939ef8b61
INFO  Part 379/393 (bytes 6341787648-6358564863/*) uploaded, checksum: 8dd1f864bfb627f6a5f26a5ce159bb4bf536ca473e5c7412b33bc048323da973
INFO  Unable to execute HTTP request: Connection reset
java.net.SocketException: Connection reset
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
....
INFO  Part 380/393 (bytes 6358564864-6375342079/*) uploaded, checksum: fb5e8d8ec6d4fbf1cb1a296e22d47ab941230f8cb992ce57f87533443144e36f
INFO  Part 381/393 (bytes 6375342080-6392119295/*) uploaded, checksum: 0085299047da43ffd1db18e6d73a5f34e73bcdaf675dd96b8ed06c4863df710b
INFO  Part 382/393 (bytes 6392119296-6408896511/*) uploaded, checksum: 5d75528e5b9a42aeb292317efbf068a6c0192af23b0635722becf15084265c44
INFO  Part 393/393 (bytes 6576668672-6584865875/*) uploaded, checksum: 854f1a7886fabe7082a3ab61913b512b0818bf78600df19b3ea6a1018d29f370
INFO  Uploaded Archive ID: UPFXHXsHKmNp1VQgP3hSlKk_1zHVrzosnPjQwCMtI2WvFE4VhQpWqxhg0ByvGNoBy1xzQeFClojerdU3O7CZGBbIMyg5rDsWKweD9iAGXDTl4tLn_syYo_II2bb-14CGdYlBHLkHag
INFO  Local Checksum: 97339a098836c8d2fc5e486c476e1e9cb5dd79117fa8b24706d5d181aa0c7c94
INFO  Remote Checksum: 97339a098836c8d2fc5e486c476e1e9cb5dd79117fa8b24706d5d181aa0c7c94
INFO  Checksums are identical, upload succeeded.

However the local and remote checksum are the same.

So, glacieruploader retry the connection without a retry message or my file is incomplete in glacier?

Thanks in advance for your answer.

Can their be better display of the data that comes back form list inventory?

I think it's worth to have a progressbar (or any kind of feedback) when uploading large files. Passing everthing through an InputStream should enable a progress output like wget.

When creating a new vault in v0.0.4 the new vault is not created in the region of the used enpoint, but in US EAST (the default region?).

Hi,
I am trying out glacieruploader and was trying to issue an inventor listing command and the
job ID I received started with a hyphen...

~/bin/glacier-cmd.sh -l
2016-feb-07 10:00:47 de.kopis.glacier.GlacierUploader findAndExecCommand
INFO: Using end point: https://glacier.eu-west-1.amazonaws.com
2016-feb-07 10:00:48 de.kopis.glacier.commands.RequestArchivesListCommand startInventoryListing
INFO: Starting inventory listing for vault backup...
2016-feb-07 10:00:50 de.kopis.glacier.commands.RequestArchivesListCommand startInventoryListing
INFO: Inventory Job created with ID
-zoDBAz5dT5tyfv9Rf-pvGbijIqDs_gpL30ud2zq1-qa4Azz11a-ABCGIS4bzbUmCzeg2U3xMrp9W6PWHV9UTaKSTX-g

Later when trying to use that job ID with the "-l" option I got
~/bin/glacier-cmd.sh -l -zoDBAz5dT5tyfv9Rf-pvGbijIqDs_gpL30ud2zq1-qa4Azz11a-ABCGIS4bzbUmCzeg2U3
xMrp9W6PWHV9UTaKSTX-g
Something went wrong parsing the arguments

It looks like that the ID is parsed as an option and that fails.
Is there some way to quote an ID to prevent the command line parsing library
from treating the ID as an command option?

Stuff like endpoint and credentials should be configurable in a preferences file like ~/.glacieruploaderrc.

I'll use glacier to do day by day backups.
In this scenario it will be nice to have the ability to schedule a deletion of file older than X days ( where X > 90 to avoid deletion fee :-).

Do you think that this feature could be useful also to others ?

Hello,

I know this isn't quite the right spot, but I'm not sure how else to contact you. I didn't see an email address or web form on your website.

It appears the Binary that you've linked to on http://download.kopis.de.s3-website-eu-west-1.amazonaws.com/glacieruploader/index.html is out of date. (It appears to be from 2013 Jan 17.) I've downloaded and tested it and I don't see the fix for Issue #40. I also don't see a way to get the binary from travis-ci.org

Is there a better location for the binary?

Thank you,
Nicholas Barnard

It would be nice to have a status of the upload progress at the command line. Is this doable - I would be willing to contribute. I was thinking that every 30 seconds it prints a % complete. Sorry - I know this is more of a think-tank than an issue

Hi,
Thank you for the great software.
Would it be possible to use AWS role for EC2 (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials) instead of providing hard coded credentials in aws.properties file?

I'm trying to use multipart upload option unfortunately I'm getting this error.

INFO: Multipart uploading file.tar.gz (1.13GB) to vault XYZ with part size 134217728 (128.00MB).
Feb 01, 2013 4:06:47 PM de.kopis.glacier.commands.UploadMultipartArchiveCommand initiateMultipartUpload
INFO: Upload ID (token): xxxxxxx
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at de.kopis.glacier.commands.UploadMultipartArchiveCommand.uploadParts(UploadMultipartArchiveCommand.java:124)
at de.kopis.glacier.commands.UploadMultipartArchiveCommand.upload(UploadMultipartArchiveCommand.java:70)
at de.kopis.glacier.commands.UploadMultipartArchiveCommand.exec(UploadMultipartArchiveCommand.java:170)
at de.kopis.glacier.GlacierUploader.findAndExecCommand(GlacierUploader.java:119)
at de.kopis.glacier.GlacierUploader.main(GlacierUploader.java:73)

I tried to change size of cheap size but without success. I'm still getting the same error.

I'm using version 0.0.8
java: J2SE 7

java version "1.7.0_11"
Java(TM) SE Runtime Environment (build 1.7.0_11-b21)
Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)