amleszk / appdailysales Goto Github PK

Add option (most likely a command line option) to specify where to store
the download file.

Add option to unzip the downloaded file. 

Original suggestion from ameir on the iPhone dev forums:

"Also if you add:

  import os

to the top and

  os.system('gunzip '+ filename)

to the bottom, it will automatically unzip the file for you as well."

How about auto-naming the file with the date or something? Apple's name is
a tad obscure ;)...

Hi. I've attached my modified version of 2.7, which includes a pair of new 
features.

- Automatically creates nested directories added to format (-f) strings (eg. -f 
%Y/%Y-%m/Daily-%Y-%m-%d.txt).  Works with outputDirectory (-o).
- New command line option "-n". Used with format (-f), any file that already 
exists isn't downloaded.

So now, combined with -d (days to download), you can make a single daily 
request for a full 14 days of reporting (-d 14).  Any days missed as a result 
of server/generation errors get filled in the next time appdailysales is run.

Enjoy. :)

Mike Kasprzak
Sykhronics Entertainment

Hi, 

This isn't an issue inasmuch as a general question.

I'm looking to (as the subject implies) take the data from the downloaded
report and load that into an SQL database.

I was just about to start writing a perl script to do the legwork but
figured it'd be worth inquiring to see if anyone has already written such a
script, or if there's a better way to do it than via perl script.

Any insight would be great.

If no such script or tool exists I'll be sure to post a link to my script
once complete.

What steps will reproduce the problem?
1. It used to work fine before 11/17/2009, however, It always says 'try 
again later' from that day, I can download the daily report manually from 
the ittues website. could you please identify this issue?

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
OS: CentOS 4.6, 
Python 2.6.4, 
appdailysales: 1.1

Please provide any additional information below.
Command: (executed on 11/30/1009)
python appdailysales.py -a MYACCOUNT -p PASSWORD -o . -D 11/27/2009 -u

Output:
11/27/2009 report is not available - try again later.
1 report(s) not available - try again later

What steps will reproduce the problem?

1. Run the script


What is the expected output? What do you see instead?

Expected output: report file should be downloaded.
Actual output:

Traceback (most recent call last):
  File "appdailysales.py", line 353, in <module>
    sys.exit(main())
  File "appdailysales.py", line 346, in main
    downloadFile(options)
  File "appdailysales.py", line 247, in downloadFile
    urlDownload = urlBase % match[0]
IndexError: list index out of range


What version of the product are you using? On what operating system?

v1.10

Please provide any additional information below.

Apple has apparently changed the way their reporting system works, or so I'm 
told.

What steps will reproduce the problem?
1. while downloading the daily sales reports, it is not working i am getting 
html file with error message as 
"There is no Sales report available for the selected period"

Version 2.7


i think there is an update in the server now i am unable to download daily 
reports

Regards.
Ramu.

What steps will reproduce the problem?
1. downloaded v 2.0 from site
2. run command: python appdailysales.py -a my_iTC_username -p my_iTC_password -v

What is the expected output? What do you see instead?
Expected: connection to iTC and download of stats file.
Instead:
-- begin script --
Traceback (most recent call last):
  File "_appdailysales.py", line 358, in <module>
    sys.exit(main())
  File "_appdailysales.py", line 351, in main
    downloadFile(options)
  File "_appdailysales.py", line 246, in downloadFile
    dailyName = match[0]
IndexError: list index out of range

What version of the product are you using? On what operating system?
2.0. Mac OS X 10.6.4. Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) [GCC 
4.2.1 (Apple Inc. build 5646)] on darwin

Please provide any additional information below.
Also tried it on a rackspace cloud server and gave the same results.

The current password entry using sys.stdin.read() is a bit clunky since there's 
no prompt and I'm not sure how to end input.  A one-line change to 
getpass.getpass() would make this much nicer.

It seems Apple updates their iPhone Developer Program License Agreement several 
times a year.  When they do, a iPhone Developer Program License Agreement 
Update window is loaded directly after logging in.  Would it be possible to 
have the script recognize when the license agreement window comes up and 
automatically select the "continue" option?

Currently, the script produces the following error if one has not accepted the 
license agreement yet:

Unable to load the sales report web page at this time. A number of reasons can 
cause this including delayed reporting and unsigned contracts. Try again later 
or sign into iTunes Connect and verify access.
Traceback (most recent call last):
  File "appdailysales.py", line 391, in <module>
    sys.exit(main())
  File "appdailysales.py", line 384, in main
    downloadFile(options)
  File "appdailysales.py", line 264, in downloadFile
    dailyName = match[0]
IndexError: list index out of range

Apparently, Apple changed their html on the itunesconnect site.  I modified
my local copy to work again.  See the following forum post for details.

https://devforums.apple.com/message/15975#15975

What steps will reproduce the problem?
1. Running the program (the bug is random, associated with non-correctly 
open/closed connections perhaps)
2.
3.

What is the expected output? What do you see instead?

Expected output is to simply download the report.

Instead it says (on verbose):

-- begin script --
Signing into iTunes Connect web site.
Accessing Sales and Trends reporting web site.
Unable to find default vendor page.
Traceback (most recent call last):
  File "./appdailysales.py", line 476, in <module>
    sys.exit(main())
  File "./appdailysales.py", line 469, in main
    downloadFile(options)
  File "./appdailysales.py", line 272, in downloadFile
    viewState = match[0]
IndexError: list index out of range



What version of the product are you using? On what operating system?
2.6, Mac OSX 10.6

Please provide any additional information below.

I'm also getting an additional error:

Traceback (most recent call last):
  File "/Users/johannes/Documents/Dev/Sales/appdailysales.py", line 476, in <module>
    sys.exit(main())
  File "/Users/johannes/Documents/Dev/Sales/appdailysales.py", line 469, in main
    downloadFile(options)
  File "/Users/johannes/Documents/Dev/Sales/appdailysales.py", line 417, in downloadFile
    filebuffer = gzipIO.read()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/gzip.py", line 212, in read
    self._read(readsize)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/gzip.py", line 255, in _read
    self._read_gzip_header()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/gzip.py", line 156, in _read_gzip_header
    raise IOError, 'Not a gzipped file'
IOError: Not a gzipped file

When using a custom format for file download and the date being accessed does 
not have a report available for it.

What steps will reproduce the problem?
1. Just type appdailysales.py -h
2.
3.

What is the expected output? What do you see instead?
File "appdailysales.py", line 117
raise AttributeError, attrname

What version of the product are you using? On what operating system?
I downloaded 1.10 but the appdailysales.py file contains "Version 1.9" in 
the script itself (perhaps someone forgot to change this?)

Running on Windows with Python 3.1.1

Please provide any additional information below.

It would be great to be able to have the option of not only downloading a 
certain number of days, 
or a certain date, but to also download all reports which haven't been 
downloaded yet. I added this 
functionality in and have added a patch to this issue which adds this 
functionality.

The script fails downloading the sales report when "the new" iPhone Developer 
Program License Agreement Update has to be accepted with pressing the 
"continue" button to get to your dashboard.

I uploaded the iTunes Connect continue page HTML source 
http://gist.github.com/612253

The button to press is in line 131.

The error msg from the script is:

-- begin script --
Signing into iTunes Connect web site.
Accessing Sales and Trends reporting web site.
Unable to find the Sales and Trends URL.
Traceback (most recent call last):
  File "appdailysales.py", line 490, in <module>
    sys.exit(main())
  File "appdailysales.py", line 483, in main
    downloadFile(options)
  File "appdailysales.py", line 266, in downloadFile
    backtrackedId = match[0]
IndexError: list index out of range


Script (v. 2.5) is running on Python 2.7 on WinXP SP3

What steps will reproduce the problem?
1. appdailsales.py -D 02/25/2010
2.
3.

What is the expected output? What do you see instead?

$ ./appdailysales.py -D 02/25/2010
Traceback (most recent call last):
  File "./appdailysales.py", line 354, in <module>
    sys.exit(main())
  File "./appdailysales.py", line 347, in main
    downloadFile(options)
  File "./appdailysales.py", line 240, in downloadFile
    urlDownload = urlBase % form['action']
TypeError: 'NoneType' object is unsubscriptable



What version of the product are you using? On what operating system?
Version 1.9, Mac OSX 10.6.mumble

Please provide any additional information below.

Apple has interposed a new page between "Sales And Trends" and the expected
page in order to advise developers that data from 02/25/2010 is currently
unavailable.  Since this page is unexpected, appdailysales blows up:

  form = soup.find( 'form', attrs={'name': 'frmVendorPage' } )

form is None since those attrs are not present in the interposed page.

I am under the impression that the interposed page is a temporary
situation, so this issue is more to reassure other folks rather than a
petition for change. 

Add option to specify the report date. Add bonus option that allows
specifying a number representing the "days ago" value used to calculate the
report date. 1 would be yesterday's date.  2 would be the day before
yesterday. 3 would be the day before that, and so on.

Hey,

I've been experiencing some issues with the script at the moment, has anyone 
else?
The "Unable To Find Sales And Trends URL" error keeps showing

I've waited for about 10 hours but the error still happens

Maybe change to iTunesconnect?

Thanks

What steps will reproduce the problem?
 In case of https proxy required

What is the expected output? What do you see instead?

 Please commit my patch(attached file)
 I added comments '# proxy support' on the modified sentences.

What version of the product are you using? On what operating system?

 latest version (2.8)

Please provide any additional information below.

This all started because I downloaded v2.7 of the appdailysales.py script 
earlier in the week and was able to successfully download sales reports from 
itunes connect using the script. As I developed an automated script to pull 
reports daily via the appdailysales.py script I noticed I was unable to pull 
reports using the script around June 29ish.

What steps will reproduce the problem?
1. Pull reports based on specific day
   ./appdailysales.py -v -D 06/26/2011 
   failed

2. Pull reports for the last few days
   ./appdailysales.py -v -d 6
   failed

3.

What is the expected output? What do you see instead?
Expected output is the gzipped sales report from itunes connect. Instead I 
don't see any new output files and receive the error "1 report(s) not available 
- try again later".

What version of the product are you using? On what operating system?
v2.7, on osX leopard, python v2.6.1

Please provide any additional information below.
I'm able to successfully access the sales reports via the itunes connect 
website with a web browser. Prior to June 29ish I was able to pull sales 
reports from itunes connect with appdailysales.py script v2.7


Note:
I figured out the issue by making the following change to v2.7 of 
appdailysales.py...

diff appdailysales.py_orig appdailysales.py 
420c420,425
<                 filename = 
urlHandle.info().getheader('content-disposition').split('=')[1]

---
>                 result = urlHandle.info().getheader('content-disposition', 
default=None)
>                 if result:
>                     filename = 
urlHandle.info().getheader('content-disposition').split('=')[1]
>                 else:
>                     filename = 'default.txt'
> 

Suggest you add that as a condition to know when Apple changes something again?

Look forward to hearing some news regarding this issue soon. Thanks!!

- Andrew

You might want to replace this line (line 287 of v 1.7):

filename = options.outputDirectory + filename

with this line:

filename = os.path.join(options.outputDirectory, filename)

That way, the script won't break if the user specified a directory without
a trailing slash.

This is a great script!  Thanks for the good work.

What steps will reproduce the problem?
I have two different iTunes logins for two different accounts.  When I try to 
use either login with this script I get the error 'Unable to find default 
vendor page'.  This script was working for me with both logins a month ago but 
I know iTunesConnect changed and I am just getting around to trying to update 
the script and fix it.

What is the expected output? What do you see instead?
I would expect it to work as my login credentials are correct and I can login 
to the site to see stats.

What version of the product are you using? On what operating system?
I experienced this with both version 2.5 and when using the latest r63 from the 
repository.

Since passing passwords via command line arguments is a considerable 
security risk (imagine two people logged into a mac, you switched to your 
user but the other user still has a ps-scraping utility running, he can 
easily steal your iTc password) I've modified the script to be able to 
read the password from stdin.

Hi,

Just tried using this script via command line and it errors out. Can anyone
tell me why? More importantly - can anyone help me get it working?

Running ubuntu server 8.04 with python 2.5.2

---

root@ev-linsrv:/var/www/itc# python -V
Python 2.5.2
root@ev-linsrv:/var/www/itc# uname -a
Linux ev-linsrv 2.6.24-19-server #1 SMP Wed Jun 18 15:18:00 UTC 2008 i686
GNU/Linux
root@ev-linsrv:/var/www/itc# python appdailysales.py 
Traceback (most recent call last):
  File "appdailysales.py", line 353, in <module>
    sys.exit(main())
  File "appdailysales.py", line 346, in main
    downloadFile(options)
  File "appdailysales.py", line 218, in downloadFile
    html = readHtml(opener, urlWebsite)
  File "appdailysales.py", line 200, in readHtml
    urlHandle = opener.open(request)
  File "/usr/lib/python2.5/urllib2.py", line 381, in open
    response = self._open(req, data)
  File "/usr/lib/python2.5/urllib2.py", line 399, in _open
    '_open', req)
  File "/usr/lib/python2.5/urllib2.py", line 360, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.5/urllib2.py", line 1115, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.5/urllib2.py", line 1082, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error (-2, 'Name or service not known')>

---

What steps will reproduce the problem?
1. Run the script

What is the expected output? What do you see instead?
Stops at "Unable to find default vendor page."

Would it be possible to get a version, or an option in this version that lets 
you download weekly sale reports.

I've attached a patch to make a small public API for this script. It's diff'ed 
with r52 (the latest svn, 
as of now).

The API looks like:

{{{
import appdailysales

scraper = appdailysales.Scraper('username', 'password')
file = scraper.getReport('01/04/2010')
file.unzip() # if you want
file.write() # with an optional argument directory='foo'
                   # and it returns the name of the file it wrote

# and one more, which (for the moment) requires BeautifulSoup
dates = scraper.getAvailableDates()
for date in dates:
   file = scraper.getReport(date)
   file.write()
}}}

The rest of the script is refactored to use this API.

Thanks for the great script! This was a major timesaver (and then I sunk 
several hours into 
refactoring this API :)

Are there any plans to support the new iAd reports, if not I might work on 
adding that functionality.

Hey,

I have set out to port this great script to PHP and there are a few problems 
I'm having so could you please help with some parts.

I started out with this script that logs into itunesconnect fine

<?php
$user = "[email protected]";
$pass = "hahanobodyknows";

$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, 
"https://itunesconnect.apple.com/WebObjects/iTunesConnect.woa"); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
$output = curl_exec($ch); 
curl_close($ch); 

$num = substr($output, strrpos($output,'action="')+8,54);

extract($_POST);
$url = 'https://itunesconnect.apple.com/WebObjects/iTunesConnect.woa/wo/'.$num;
$fields = array( 
            'theAccountName'=>urlencode($user),
            'theAccountPW'=>urlencode($pass),
            '1.Continue'=>urlencode('0'),
        );
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; } 
rtrim($fields_string,'&');
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string);
$result = curl_exec($ch);
curl_close($ch);

$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, "https://reportingitc.apple.com/"); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
$output = curl_exec($ch); 
curl_close($ch); 
?>

The problem is the last section where it gets html from 
"https://reportingitc.apple.com/" which strangely returns nothing

I have tried to follow the python version as close as possible and there may 
have been something missed out

Any ideas?
Thanks

What steps will reproduce the problem?
1. I run the script with my login info, unzipFile=false and verbose=False, this 
happens every time.

What is the expected output?

I expect it to download the most recent report.

What do you see instead?

Traceback (most recent call last):
  File "./fetch_sales_report.py", line 47, in main
    filename = appdailysales.downloadFile(options)
  File "/home/inaday/reports/appdailysales.py", line 290, in downloadFile
    raise ITCException, 'Unable to load the sales report web page at this time. A number of reasons can cause this including delayed reporting and unsigned contracts. Try again later or sign into iTunes Connect and verify access.'
ITCException: 'Unable to load the sales report web page at this time. A number 
of reasons can cause this including delayed reporting and unsigned contracts. 
Try again later or sign into iTunes Connect and verify access.'


What version of the product are you using? On what operating system?

appdailysales.py version 2.3 that I downloaded this morning. I'm running on 
Linux with a 2.6 kernel (Debian Lenny).

Please provide any additional information below.

I'm able to load the view in my browser just fine. I added a few lines to the 
script to output the html it was trying to parse looking for 
javax.faces.ViewState. I've attached that output. I know it looks like there 
are multiple html documents in the output and I have no idea why, but that is 
what the html variable on line 278 contained.

Thanks!

What steps will reproduce the problem?
1. running my modified script of your appdailysales.py never produces the 
download.   The [I have replaced the user and password for obvious reasons...]
[itunesjob]$ python app1.py -a [email protected] -p music -v
-- begin script --
Signing into iTunes Connect web site.
Accessing Sales and Trends reporting web site.
viewState:  j_id16060:j_id16061
defaultVendorPage:  j_id_jsp_645493535_2
ajaxName:  j_id_jsp_645493535_0
Accessing sales report web page.
viewState:  j_id16066:j_id16067
dailyName:  theForm:j_id_jsp_49459500_6
ajaxName:  theForm:j_id_jsp_4945950_2
dateName:  theForm:j_id_jsp_4945950_8
selectName: theForm:j_id_jsp_4945950_32
dateListAvailableDays:  ['05/24/2011', '05/23/2011', '05/22/2011', 
'05/21/2011', '05/20/2011', '05/19/2011', '05/18/2011', '05/17/2011', 
'05/16/2011', '05/15/2011', '05/14/2011', '05/13/2011', '05/12/2011', 
'05/11/2011']
dateListAvailableWeeks:  ['05/22/2011', '05/15/2011', '05/08/2011', 
'05/01/2011', '04/24/2011', '04/17/2011', '04/10/2011', '04/03/2011', 
'03/27/2011', '03/20/2011', '03/13/2011', '03/06/2011', '02/27/2011']
reportDates:  ['05/24/2011']
Downloading daily sales reports.
dateString-> 05/24/2011
dateString in the dateListAvailableDays!
Downloading report for:  05/24/2011
urlHandle-> Server: Apache-Coyote/1.1
X-Powered-By: JSF/1.2
Content-Type: text/html;charset=ISO-8859-1
Content-Language: en-US
Transfer-Encoding: chunked
Date: Wed, 25 May 2011 19:30:25 GMT

Server: Apache-Coyote/1.1
X-Powered-By: JSF/1.2
Content-Type: text/html;charset=ISO-8859-1
Content-Language: en-US
Transfer-Encoding: chunked
Date: Wed, 25 May 2011 19:30:25 GMT

05/24/2011 report is not available - try again later.
1 report(s) not available - try again later


What is the expected output? What do you see instead?
The expected output is the download should produce a file..


What version of the product are you using? On what operating system?
v 2.7 of appdailysales on Centos Linux

Please provide any additional information below.

What steps will reproduce the problem?
1. I ran "python appdailysales.py -f %Y-%m-%d-daily.txt.gz"
2.
3.

What is the expected output? What do you see instead?
I was hoping to see a txt file with a formatted name

What version of the product are you using? On what operating system?
I'm using 2.1 on Ubuntu 8.10 with Python version 2.5.2

Please provide any additional information below.

I created this issue ticket a few days ago but it seems to be missing now and I 
didn't receive any notice of changes in the issue, so I'm re-posting in hopes 
that it gets some attention from a Dev and hopefully gets fixed.

Basically the script was working fine up until a few days ago when Apple 
changed the way their reports system works. Now instead of the script producing 
a CSV report file, it spits out this error:

Traceback (most recent call last):
  File "appdailysales.py", line 353, in <module>
    sys.exit(main())
  File "appdailysales.py", line 346, in main
    downloadFile(options)
  File "appdailysales.py", line 247, in downloadFile
    urlDownload = urlBase % match[0]
IndexError: list index out of range

What steps will reproduce the problem?
1. run appdailysales with userId/password without arguments

What is the expected output? What do you see instead?
Reports are not being downloaded. Instead:

/Library/Python/2.6/site-packages/gdata/tlslite/utils/cryptomath.py:9:
DeprecationWarning: the sha module is deprecated; use the hashlib module
instead
  import sha
-- begin script --
reportDates:  ['11/20/2009']
11/20/2009 report is not available - try again later.
Traceback (most recent call last):
  File "/Users/ejo/SymbolicArmageddon/SalesData/SAReport", line 46, in <module>
    appdailysales.downloadFile(options)
  File "/Users/ejo/SymbolicArmageddon/SalesData/appdailysales.py", line
323, in downloadFile
    raise ITCException, '%i report(s) not available - try again later' %
unavailableCount
appdailysales.ITCException: '1 report(s) not available - try again later'


What version of the product are you using? On what operating system?
appdailysales 1.9, python 2.6.1, Mac OSX 10.6.mumble

Please provide any additional information below.

The error output is from my script which calls appdailysales, but the same
error results ( sans failed exception catching ) when appdailysales is run
standalone with the proper credentials.

This behavior began three days ago.  My software configuration has not
changed.  I'm guessing the problem is due to Apple changing the iTunes
Connect, but I haven't had the time ( or frankly the expertise ) to track
it down.

I've attached the script I'm using to drive it