morketh / icemelt Goto Github PK

./IceMelt.py:155: Warning: Data truncated for column 'guild' at row 1 
  cursor.execute(SQL,sqldata)
./IceMelt.py:155: Warning: Data truncated for column 'realm' at row 1
cursor.execute(SQL,sqldata)

not exactly sure what this means just yet

so it seems something i making it miss the LAST entry in the guilds roster.

6212ea6
This commit has some notes in it. Rebuilding IceMelt.py needs to happen now that the DB structure has been changed a lot a simple bash script for data collection is not going to be comprehensive enough for the data entry for guild additions to work in a smaller database these few scripts should be reworked into python.

Loop Index: 3464
Loop Total: 2059686

After after loops through the script crashes.

so apparently looking at a guild roster Ive been missing exactly 1 toon per roster. should be a simple fix but you never know lol the rest of the project has been not exactly flawless

Apparently Ive triggered an IDS/IPS response from the server i would need to wait for X amount of time and then resend data and continue until the error happens again

well so far most of the errors are fixed. however now it seems there is multiple records that are being inserted into the guilds table anywhere between 2 and 40 guilds are being inserted per realm
as we know each realm group can only have 1 guild
rexxar/misha is a realm group

i believe the URL needs a new field separator in order to look up guilds from different Regions

At currant point the scripts all use INSERT statements how ever this means we need to clean up the database after words due to duplicate entries we should be using some form of UPDATE WHERE clause in order to update the data in stead of pushing new data into the DB

"index" "guild" "region_id" "realm" "status" "fid"
"2" "Hallowed Moon" "1" "garrosh" "200" "3"
um last time i checked there were only TWO factions
the code in question should only be setting a 1 or a 2
1 = Alliance
2 = Horde
so whats with the 3........?

Traceback (most recent call last):                                                                                                      ]   0% ETA:  --:--:--
  File "./IceMelt.py", line 152, in <module>
    cursor.execute(SQL,sqldata)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '), 'aegwynn')' at line 1")

so this needs to be fixed before IceMelt will work

ginsert: line 34: syntax error near unexpected token `('

After getting several hundred lines of code into the project and building the database from scratch i figured i might just be able to tweak my program to use the MaNGOS database for data collection seeing as this is the type of data we are going to be scrapping from blizzards webpage

I've been doing a little bit of preliminary research pertaining to the incomplete data scrape and I've come to a conclusion:

There is only one method to attain 100% result set. World list generation.
Theoretically using a program like crunch for example we can build an exhaustive list of names based on KNOWN name factors for toon names min value = 3 Max value = 12 all alpha based names (I.E. no spaces numbers or special chars)
RESULT Set:

1286229649468814960 bytes
1226644181698 MB
1197894708 GB
1169819 TB
1142 PB
number of lines: 99246114928148760

with today's storage capacity this is out of the question.
BUT if that list was generated on the fly (basic Time-Memory trade off) we could grab each item in the list as it was generated we could ask for a result from each realm PER item if a result was a 404 we wouldn't care about it. If it was a 200 we store the result in the Char table with the basic URL data so we can then pull results for population statistics.

Total Guilds not Processed: 3494 out of 4159
Traceback (most recent call last):                                                                                                      ]   0% ETA:  --:--:--
  File "./Cold-air-funnel.py", line 111, in <module>
    bar.update(index)
  File "/usr/lib/python2.6/site-packages/progressbar-2.3-py2.6.egg/progressbar/__init__.py", line 271, in update
    raise ValueError('Value out of range')
ValueError: Value out of range

The results in the guild table is higher then the Max Count of the progress bar

ERROR 1045 (28000): Access denied for user 'icemelt'@'localhost' (using password: YES)