MSDN_Crawler: many tilib.exe errors about flare-ida HOT 17 CLOSED

I am facing the same issues of multiple tilib.exe errors

from flare-ida.

I use about 3.* version BeautifulSoup, then debug the msdn_crawler.py , fix some error, etc:

constant_names = re.findall(
    "<dl><dt>(.*?)</dt>", descriptions[i])
 if not constant_names:
    continue
constant_names = [strip_html(unicode(c, 'utf-8'))
      .encode('utf-8') for c in constant_names]
parsed_html = BeautifulSoup(descriptions[i])
 constant_descriptions = []
for string in parsed_html.findAll(width='60%'):
     constant_descriptions.append(strip_html(string.text.encode('ascii')).encode('utf-8'))

the result is :

Parsed 341278 files
Extracted information about 34218 functions
ERROR processing 197 files

the size of msdn_data_nn.xml is 33.7Mb, what about others?

I'll upload this file in my blog site, so people can download this, now i'm focusing on the rest error file..

from flare-ida.

It's still not work,although i change the py file

from flare-ida.

@flypuma can u paste the error infos in ur post?

from flare-ida.

@niklaus520

Traceback (most recent call last):
File "C:\flare-ida-master\MSDN_crawler\msdn_crawler.py", line 414, in

main()

File "C:\flare-ida-master\MSDN_crawler\msdn_crawler.py", line 399, in main
(file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
File "C:\flare-ida-master\MSDN_crawler\msdn_crawler.py", line 372, in parse_fi
les
result = parse_file(os.path.join(root, file), const_enum)
File "C:\flare-ida-master\MSDN_crawler\msdn_crawler.py", line 277, in parse_fi
le
return parse_new_style(file, content, const_enum)
File "C:\flare-ida-master\MSDN_crawler\msdn_crawler.py", line 185, in parse_ne
w_style
constant_descriptions.append(strip_html(string.text.encode('ascii')).encode(
'utf-8'))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 64:
ordinal not in range(128)

from flare-ida.

@flypuma which line have u changed? How many files caught error while processing?

from flare-ida.

@niklaus520 just line you pasted, it begins from line 176. If I do not chang these lines, It has the issue just like #2. I didn't show the number

from flare-ida.

@niklaus520 I run the original msdn-crawler.py and txtracted about 33984 functions, just colse to yours. How could i get the file in you blog site.

from flare-ida.

@flypuma http://blog.depressedmarvin.com/upload/2015/02/09/msdn_data_nn.xml

you can just wget it

from flare-ida.

@niklaus520 Thanks a lot. Could you upload the file msdn_crawler.py?

from flare-ida.

http://blog.depressedmarvin.com/upload/2015/02/10/msdn_crawler.py

well, now u can try my script, see if there are still errors.

Then u can compare them, maybe some lines are different~

from flare-ida.

i got the issue when run python script annotate_IDB_MSDN, please help me

Traceback (most recent call last):
File "C:/Program Files/IDAPro6.6/python/flare/annotate_IDB_MSDN.py", line 117, in on_ok_button
IDB_MSDN_Annotator.main(config)
File "C:/Program Files/IDAPro6.6/python/flare\IDB_MSDN_Annotator__init__.py", line 523, in main
functions_map = parse_xml_data_files(msdn_data_dir)
File "C:/Program Files/IDAPro6.6/python/flare\IDB_MSDN_Annotator__init__.py", line 486, in parse_xml_data_files
additional_functions = xml_parser.parse(xml_file)
File "C:/Program Files/IDAPro6.6/python/flare\IDB_MSDN_Annotator\xml_parser.py", line 283, in parse
parser.parse(xmlfile)
File "C:\Program Files\IDAPro6.6\lib\xml\sax\expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "C:\Program Files\IDAPro6.6\lib\xml\sax\xmlreader.py", line 123, in parse
self.feed(buffer)
File "C:\Program Files\IDAPro6.6\lib\xml\sax\expatreader.py", line 211, in feed
self._err_handler.fatalError(exc)
File "C:\Program Files\IDAPro6.6\lib\xml\sax\handler.py", line 38, in fatalError
raise exception
xml.sax._exceptions.SAXParseException: C:\Program Files\IDAPro6.6\python\flare\annotate_IDB_MSDN.py:1:2: not well-formed (invalid token)

Thank you very much!

from flare-ida.

to niklaus520
if you change this instruction you can eliminate all error related to unicode:

        for string in parsed_html.findAll(width='60%'):
            try:
                constant_descriptions.append(strip_html(string.text.encode('ascii')).encode('utf-8'))
            except Exception,e:
                constant_descriptions.append(strip_html(string.text.encode('utf-8')))

please upload to your code so all can download it.

Ivan

from flare-ida.

@I-VANN Here is the code: http://blog.depressedmarvin.com/upload/2015/02/10/msdn_crawler.py && here is the data file: http://blog.depressedmarvin.com/upload/2015/02/09/msdn_data_nn.xml

from flare-ida.

I've just modified the file with my suggestion, it was only for other people.
So if you think that this change to your modified file is acceptable you can modify for all.
Thank you for your availability.

from flare-ida.

@I-VANN cool, thanks for your suggestion

from flare-ida.

Closing this old issue. Please check if the following file works for you after unzipping it.
https://github.com/mr-tz/flare-ida/blob/master/MSDN_data/msdn_data.zip
Please reopen this issue if you need further assistance.

from flare-ida.