Handle huge log files (was crashes on a large custom log file) about gource HOT 8 CLOSED

I will have a think about how to handle this, its seems like an edge case 
though. The
largest real-world log I currently test with is from the Linux Kernel, and 
that's
only about 40 megs.

Thanks for the report.

Just to let you know this is a log from a real-world clear case. I was able to 
get it
to display (albeit slowy) in codeswarm, since codeswarm supports a pre-sorted 
log file.

Hi. This should be fixed now hopefully.

If the log file given to Gource is bigger than 100 megs, it will fall back to 
just
seeking the file handle rather than loading the file into memory (which makes 
the
seekbar feature a bit clunky, but should at least work). 

http://gource.googlecode.com/files/gource-0.24-beta2.tar.gz

Dudley, would you be willing to share your script for creating the custom log 
file
format from cleartool lshistory output?  Thanks.

@djpotter77

Sure. I will need to dig it up, but I should be able to get it this week 
sometime.

# This is the basic script which actually converts to a codeswarm format, but 
you 

# can easily modify it to work for gource. I actually wrote a separate script to

# convert from code_swarm to gource.


#!/usr/local/bin/python

"""
   Take the output of the clear case history command
   ct lshistory -all -fmt "Element: %n| Date: %d| User:%u| Operation: %e| Object:
%[type]p| SimpleType: %m| OperationKind: %o\n"
   And turn it into something usable by code_swarm
"""

fileTypesWeCareAbout = [ 'compressed_file', 'compressed_text_file', 'file',
'html', 'text_file', 'xml']



import sys
import time

def processDate(date):
   d = date[:-6]
   d = time.strptime(d, "%Y-%m-%dT%H:%M:%S")
   return int(time.mktime(d))*1000

def processElement(e):
   return e.split("@@")[0] # just strip off the version info

def processLineIntoTuple(line):
   """Take the line and split it out into a dictionary"""
   d = {}
   for i in line.split("|"):
      l = i.split(":",1)
      if (len(l)) == 2:
         d[l[0].strip()] = l[1].strip()
      else:
         d[l[0].strip()] = ""
   return d

def UseThis(d):
   try:
      if d['OperationKind'] != "checkin":
         return 0
      if d['Object'] not in fileTypesWeCareAbout:
         return 0
   except:
      return 0

   return 1

def XMLize(d):
   print '<event date="%d" filename="%s" author="%s" />' % (d['Date'], d['Element'],
d['User'])


for x in sys.stdin.readlines():
   d = processLineIntoTuple(x)
   if UseThis(d):
      d['Date'] = processDate(d['Date'])
      d['Element'] = processElement(d['Element'])
      XMLize(d)

Hi

Getting this error when running the script:
    print <event date="%d" filename="%s" author="%s" /> % (d['Date'], d['Element'], d['User'])
                    ^
SyntaxError: invalid syntax

What could be wrong? 

thanks

It looks like you are missing the single quotes around the first part.

You have:
   print <event date="%d" filename="%s" author="%s" /> % (d['Date'], d['Element'], d['User'])

and it should be:
   print '<event date="%d" filename="%s" author="%s" />' % (d['Date'], d['Element'], d['User'])