simonbowly / grblogtools Goto Github PK

See failing regression tests on master: missing Log, LogFilePath, ModelFile, Seed in the timelines dataframes. How much of this information should be included?

To finalise this:

• remove grblogtools.py
• remove the regression test
• maybe make a few of the end-to-end tests more 'exacting' by storing the expected summary data?

Replicate behaviour of merged_logs=True. This is a pretty small change to the main loop, just needs to keep a new SingleLogParser in reserver to check for the start of a new log.

We have many sections like

for pattern in PresolveParser.presolve_intermediate_patterns:
    match = pattern.match(line)
    if match:
        self._summary.update(typeconvert_groupdict(match))
        return True

which could probably be turned into a utility function, e.g.

def run_regexes(patterns, line, target):
    for pattern in patterns:
        match = pattern.match(line)
        if match:
            target.update(typeconvert_groupdict(match))
            return True
    return False

Should this be done by header, presolve, or in post-processing?

The regexes would be better as globals in their respective modules, putting them under classes makes them ambiguous to address (lead with class name or self?).

• is the boolean return of continue_parsing cumbersome compared to its benefits?
• in SingleLogParser, it might help for this to return if whether any match was found
• checking overhead could be reduced with an internal _stop flag in a parser that wants to stop parsing future data

glass4 relaxation is too easy and we have no pure LPs, so there is currently no simplex/barrier log data in the repo for testing. Would like to add this to the regression tests before changing this part of the parser.

Let's do this in a meeting to avoid future merge conflicts.

• Consistent naming for the parser modules
• Consistent naming for test files
• Move parsers and their tests to submodules?

This should cover:

• primal/dual simplex
• barrier
• concurrent

First thought is that this should be a composition of other parsers for continuous models, e.g.

class RelaxationParser:
    def __init__(self):
        self.barrier_parser = BarrierParser()
        self.simplex_parser = SimplexParser()

    def start_parsing(self):
        # custom matchers for concurrent indicator lines / root node info
        # defer to barrier or simplex parsers for who should take over from then

    def continue_parsing(self):
        # defer to active barrier or simplex parser
        # watch out for termination lines like "barrier solved model" from concurrent solves

Naming?

• RelaxationParser
• ContinuousLogParser (i.e. more general)

To discuss - should we now merge continue_parsing and start_parsing to a single parse() method for most classes?

The return value meaning for both are now the same (True = the line matched something).

Classes could keep internal state so that they know whether they have been "started", but this is not strictly necessary, it would just reduce the number of patterns to check initially. It's perhaps better for performance oriented things like this to be handled internally.

This could be much cleaner now, without column regexes, since we store parameter values in a separate dataframe.

• conform to same start_parsing/continue_parsing pattern
• have lower level parsers as attributes
• SingleLogParser parses one log and stops at termination
• MultipleLogParser creates SingleLogParsers as it goes to parse a series of logs in one file?

From discussion in #22

Simon:

Integrating the termination parser with the different termination messages from barrier and nodelogs is looking tricky. Switching the termination parser settings on the fly once we determine the model type doesn't seem to fit very well. What do you think of the following?

1. Include "Optimal objective" in the barrier parser
2. Include "Best objective/best bound/solution count" as well as "Optimal solution found" in the nodelog parser, since they are specific to models with a node log
3. Instead of the termination parser, have a status message parser which picks up "Time limit reached", "Interrupted" and so on (without interrupting other parsers).
4. In the case of a MIP where barrier solves the relaxation but the node processing times out, when merging all the log info, Status="TIME_LIMIT" from the status parser would then taken precedence over Status="OPTIMAL" coming from the barrier parser, and we would get the right result.

Maliheh:

One other idea to address the termination stage is to treat this parser differently because the termination messages might be scattered throughout the log.

The idea would be that we do not include the termination parser in the list of future parsers. In the method continue_parser() of the SingleLogParser(), if the line is not already matched with a pattern in the current parser and if the line does not match the start pattern of any other parser, we pass the line to the termination parser (we just need to have one method named parse for this parser) and save the information in a dict (the same as we do in the other parsers).

This will work because the lines that appear later in the log file will always overwrite whatever optimal objective value and status we have already saved in the termination parser.

ParseResult code could be tidied up a bit - a pandas join could be used to add common data to the progress log.

• Add contributing.md to specify style
• Add pre-commit.yml & CI test action to enforce styles on commits