Based on forum post here

"RecordProcessor: Warning - Information lost. Record skipped with duplicate formID: 00156F6A"

Generated by a duplicated record. Problem is there's no context information about which mod in the load list has the problem, and what was the problem in the first place.

Vorians provided an ESP to help reproduce the problem, which will end being uploaded as part of a test case. But for now I've put it on my Dropbox here since I cannot attach that type of file to GITHUB (it probably scans the ZIP and rejects the ESP inside since it doesn't know about it.)

Currently fully compiling CBash from sources on Windows takes ~15 mins (see -> https://ci.appveyor.com/project/leandor/cbash/history) which makes development a bit annoying.

It's hard to try anything that affect the build setup created by CMake without triggering a full recompilation afterwards.

That's why I'm about to start trying some options to see if I can speed up compilation a bit.

First thing I'm gonna try is to setup CMake to use pre compiled headers for Visuall Studio, by using one of these (cough_hacks_cough) solutions:

• https://github.com/larsch/cmake-precompiled-header
• http://stackoverflow.com/questions/148570/using-pre-compiled-headers-with-cmake
• https://github.com/sakra/cotire

Second thing, even if the above works, I intend to split up CBash project in a couple of static lib projects + one static lib project per game to hold all the records and modfile.

In the original SVN repository (cloned here, for now) there were some files on the release folder that ran some test suit on the compiled CBash.dll

• https://github.com/leandor/CBashSF/blob/master/release/ObTest.py
• https://github.com/leandor/CBashSF/blob/master/release/FNVTest.py

Maybe split those into atomic tests and run them all in sequence, at least for those ones that make sense to automate.

TODO

• Recover NewVegas tests from sourceforge (need to think proper directory structure for handling several games tests)
• Split big ObTest.py into atomic tests.
• Convert tests to unit-tests which rely on check premises and fail.
• Refactor tests to not depend on Oblivion.esm (generally, on any <game>.esm), by recreating a file that contains the proper records needed for each test.
• Alternatively, if present, detect game and copy master file to test directory as part of the startup code.

I'd like to bring forth a proposal - this is obviously a long-term goal but from short discussions with @Infernio I believe it's something achievable (since part of it already has been) and can be considered. Wrapping CBash in Rust, RBash.

TL;DR - Memory-safe with no loss of performance wrapper. Python 3 64bit only. Long term goal. Example. Built wheel.

CBash is written in C++ and afaik I believe none of the current devs have the skill in it needed to tackle working on CBash. The biggest draw of C/C++ is the perfomance, mainly via the low-level access, which let's you bypass some slower, safer abstractions to achieve it. It also benefits from being an old language, pretty much everything has been done with C++ and you can leverage that. And if there is something you need that hasn't, you can just use C, which was created around the time the primordial dust was gathering into what would one day become Earth.

Rust, on the other hand, is the hipster newcomer. Version 1 in 2015(?), with a new edition in 2018. A memory-safe systems programming language with influence from functional programming. No more segmentation faults or memory leaks. It seems to be more and more talked about as a credible alternative to C/C++ but it could just be confirmation bias on my end. Pretty easy to learn, too. Biggest drawback is its age - the existing material for it is in no way comparable to the others above. BUT, much like C++, we can just use C behind the scenes and hope it doesn't segfault or memory leak there :P In terms of performance I've seen benchmarks that placed it above C++ and below C, and I know that benches are not real life situations but the point is that performance shouldn't be an issue.

Rust also allows us to create a python extension module from code completely in rust. In the example repo I'm using pyo3, although there is another crate who can also create python extensions. This can be discussed later if this proposal is accepted.

As far as specific goals for the wrapper, I'd like to divide them into 3 stages. These are all long term goals that will be done slowly, over time.

• Stage 1 - Create a simple wrapper around CBash public API and expose it to python. Essentially a replacement for ctypes in cint. This should already be accomplished in this repo, but I'm sure I forgot something or there's some mistake, can't really test right now.

• Stage 2 - Consume cint. Slowly take over cint, start moving the python code there into rust. The templates/macros in rust should also help with reducing the monstruosity that is that module right now and provide a better API. At the end of this stage the public RBash API should be stabilized.

• Stage 3 - Consume CBash. This is a very long term goal, but if done slowly over time we could do it. Not much to say here, CBash is ridiculously huge, who knows what secrets it hides.

Over these stages, we'll never lose access to CBash and indeed we'll be using as RBash's own backend, so there's never any "big" breaking step. Keep in mind this is just a proposal and nothing is really set in stone.

EDIT: A handy flowchart to help with deciding :P

Hope this gets greenlit!

During my debug sessions these last days, I compiled vs the debug version of Boost and the STL, and got an assertion while trying to build the Bashed Patch that pointed to the current sorting callback not being correct.

I'm referring to this function:

bool sortMod(ModFile *lhs, ModFile *rhs)
    {
    //Esp's sort after esm's
    //Non-existent esms sort before existing esps
    //Non-existent esms retain their relative load order
    //Existing esps sort by modified date
    //Non-existent esps sort before existing esps
    //Non-existent esps retain their relative load order
    //New esps load last
    #ifndef LHS_BEFORE_RHS
        #define LHS_BEFORE_RHS true
        #define LHS_AFTER_RHS false
    #endif
    if(lhs->TES4.IsESM())
        {
        if(rhs->TES4.IsESM())
            {
            if(lhs->ModTime == 0)
                {
                if(rhs->ModTime == 0)
                    return LHS_BEFORE_RHS;
                return LHS_AFTER_RHS;
                }
            if(rhs->ModTime == 0)
                return LHS_BEFORE_RHS;
            return lhs->ModTime < rhs->ModTime;
            }
        return LHS_BEFORE_RHS;
        }
    if(rhs->TES4.IsESM())
        return LHS_AFTER_RHS;

    if(lhs->Flags.IsCreateNew)
        {
        if(rhs->Flags.IsCreateNew)
            return LHS_BEFORE_RHS;
        return LHS_AFTER_RHS;
        }
    if(rhs->Flags.IsCreateNew)
        return LHS_BEFORE_RHS;

    if(lhs->ModTime == 0)
        return LHS_BEFORE_RHS;
    if(rhs->ModTime == 0)
        return LHS_AFTER_RHS;
    return lhs->ModTime < rhs->ModTime;
    }

The idea seems to be, according to the rules that are spelled out on the header, to break the set into two partitions, ESMs|ESPs, and within those partitions each file sorts according to mod time, but taking into account that new files (whose mod time is 0, or that has the CreatedNew flag set) need to sort last.

So, I simplified it to be more clear and to remove the assertion. It became this:

bool sortMod(ModFile *lhs, ModFile *rhs)
    {
    //Esp's sort after esm's
    //Non-existent esms sort before existing esps
    //Non-existent esms retain their relative load order
    //Existing esps sort by modified date
    //Non-existent esps sort before existing esps
    //Non-existent esps retain their relative load order
    //New esps load last
    #ifndef LHS_BEFORE_RHS
        #define LHS_BEFORE_RHS true
        #define LHS_AFTER_RHS false
    #endif

    const time_t MIN_TIME = std::numeric_limits<time_t>::min();
    const time_t MAX_TIME = std::numeric_limits<time_t>::max();

    bool leftEsm = lhs->TES4.IsESM();
    bool rightEsm = rhs->TES4.IsESM();

    bool leftNew = (lhs->ModTime == 0) || lhs->Flags.IsCreateNew;
    bool rightNew = (rhs->ModTime == 0) || rhs->Flags.IsCreateNew;

    time_t leftTime = leftNew ? MAX_TIME : lhs->ModTime;
    time_t rightTime = rightNew ? MAX_TIME : rhs->ModTime;

    if (leftEsm != rightEsm)
        {
            return leftEsm ? LHS_BEFORE_RHS : LHS_AFTER_RHS;
        }
    else if (leftNew != rightNew)
        {
            return leftNew ? LHS_AFTER_RHS : LHS_BEFORE_RHS;
        }
    else 
        // They are in the same "partition", strict ordering applies within each
        {
            return leftTime < rightTime;
        }

    // should never reach here, but not important enough to assert IMO.
    return LHS_BEFORE_RHS;
}

It seems to be 'doing the right thing' ^™️, but I haven't tested extensively yet.
I've built a full Bashed Patch after the change, though, and the assertion was gone.

I'll probably gonna create the first test for checking this function and verify the ordering :)

I hope I'm not missing something obvious 😨

Here's the existing info:

• CBash-vs-PBash

Quote extracted from wrye-bash/wrye-bash#332:

what I mean is that those are elementary - we need a walkthrough, with the debugger - I admit I don't know what the patcher does

Probably on Load ? This could be the root of many hard to track bugs in Bash and must be fixed

Converted from the wiki into a handy issue that can be referenced in commits.

Records Missing Data

• CELL
  • Most things read in raw, missing a few sub-records
• LAND
  • ATEX
  • VTXT
  • DATA read in raw (not decoded in xEdit either)
• WRLD
  • Missing RNAM (CK leftover)
  • MHDT
  • WCTR
  • LTMP
  • XLCN
  • MODL
  • MODT
  • XWEM
  • MICO
  • IMPF?
  • NAMA read in raw

Unimplemented

• ACHR
• AMMO
• ARMA
• ARMO
• AVIF
• BOOK
• BPTD
• CAMS
• CLAS
• CLFM
• CLMT
• COBJ
• CONT
• CPTH
• CSTY
• DEBR
• DIAL
• DLBR
• DLVW
• DOBJ
• DOOR
• DUAL
• ECZN
• EFSH
• ENCH
• EXPL
• EYES
• FACT
• FLOR
• FLST
• FSTP
• FSTS
• FURN
• GLOB
• GMST
• GRAS
• HAZD
• HDPT
• IDLE
• IDLM
• IMAD
• IMGS
• INFO
• INGR
• IPCT
• IPDS
• KEYM
• LCRT
• LCTN
• LGTM
• LIGH
• LSCR
• MATO
• MESG
• MGEF
• MISC
• MOVT
• MSTT
• MUSC
• MUST
• NAVI
• NAVM
• NPC_
• PACK
• PERK
• PGRE
• PHZD
• PROJ
• QUST
• RACE
• REFR
• REGN
• RELA
• REVB
• RFCT
• SCEN
• SCRL
• SLGM
• SMBN
• SMEN
• SMQN
• SNCT
• SNDR
• SOPM
• SOUN
• SPEL
• SPGD
• STAT
• TACT
• TREE
• WATR
• WEAP
• WTHR

We talked about this on (#9), and since the I've been investigating a bit what options are out there, I thought I'd dump what I've found in an issue for future reference.

First, the links:

• The LZMA SDK
• And a github repo that has a clone of that (official sources are on Sourceforge)
• The "master" docs
• Some existing python projects that already provide the functionality:
  • https://github.com/fancycode/pylzma
  • https://github.com/harvimt/pylib7zip
• Related Stack Overflow questions
  • http://stackoverflow.com/questions/6610683/using-lzma-sdk-in-c
  • http://stackoverflow.com/questions/222030/how-do-i-create-7-zip-archives-with-net
• Helpful links (mainly as examples of how to use the 7z.DLL)
  • http://www.codeproject.com/Articles/27148/C-NET-Interface-for-Zip-Archive-DLLs
  • http://7zsharp.codeplex.com

For now I'll create this as a repository of information, until we decide what to do.

It's nothing major, I've already patched the needed changes on my local copy.

Basically this is what has changed on CBash's side:

• All published functions got prefixed with cb_
• Function LoadCollection (now cb_LoadCollection) has an extra parameter for receiving a progress callback that gets invoked inside the loop, and can abort the process if it returns False.

I'm gonna make a Pull Request on Wrye Bash for these changes when I'm sure that the new CBash version works and has no 'hidden' problems.

As mentioned here -> (#7).

Low priority for now, just don't want to forget.

• Taken from wrye-bash/wrye-bash#264

  Catch all issue to start investigating/splitting/fixing cint.py

• We are loading all fallout classes for naught - must be split into a package with a structure similar to the games one
• we must investigate if/how we can compile the dll for linux
• Taken from wrye-bash/wrye-bash#181
  • CBash patcher does not import names from csv files

Describe what are the current steps involved in compiling and testing the CBash binaries produced from this repository.

(see original discussion here too -> wrye-bash/wrye-bash#338)

The official lz4 is here: https://github.com/lz4/lz4

TODO

• Incorporate dependence on LZ4 distribution
• (1) Wrap simple functions from the library into python functions
• Add python tests for simple functions in (1)
• (2) Next step is to wrap archive support functions
• Add python tests for (2)
• (3) Implement high level interface supporting python idioms and primitives based on the simple wrappers on (1) & (2)
• Add tests for functionality on (3)
• Documentation

AppVeyor is already setup to monitor this repository's commits.

TODO

• Finalize configuration by added test execution and package artifacts to upload as a github release
• Make sure that after everything is setup correctly, only execute builds on dev and master branch (not sure on this yet)
• Document details of utilizing AppVeyor (and eventually travis-ci) for future members of the project
• Check how to make AppVeyor (and eventually travis-ci) dashboard and configuration editable by team members.

Some tracebacks / bugdumps:
https://gist.github.com/Infernio/ce154cfe92454a6204b180380a559c2d
https://gist.github.com/Infernio/816a95c2592bf038bfe48f32b4ed8286
https://www.reddit.com/r/oblivionmods/comments/bxj3d4/help_merging_legionoccupied_forts_ooo_fcom_patch/

CBash is one of two reasons WB is stuck on 32bit (the other is wrye-bash/wrye-bash#431), denying the entire program (but especially PBash) a ~30% speedup. We are pretty close - most methods seem to work - but a lovely traceback stands in our way:

Traceback (most recent call last):
  File "bash\balt.py", line 2500, in __Execute
    self.Execute()
  File "bash\balt.py", line 1604, in _conversation_wrapper
    return func(*args, **kwargs)
  File "bash\basher\mod_links.py", line 932, in Execute
    if not self._Execute(): return # prevent settings save
  File "bash\basher\mod_links.py", line 976, in _Execute
    doCBash=self.doCBash)
  File "bash\bosh\__init__.py", line 2025, in rescanMergeable
    return self._rescanMergeable(names, prog, doCBash, return_results)
  File "bash\bosh\__init__.py", line 2054, in _rescanMergeable
    canMerge = is_mergeable(fileInfo, self, reasons)
  File "bash\bosh\_mergeability.py", line 224, in isCBashMergeable
    if not _modIsMergeableLoad(modInfo, minfos, reasons) and not verbose:
  File "bash\bosh\_mergeability.py", line 212, in _modIsMergeableLoad
    reasons.append(_(u'Is a master of non-mergeable mod(s): %s.') % u', '.join(sorted(dependent)))
  File "bash\cint.py", line 15301, in __exit__
    self.Close()
  File "bash\cint.py", line 15317, in Close
    _CDeleteCollection(self._CollectionID)
WindowsError: exception: access violation reading 0xFFFFFFFFFFFFFFFF

Firing up Visual Studio and breakpointing cb_DeleteCollection shows that the access violation actually occurs in the Collection destructor:

Breakpointing inside the destructor shows this:

The first two ModFile instances (Oblivion.esm and Unofficial Oblivion Patch.esp, if you're curious) delete correctly, but the third one (Oblivion Citadel Door Fix.esp.ghost) crashes. As you can see from the screenshot, there's definitely something off about the third ModFile - it seems to be missing everything from TES4File.

Steps and tasks

Step 1

• Use pybind11 to generate a python compatible DLL that can replace cint.py
• Generate a python binding that compiles and links against CBash as static library
• Provide access to basic existing exported functions and export the required types as opaque
• Provide general test cases that allows to diagnose the API health from release to release.

Step 2

• Export the classes that are part of the existing API (ModFile, Collection, etc)
• Deprecate fake accessor functions (wrappers that just take an opaque type and simply calls a method on it)
• Refactor Bash code to convert from using the accessors to use the real methods

Step 3

• Extend the visible types (where reasonable) to publish helper C++ only types that are useful.
• Implement standard python constructions for where a class already exists that matches the concept (like Collection: provide iterators, indexers, etc) just extending what's already there, without refactoring the actual API.

Step 4

• Analyze and refactor the API to implement Python patterns where needed, where they apply.

Step 5

• Analyze the possibility of implementing additional constructions to help speed up the API.

Taken from #4. Landed in 0.7.0, but doesn't close that issue (duh).

In the original SVN repository (cloned here, for now) there were some files on the release folder that ran some test suit on the compiled CBash.dll

• https://github.com/leandor/CBashSF/blob/master/release/ObTest.py
• https://github.com/leandor/CBashSF/blob/master/release/FNVTest.py

This issue is about reviving ObTest.py and at least including it in our CBash repo.

I HATE CMake :P

But is simply the only option available for providing quick cross-platform compilation, without requiring POSIX tools on Windows.

Problem that I faced trying to use CMake on Windows with VC++, is that there seems to be some dumbness in the handling of dependency discovery (specifically Boost) and I spent several hours trying to make my build generate and compile with CMake, while if I was using VC++ directly it would have been a matter of minutes.

Another thing that bothers me, is compilation speed. CMake doesn't support pre-compiled headers, and for any project that uses templates (boost or STL are prime examples) they are almost a requirement, since they cut compilation times by an order of magnitude.

Full compiling of CBash take ~10 mins on my PC, while most of the time lost is spent recompiling the same set of headers that are used in most source files.

I found some solutions for this around (this or this and some here) but I have yet to put them to test.

I'd like to provide a batch script that better handles dependency setup, like, for example, download/clone dependencies from GITHUB (all of them are here) and execute their builds and afterwards, setup our own dependency variables with the correct paths to them.

It's very low priority for now, but I lost so much time to this, that I would hate that the same happens to anyone else.

((Also, and this is personal opinion, another thing I hate is the CMake language/format itself is SO clunky and amateur... it seems more a patchwork of ideas pieced together than a formal language.))

Pycharm emits no less than 120 warnings outside cint on _RecordID, _Type (Type ?), and _ModID attributes access. Although as we discussed PEP8'ing the API is not an immediate priority (although should be done in the final API), those in particular should be squashed in the next API iteration (and _Type renamed to something more descriptive).

In general I have painfully and meticulously squashed 1000s of warnings and those were of the most annoying kind, as I could not/would not mess with cint. I will be very glad to see those squashed.

It seems to have been auto-generated at some point.

I'll remove everything not related to this repositories' content and re-add by hand the patterns needed as they come up.

Error:

Code

NonNullStringRecord::~NonNullStringRecord()
    {
    if(DiskSize == 0)
        delete []_value;  // <<<< HERE
    }

Probably caused by double delete of that pointer.

See: http://forums.bethsoft.com/topic/1615914-wrye-bash-thread-114/page-6#entry25325056

It is probably a memory issue or something else but still at least a nice error message...

Bugdump from post above:

Wrye Bash starting
Using Wrye Bash Version 307.201612302300 (Standalone)
OS info: Windows-7-6.1.7601-SP1
Python version: 2.7.12
wxPython version: 2.8.12.1 (msw-unicode)
input encoding: None; output encoding: None; locale: ('sv_SE', 'cp1252')
filesystem encoding: mbcs
Searching for game to manage:
bush.pyo   71 _supportedGames: Detected the following supported games via Windows Registry:
bush.pyo   73 _supportedGames:  oblivion: D:\Games\Bethesda Softworks\Oblivion
bush.pyo  131 _detectGames: Detecting games via the -o argument, bash.ini and relative path:
bush.pyo  137 _detectGames: Set game mode to oblivion found in parent directory of Mopy:  D:\Games\Bethesda Softworks\Oblivion
bush.pyo  172 setGame: No preferred game specified.
bush.pyo  152 __setGame:  Using oblivion game: D:\Games\Bethesda Softworks\Oblivion
testing UAC
mods_metadata.pyo  224 __init__: Using LOOT API version: 0.10.2
Traceback (most recent call last):
  File "bash\balt.pyo", line 417, in <lambda>
  File "bash\balt.pyo", line 1563, in _conversation_wrapper
  File "bash\basher\patcher_dialog.pyo", line 199, in PatchExecute
  File "bash\basher\patcher_dialog.pyo", line 299, in _save_cbash
  File "bash\cint.pyo", line 14269, in save
WindowsError: exception: access violation reading 0x0BFEB40F 

Repro in the post above - initial instructions in http://forums.bethsoft.com/topic/1615914-wrye-bash-thread-114/page-6#entry25323182 but slimmed down in bugdump post

I'd want to make space for introducing separate projects for test cases, and also, break up main CBash sources into static lib packages, with the least possible dependencies with one another.

Also, in the future, I'd want to create a new project for the boost.python generated binding, to replace the current cint API.

My initial idea would be to create a static library for each game which will contain all the record definitions (initially... at least.)

End goal is to speed up a bit the full recompilation of the main CBash project (since the bulk of time recompiling is lost on the individual record files), and also, will help in modularize the source code, as we can use those projects in isolation of one another, and detect unwanted dependencies.

Major goal of 0.7.0, that thing was a bane because it didn't actually specify what type of record caused the problem, which mod it originated from, etc.