Hello everyone, I want to develop a tool to reverse *.tex texture about silent hill2 (pc version).
I have campared texture.hpp from sh3redux and tex file in hex value, but I confused, it seems different.
Could you give me some advise?

On g++6.2.0 I can't build due to this line.

It would seem somewhere, ERROR is already defined (I assume by the compiler?). Changing this to ERR fixes it.

We should be able to get input at least from keyboard and mouse.
I think a stateful approach, basically maintaining a list of all pressed keys and buttons, is the best for our use case.

I'm working on a system based on SDL that also uses timestamps that can track for how long between frames a key was pressed; the time is normalized in the range [0, 1], so that it can just be multiplied for movement for instance.
It also counts how many times a key was pressed, so if something is toggled, for instance the flashlight, one can use count % 2 == 1 to decide whether to turn it on.
I'm hoping this can improve the response in the presence of lags.

I'm ripping this code from another project of mine. Here's some sample that shows the API's usage.
The code only checks for State::Pressed, but there's also the Released, JustPressed and JustReleased states. The Just* states refer to the key having been started to be pressed/released for the current update and can be used for firing a weapon, for instance.

I can not find anything useful.

This is a very nice project, well done guys!
Q1: are you aware of any similar project working on Silent Hill 1/2 ? (I would die for...)
Q2: did you evaluate creating an engine for ResidualVM instead of a stand-alone project ? It is a nice multi-platform portable application (with a solid user base), so I can figure out it as a safe place to raise a new 3d game engine into.

Hi everyone,
I've been having problems with building the game engine. I tried following the steps on the wiki and the README but couldn't find the proper files like build.bat. I was able to get everything but sdl2 to work in cmake by adding the locations of the config.cmake for every library,but for some reason I can't get this approach to work for sdl2. I am on a windows machine running Windows 10 and had to install glew,zlib,boost,glm,and freetype2 to get this far. Is there an easier way to build this that I am missing or is something else wrong?

There seems to be a well defined inline-documentation format via code comments for files and functions.
I'm not familiar with it; is there a tool ala Doxygen to extract it?

I'd also like to question the "revision history" part of the documentation. It's easy to get out of date - instead one can use git log $FILE (maybe with --oneline) to get one and then one can also easily check the specific code changes made via git show.

We get some annoying warnings from -Weffc++ about initializing members via the member initialization list, even though their default constructor is perfectly fine. See GCC Bug 81431 for an upstream bug report, the referenced GCC Bug 16166 is also relevant.
We should try to replace this warning with other flags to report things we do want to hear about. At worst we could just get rid of the flag and live with less warnings.

For some reason, this section of code

` for(sh3_arc_section& candidate : c_sections)
{

    auto files = candidate.fileList.equal_range(filename);
    if(files.first == end(candidate.fileList))
    {
        continue; // No filename found in this section, continue over
    }
    else // We've found it
    {
        section = &candidate;
        // files.first is the std::map iterator
        // files.first->second is the value of the entry the iterator is pointing at
        index = files.first->second;
        if(next(files.first) != files.second)
        {
            Log(LogLevel::WARN, "Multiple files with name %s exist.", filename.c_str());
        }
        break;
    }
}`

on line 97 in 'types.cpp' incorrectly detects what section the file is in. I've verified that the names and indexes are being mapped correctly (at least I think so), but for some reason this loop detects that a LOT of files are in the tex section when they are definitely not. It also seems it grabs the same index when it does this.

As pointed out in #50, calling exit() from main() does not run destructors.
This could result in unexpected behaviour. Same with die() and assert() if you think about it; any objects on the stack will not have their destructor called.

Files are flushed and closed by the runtime before exiting, so nothing will get lost there. If any logging is done in the destructors of objects, that will be missing though.
Other visible effects would involve IPC that is not done through files? Plus again any IPC done in destructors.

This is probably not as significant for die() and assert(), which are for abnormal termination anyways. Still, there are alternatives: we could never use these and instead use error-codes everywhere. Seems a bit excessive for exceptional situations IMO.
We could throw exceptions, which unwind the stack and call destructors.

For main(), if we want to have the destructors of the local variables called there are a bunch of things we can do.
We could just return static_cast<int>(error_code::*);, which is a little longer, but we won't have to write it often anyways.
We could let main() just forward to another sh_main() function, main() not having any objects on the stack at all.
We could also wrap everything except the exit() call in another scope, like so:

int main()
{
    error_code code = error_code::SUCCESS;
    {
        //...
    }
    exit(code);
}

Which is more or less the same as the previous suggestion.

I think it makes sense to just static_cast.
For die() we should perhaps stick a @warn in the documentation that destructors will not be run on objects on the stack. We can also replace assert() with a custom variant, which also tries to log the assertion, and perhaps supports stuff like continuing anyways, for purpose of debugging.

I introduced a couple of compile-time and run-time range checks that make sure that numeric casts work as expected.
While it's certainly useful to have these for debug builds, they are only really needed to verify input, i.e. making sure data-offsets specified in file headers are in range. And there they shouldn't just be ASSERTs IMO, but regular checks done in any build.
For debug-builds we could also rely on UBSan (clang can check for unsigned over flow as well) instead of checking ourselves, which would make the code more readable. We should a build-option for that.

This is unrelated, but generally options to enable the different sanitizers might be nice. Sure, you can do that via CMAKE_{CXX,{EXE,SHARED,STATIC}_LINKER}_FLAGS, but I think it'd be easier to just have a SANITIZERS variable or something. Then we can also specifically enable unsigned-integer-overflow and also the nullability-* options for UBSan.

Going to open this here so everyone can see. So far, the file formats that we have reversed are as follows:

.anm (partial) [ANiMation]
.tex [TEXture]
.tbn2 [Exported from KONAMI's SOFTIMAGE/PhotoShop .pic plugin]
.mdl [MoDeL]
.map [MAP]
.cld (partial) [CoLlision Data]

The file formats that we currently do not have reversed (or any idea how they are used) are:

.ded Dynamic Effects Data (ded). Probably crap to do with material files.
.cam (how does the camera interpret these values??) [CAMera data]
.kg1 Character and weapon shadow maps
.kg2 Level Shadow Maps
.fcl [???]

So far, it seems that (to some extent) the trigger/interaction data is stored in the .cld, however, what is unclear to me is where the motion capture data for cutscenes is. For SILENT HILL 2, this is in the .anm files, so I would assume it is the same here (they are stored in the /demo/ folder in SH2), however the absolutely horrendous directory structure and file naming makes it near impossible to deduce without having animations working properly in-engine.

If anybody has any idea, leave you comments/suggestions in this issue.

I've switched over to Lubuntu (I gave up on Arch literally needing 24/7 care to work..) and I still have problems with cmake, though they aren't as bad as Arch.

1. cmake still complains about glm not being found, however the error now gives is file or directory //include referenced by variable GLM_INCLUDE_DIRS does not exist !, which can be fixed by directly editing and removing a / from the /../../ string in usr/lib/cmake/glm/glmConfig.cmake

2. I'm not sure what distro you're using @z33ky (you've mentioned it before but I've forgotten), but for me ld will fail trying to link -lOpenGL and -lZLIB, on Lubuntu they should be -lGL and -lz, which is similar to Windows (!?)

After all of these changes, I get an executable that can be run.

I'm going to start using cmake properly when I start working on this again (Code::Blocks on Windows though :P ), so I think these should be fixed up (plus I can learn a bit more about cmake)

I think I have all dependencies satisfied.
Here is the output:

$ cmake .
-- The CXX compiler identification is GNU 7.3.1
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test HAVE_fcolor-diagnostics_FLAG
-- Performing Test HAVE_fcolor-diagnostics_FLAG - Failed
-- Performing Test HAVE_fdiagnostics-color=always_FLAG
-- Performing Test HAVE_fdiagnostics-color=always_FLAG - Success
-- Forcing colored compiler output
-- Performing Test HAVE_Weverything_FLAG
-- Performing Test HAVE_Weverything_FLAG - Failed
-- Performing Test HAVE_Wcxx98-compat_FLAG
-- Performing Test HAVE_Wcxx98-compat_FLAG - Failed
-- Performing Test HAVE_Wcxx98-compat-pedantic_FLAG
-- Performing Test HAVE_Wcxx98-compat-pedantic_FLAG - Failed
-- Performing Test HAVE_Wexit-time-destructors_FLAG
-- Performing Test HAVE_Wexit-time-destructors_FLAG - Failed
-- Performing Test HAVE_Wignored-qualifiers_FLAG
-- Performing Test HAVE_Wignored-qualifiers_FLAG - Success
-- Disabling -Wignored-qualifiers
-- Performing Test HAVE_Wpadded_FLAG
-- Performing Test HAVE_Wpadded_FLAG - Success
-- Disabling -Wpadded
-- Boost version: 1.66.0
-- Found GLEW: /usr/include
-- Found OpenGL: /usr/lib/libOpenGL.so
-- Found ZLIB: /usr/lib/libz.so (found version "1.2.11")
CMake Error at source/CMakeLists.txt:16 (include_directories):
include_directories given empty-string as include directory.

-- Boost version: 1.66.0
CMake Error at tests/CMakeLists.txt:16 (include_directories):
include_directories given empty-string as include directory.

-- Found Doxygen: /usr/bin/doxygen (found version "1.8.14") found components: doxygen dot
-- Configuring incomplete, errors occurred!
See also "/home/broken/sh3redux/CMakeFiles/CMakeOutput.log".
See also "/home/broken/sh3redux/CMakeFiles/CMakeError.log".

CMakeOutput.log
CMakeError.log

For people interested in contributing we should have a list of TODO items.

We should have some more abstract, future goals so we know what we're working towards, as well as more concrete, easier to reach ones so people know what they can work on now.

We should open issues and assign ourself for things that we are currently working on and leave a note in the TODO list to check the issues first, so that there's no wasted effort. A label ("WIP"?) would be useful to distinguish these tickets from other issues.

The documentation mentions that sub arcs (subarc and sub .arc as well) are synonymous with arc sections. There should be a single term used consistently.
I am slightly confused, but it seems that for the MFT, "section" refers to the stored information about the subarcs and the term "subarcs" refers to the actual files. In any case, I think this should be cleared up.

Consider using fmt for Log().

Considering the issue that has been raised in #136, I propose that we control what version of the libraries we have by either including them as submodules, or the source code themselves, similar to which can be found in this project: https://github.com/xoreos/xoreos.

We could then have targeted build scripts (similar to build.bat or build.sh) that would build the required libraries and put them in a lib folder, which we then link against. This should help homogenise building and prevent per platform errors that Windows users (which I assume most normal developers are running day to day) seem to be having. We won't have to rely on strange cmake hacks either.

The files fontdata_j.bin and fontdata_k.bin seem to be extracted incorrectly, as mentioned in #84:

You have something wrong with arc file management. It looks like it adds extra bytes, even size of file is bigger.

Side-note: Sorry I've been quiet lately. My bachelor's thesis is due in a month and the days suddenly seem to pass quicker than usual...

Currently we use a merge-heavy workflow, which of course results in plenty merge commits. Many of those are fast forwardable, i.e. "unnecessary" merges.
Having merges even when fast forwardable commits are integrated preserves the branch structure. On the other hand, it litters the log with merge commits.
So to get to the point, I am wondering whether we want to continue with a merge-heavy workflow, or if we want to use fast forwarding (i.e. avoiding merge commits) when possible. Note that in pull requests you have the option of using rebase instead of merge to integrate the changes, which avoids the merge commit.
I can also --force push a modified, linear history for the existing history, which would require everyone else to do a git reset --hard on their master branches (assuming no additional commits were made in private; a git pull --rebase might throw a tantrum about the changed history) though - but only as a one-time thing.

To load resources like textures and models from the ARC files, a templated resource class was proposed:

template<typename Header> // where Header is sh3_texture_header for example
class resource
{
    ...

    Header& header() { return reinterpret_cast<Header&>(raw.front()); }

    using iterator_range = std::pair<raw::iterator, raw::iterator>; // we could use boost::iterator_range instead
    iterator_range data() { return std::make_pair(begin(raw) + header().data_offset(), end(raw)); }

    ...

    std::vector<uint8_t> raw;
};

A texture would then be a resource<sh3_texture_header>, which provides us access to the header by simply aliasing the bytes in resource::raw , as well as convenient access to the "body" (non-header bytes). Reading in the vector is then enough to load a resource from a file.

Some resource types, e.g. models, may have multiple sub-headers. This is what the data_offset() function of the header is for.
For headers without sub-headers, the implementation is just return sizeof(*this);, while for those with sub-headers, the implementation is return sizeof(*this) + num_sub_headers * sizeof(SubHeader); to account for them.
The sub-headers in models refer to embedded textures. These resources can also be handled with an appropriate data_offset() function.

This one's probably just more of a personal preference, but I'd like to garner some opinion of others. I've done a bit of experimenting with premake5, which is similar to cmake, however a lot less complex. The scripts are basic .lua scripts, that look similar to the one attached. One neat feature is the ease of specifying build targets, as well as allowing recursive file finding (e.g files {"source/sh3/**.cpp", "source/sh3/**.hpp"}) instead of having to specify files we want to compile. Though simpler, however, there IS a lot of functionality that it does not contain that cmake does, such as initial compiler scanning etc.

An example of a script can be found here: https://github.com/Palm-Studios/OpenHill/blob/master/premake5.lua

For some reason, unless certain dlls are copied over to the application directory (libgcc_s_sjlj.dll, libstdc++6.dll and libwinpthread-1.dll), the i686 version of sh3rd.exe will fail to launch, with the error:

The application is unable to start: 0xc000007b

From what I understand, this is because the application is attempting to load a 64-bit binary into the 32-bit executable. Is there a way to rectify this at compile time (I'm using mingw-w64 6.2.0 on Windows 7)

And if so, what's the status, and how would one go about getting it to read the SH3 game files?

I was wondering if Linux support is planned.

Thanks!

Hello, i looked at the old blog and saw that you figured out how to unswizzle the 8bpp textures and apply the correct palette data, i am looking at the texture.cpp code and saw a sort of breakdown, but it's kind of messy. I'm working on my own sh3 texture tool and was wondering if you could explain and breakdown the textures so i can implement it in my code?

glfw looks a little bit friendlier and less bloated imho, I also like the idea of having callbacks instead of having to dispatch events ala SDL. Apart from this and a few other things, they're practically the same.

It apparently also deals with OpenGL in a slightly more cross platform way than SDL.

SILENT HILL 3 has a few *.000 videos inside of /data/movie/.

I have code to convert these into .mpg (MPEG1) files, but it's a bit of chore to actually get these to play in SDL (get frame, convert to texture and draw). Should we use ffmpeg for this, as it already does all of the work decoding the output? There's a tutorial on doing this with SDL, but it involves .avi instead of .mpg, though I believe it is possible to convert.

This is more for my own sanity and the sanity of others. I've hit 3-4 bugs in GCC (though I've only outlined 2) that people may encounter.

@z33ky You can see what I mean here: https://github.com/Quaker762/sh3redux/tree/bugs/bugs

I'm going to open this issue now so everyone that's watching the repo, e.g @piratesephiroth @belek666 can discuss how we might overcome the hard-coded in game logic. Things such as events, level transitions etc are ALL hard-coded into the Silent Hill 3 executable. As @JokieW discoved and shows here, it's possible via memory editing to trick the game into changing levels for certain actions, such as in this case, viewing the mirror. There is no data driven programming at all in the original, C based engine.

The idea I have (@z33ky will hate me for this, but I can't see another way besides disgusting global lookup tables) is to have one class PER LEVEL. Obviously, this is very bad, and we'd need to have a complete list of level transitions, but I feel this would clean up a lot of the lookup table spaghetti that TEAM SILENT created. This way we can contain all hard-coded stuff in the class that it is needed, instead of having it spread out all over the place (and violate the nice OOP we have going).

Obviously way in the future, but it's probably good to at least discuss it.

We're nowhere even near the time fix these, but just for reference, this list is what we should aim to fix when we're ready to actually implement game stuff.

EDIT: Mainly note the key points. A lot of the outfits were removed, however, the data is still contained within Heather's mesh.

Printing out the buffer in it's entirety with

for(int i = 0; i < 2048; i++) std::printf("0x%x ", buffer[i]);

on Line 190-191 in types.cpp results in only 0s being printed. This is the case when passing a vector reference to it in texture as well.

Also, how would I copy values from the read std::vector into a struct? Google isn't really giving me any answers.

So I was trying to load a few textures to try and bind onto a quad, when I noticed I was getting strange reads from sh3_texture. As it turns out, pic.arc contains a weird header (I've taken to calling '40-bytes of bullshit') before the actual header, as seen below..

It would seem not all textures have this, and the only way to check would be to dump every texture file from pic.arc, which is possible.

How do you suggest we tackle this? I can always e-mail Mike or some other SH3 modders if we get too stuck. I thought it may have been the elusive colour palette for 8-bit textures, but it seems it's the same 40-bytes, even for 24-bit textures...

I'm getting weird values in the color palette of sys_konami.tex (section pic.arc, directory offset 0xE10, file offset 0x56C3E80). For some reason it's adding a few values of 0x0D in there? I'm not sure if this is due to the vfile buffer, or whether the problem lies in sh3_arc::LoadFile

Here's the problem, I'm stumped as to why this is happening..

I can confirm that Dump2Disk() isn't the issue, as the palette I'm dumping also contains the rogue values.

Because some OGL objects need a context before they are initialised, I think it's best if we provide a default constructor so functions such as Load( ) (for example, a shader program) can be called later instead of being automatically called by the constructor.

This also makes a bit more sense, as the we can specify something like

sh3::gl::program test;test.Load("program");

instead of having to specify the program name when the object is declared, so we can defer the loading to when we actually need it (like loading an msbmp later, not during object construction).

This will mainly affect sh3::gl::program, sh3::graphics::texture, sh3::graphics::msbmp etc.