[BUG] Memory consumption is very high during the extraction of files in MSIX about msix-packaging HOT 6 OPEN

Sadly the MSIX SDK doesn't support parallel unpack, but I don't see how just calling CopyTo in a different thread after the IAppxFile was obtained via IAppxFilesEnumerator::GetCurrent will fail.

You just need to make sure that the IAppxFilesEnumerator calls are made in the main thread and being careful about the ref count for it.

I might be wrong, but it is worth trying.

from msix-packaging.

The memory of the input streams are not released because the owner is the IAppxPackageReader object (by the time the stream is retrieve in ExtractFile the ref count should be 3 or so). The output stream is released once the ExtractFile call is completed.

What is happening here is that one the stream gets copied into the output stream, if the stream hasn't been inflated we will allocate some buffers. Those allocations are what you are seeing between the two breakpoints in the screen shot.

Now, I agree that this is problematic for big packages. The easiest way to fix it would be to clear those buffers once the inflate is done which is basically calling

                self->m_compressedBuffer.get()->clear();
                self->m_compressedBuffer.get()->shrink_to_fit();
                self->m_inflateWindow.get()->clear();
                self->m_inflateWindow.get()->shrink_to_fit();

in this block

If you can easily repro can you try it?

As of the system account issue, I'm not that really familiar so I can't assure that there's a memory limit where there are no users logged in.

from msix-packaging.

@msftrubengu Thanks a lot for your reply. We have tried it and it is working. I have one query wrt parallel installation of multiple MSIX packages via MSIX SDK.
The ExtractFile is using IStream class which is inherited from ISequentialStream. Is it right to invoke fileStream->CopyTo from different threads simultaneously ?
We have observed some crashes in the past wrt parallel extraction of MSIX packages and hence, we want to understand whether the parallel extraction using ISequentialStream is supported or not.

from msix-packaging.

@msftrubengu Just want to confirm wrt parallel unpacking of MSIX packages using MSIX-SDK
Is it ok to perform unpack operation for different MSIX packages together ?
So, we can always perform the unpack operation of a single MSIX package in a single thread but it is completely alright to invoke the unpack of multiple MSIX packages together in separate threads.

As, per your above comment: You just need to make sure that the IAppxFilesEnumerator calls are made in the main thread
Currently, we are keeping a global mutex here:

Due, to this change - we are facing performance issues and hence, I am exploring the option to keep the unpacking of multiple MSIX packages in separate threads at the same time.

from msix-packaging.

@mtodd @tpope @sverrejoh @msftrubengu : I am still waiting for the reply on my above query.

from msix-packaging.