kungfooman / appstract Goto Github PK

The DeleteFile function should be implemented:
http://msdn.microsoft.com/en-us/library/aa363915(VS.85).aspx
And maybe also the RemoveDirectory function:
http://msdn.microsoft.com/en-us/library/aa365488.aspx

Start implementing unit tests for existing code,
and use unit testing where possible for all future code.

Now that the registry virtualization engine's basic development is running to 
an end and that the file system virtualization engine has already proved to be 
reliable, it's time for a major clean up of the complete virtualization engine.

This clean up should at least fix the following issues:
* Inconsistent naming between registry and file system functions
   For example: DoCreateFile vs RegCreateKeyEx_Hooked
* Centralize the declaration of native types and functions
* Encapsulate and centralize calls to native types and functions
* ...

A lot has changed since the original architecture has been defined.
During implementation the core steadily shifted from host side to target side.
As a result, some existing code became overdesigned or is now just in the wrong 
place.

The system is now quite close to being somehow functional for native targets, 
therefore it might be a good time to revise the existing architecture.
A revised architecture would ensure that in the future the virtualization 
engine can be easily expanded and that new features can be added without 
unnecessary efforts.


At least the following areas need attention:

- Redefine Core project.
Core contains a mix of code, targeted at either the host or server side, or at 
both sides.
Perhaps the current Server project should be part of the new Core, while the 
current Core should be split up into the existing and/or new projects.

- Revise database access methods.
Is there still a point in only allowing the host to access the database? Doing 
this causes (unnecessary?) expensive transfers of data between host and target, 
while the host side doesn't really do anything with the data read and sent. One 
of the original reasons for using this system was to avoid conflicts between 
the virtualization engine and its own database access.

- Revise database access API.
Currently a generic command based system is used.
During development of the virtualization engines, database access has become 
significantly less complex (e.g. the filesystem engine doesn't use any database 
anymore). Therefore it should be investigated if the current system is 
overdesigned for its current goals, and/or for its future goals.

- Revise purpose of Watcher.
Can the Watcher be merged with the Host? Needing two processes to support a 
single target process is simply not justifiable.
The original purpose of Watcher was to ensure having a simple/stable process 
which would always ensure a clean GAC, in case the Host would leak some garbage 
there.
It should be investigated if, as an alternative, it would be save to let the 
watcher run in a separate application domain in the host process.

- Revise lifetime of Host.
Is it necessary to leave the host running once the target process is set up?
Note that this conflicts with the previous statement, since combining both 
statements would result in leaving target process' running on their own.
Is this safe? How will a clean system be ensured?

Currently the guest sends it's log messages synchronously to the host, where 
the message is then written synchronously to the output stream.

Marking the log-methods specified in IServerReporter with the 
OneWayAttribute makes target processes start up in mere milliseconds, 
whereas the current start up time may take up to 10 seconds. A negative side-
effect of this change is that log message are processed extremely slow. And 
messages that didn't arrive before the guest process terminates will never 
make it to the log.

Therefore log messages must be sent fast and asynchronously. Either by 
giving each message it's own thread from a thread pool, or by using a 
message queue which sends multiple messages as a batch.

It is documented by Microsoft that RegQueryValueEx returns the unnamed/
default value of the key (if available) when the lpValueName argument is 
null or an empty string. The virtual registry should also implement this 
behavior.

http://msdn.microsoft.com/en-us/library/ms724911(VS.85).aspx

The MarshallingHelpers-class corrupts marshals corrupted data to unmanaged 
memory.
Currently all data is converted to a byte-array before marshaling, this is 
not a reliable (or working?) solution.

When the target is a console application developed in for example C#, an 
AccessViolationException occurs almost instantly.

Implement the following functions:
http://msdn.microsoft.com/en-us/library/ms724862(VS.85).aspx
http://msdn.microsoft.com/en-us/library/ms724865(VS.85).aspx

Implement the following functions:
http://msdn.microsoft.com/en-us/library/ms724845(VS.85).aspx
http://msdn.microsoft.com/en-us/library/ms724848(VS.85).aspx

Steps to reproduce the problem:
1. Put a string in VirtualRegistryValue.Data
2. Send that object inside a VirtualRegistryKey to RegistryDatabase, so 
it's written to the SQLite database (remember to let the thread wait, 
writing is asynchronous)
3. Read the object using RegistryDatabase
4. Verify if VirtualRegistryValue.Data equals the value used in step 1

Expected output would be the exact same value before writing and after 
reading.
In stead SQLite returns a byte[] for the string. This is somehow expected 
behavior of SQLite, therefor VirtualRegistryValue.Data should either be a 
byte array or should either automatically convert to the expected type 
(based on VirtualRegistryValue.Type) before exposing the value.

A third kind of registry provider should be introduced, next to the 
virtual and transparent registry implementations.

This new provider must ease the manageability of the virtual registry,
by providing the following features:
- Block the retrieval of a handle for requests that match an underlying 
dynamic list of registry keys and values.
- Make specific keys transparent or not-transparent, overriding the 
default AccessMechanism
- Must have priority over the other registry implementations.
- Keep track of a hit count and perform a specified action when reached 
(eg removal of the block-list)
- Underlying lists must be completely dynamic
- ...


By introducing such a provider, the following could be fixed and/or 
improved:
- Fix support for managed targets
When starting a managed target a blacklist could be created with the keys 
and values needed by .NET to dynamically invoke a method. A maximum hit 
count can ensure the removal of these entries when the method invocation 
has completed.
- Provide means to manage the virtual registry
For example, currently installers are in some cases able to detect they 
have been installed already. System administrators could trick targetted 
installers by specifying a list of keys and values for which the error 
code FileNotFound must always be returned.

The system writes and reads duplicate values to the file and registry 
databases.
The source of this problem lays either within the virtualization engine or 
in the database logics.

The virtual registry should be able to recover from unknown registry key 
handles.

Necessary steps:
- Retrieve associated name
- Retrieve required access mechanism
- Pass the data to the associated virtual registry implementation (which 
data? and how to?)

The Engine Rule Framework should basically provide users with the ability to 
onfigure specific AccessMechanism values for specific keys, key trees, or 
key values.

The EasyHook library is provided with functionality guaranteeing the GAC 
is cleaned each time the installed dll's aren't needed anymore. EasyHook 
achieves this by running an instance of the EasyHookSvc.exe.
This results in AppStract needing 3 processes to run a virtual application
 - the host process (AppStract.Manager.exe or AppStract.Host.exe)
 - the guest process (the virtualized application)
 - an EasyHookSvc.exe process

This has 4 major disadvantages:
 1) An extra library requirement is introduced (for EasyHookSvc.exe)
 2) The need for 3 processes is a relativly expensive requirement
 3) The EasyHookSvc library can't be easily expanded and debugged
 4) The EasyHookSvc library can't easily use the same log functionality as 
shared by the server and guest process

But also, in some rare cases the EasyHookSvc.exe might crash unexpectedly 
while the guest process is still running. This means libraries are left in 
the GAC, which might result in the AppStract system becoming unstable and 
even unusable on the users operating system, until a system administrator 
is able to clean the assemblies from the GAC.

In order to make all of this more efficient and reliable, the following 
steps have to be taken:
 1) Analyze the functionality provided by the EasyHookSvc library
    a) Determine whether this functionality should be moved to a more 
suitable library
    b) Determine which process should be responsible for installing 
assemblies in the GAC and/or removing assemblies from the GAC.
 2) Implement the conclusion

The warning saying that users should not choose to start the application 
when the installer exits, should be removed.
In stead the CreateProcess function should be hooked and stopped.

The AccessMechanism.Virtual value would specify that only the virtual 
registry must be used, and that the host's registry must never be accessed.

The Virtual value should only be applied on a specific selection of keys.
An example use case could be HKEY_LOCAL_MACHINE, which always uses 
TransparentRead, while in some cases (for example while packaging) 
HKEY_LOCAL_MACHINE\SOFTWARE\ might have to be accessed completely virtual 
except for for example HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion
\Policies

The most convenient place to declare a list of such specific keys would be 
in a rules framework (see Issue 16)

Invoking the main method of the target application is broken. It's assumed that 
invoking any method from any assembly other than the injected ones, is also 
broken.

When loading an assembly the CLR requests registry keys which the virtual 
registry isn't able to provide. These are all related to policies which the CLR 
needs to apply on the assembly.

The problem is tracked down to (for example) the following key: 
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion\PublisherPolicy\Default
Running with administrator privileges, it's possible to get handles until the 
Fusion subkey. Trying to get a handle to PublisherPolicy and beyond always 
results in ERROR_FILE_NOT_FOUND, even if these keys are visible and accessible 
with the regedit utility. The result is the same for both C# and native C++.


A possible cause for this issue would be that the CLR uses custom options and 
custom access rights matching the original security descriptor. For example, 
when querying such policy keys the CLR uses the custom access rights 0x20119. 
While the engine always uses default access rights.

The file system's virtualization engine should be reviewed and reworked.
The main goal is to make the engine's footprint as small as possible by 
reconsidering the complete implementation of the engine.

One of the main components that possibly can be dropped would be the virtual 
file system table. It might be possible to replace this completely by an 
extremely robust and flexible path-redirecting-engine.
This could dramatically improve performance, because the file system's 
synchronization cycle between guest and host (database) would become obsolete.

During this rework, the engine must also be expanded with an engine rule 
framework, similar to the one in the registry engine.

The current registry virtualization is completely broken and unreliable.

The log would start with entries like this, and continue like this for the 
next few 100 lines:
...
2010-03-04 15:40:32.817000 [Debug] [Guest 10008] [8] QueryValue(HKey=896 
ValueName=CurrentConfig) => InvalidHandle
2010-03-04 15:40:32.855000 [Debug] [Guest 10008] [8] CloseKey(HKey=924)
2010-03-04 15:40:33.182000 [Debug] [Guest 10008] [8] OpenKey(-2147483646\
\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer) => 931
2010-03-04 15:40:33.196000 [Debug] [Guest 10008] [8] OpenKey(-2147483646\
\software\microsoft\windows\currentversion\policies\explorer) => 931
...
Eventually (if lucky) the targeted process starts, making it look like the 
virtualization engine "works".
But in reality the virtualization engine provides keys starting at handle 
21, obviously handles like 931 aren't provided by AppStract if the engine 
didn't provide the previous 920 handles.
OpenKey is able to return those handles because they are buffered an 
written to the databases by the virtual registry.


This issue is submitted to be able to keep track of enhancements and fixes 
to the virtualization layer.