This is the repository for paper Eigen component analysis: A quantum theory incorporated machine learning technique to find linearly maximum separable components. It includes two main parts for the experiments, Eigen component analysis (ECA) and eigen component analysis network (ECAN). Either ECA or ECAN can be trained with vanilla eigen copomnent analysis (VECA) or approximated eigen copomnent analysis (AECA). As the article mentioned, VECA often result in a sparse result and better option for dimension reduction.

ECA, in my humble opinion which I cannot say in the article, is a top-ranking feature extraction or dimension reduction algorithm. The obtained eigenfeature matrix (EFM) and eigenfeature-class mapping matrix (ECMM) could be used to conduct the concrete dimension reduction. The concrete number of dimension reduction is near to the rank of whole data set if without background or noise. In addition, the variance ratio is close to 1 with this concrete dimension reduction. For example, the concrete number of dimension reduction for MNIST data set is 110 using VECA (or 328 using AECA) neither more or less. This could mean that the MNIST data set or the background/noise-free data set only occupy a subspace with dimension 110. The difference between the result of VECA and AECA is that VECA ignored some less important information. In ECAN, with the dimension operator, the nonlinear dimension reduction outperforms many classical algorithms which will be reported in our future work.

• Eigen component analysis (ECA) introduction
• Quick start
  • Directory structure
  • Train VECA
  • Train AECA
  • Train the 2-fold ECAN with AECA on MNIST data set
  • Train the 2-fold ECAN with VECA on MNIST data set
• Dimension reduction
  • ECA
  • 2-fold ECAN

The core algorithm is implemented in real_eigen.py (complex_eigen.py is independently implemented). The base model for training is named with prefix base, which could be run independently or work as a module. And ECAN related file are suffixed with network. Data loading is implemented in load_data.py. The comparison with other models is implemented in other_models.py. The obtained EFM, ECMM, RaDO or ReDO are stored in directory history.

• Analytic ECA: analytic_eca.py, which can find an analytic solution for full rank data sets
• Approximated ECA: base_approx.py
• Complex ECA: complex_eigen.py, base_complex_eigen.py

All the data_tag in analytic ECA and base model could be changed to train other data sets. The history and checkpoints are managed by MAGIC_CODE in real_eigen.py/complex_eigen.py and WORK_MAGIC_CODE in each to-be-executed file.

Two hyperparameters for the constraints on EFM and ECMM should be set before training:

RealEigen.HP_ORTHONORMAL = 0.001
RealEigen.HP_EIGENDIST = 0.001

They both can be set as a small number (empirically 1e3 or smaller) with a relative large number of training epochs. For data set with lower dimension, a relative large value on these two hyperparameters will accelerate convergence. The relaxing constraint on EFM usually make no difference. However, if the obtained ECMM is not binary, the corresponding hyperparameters should be set larger. Generally, training VECA is easier than AECA , because of the binarity and sparsity of ECMM, which reflected in lazy hyperparameters setting of VECA.

• The files include twodim.py (data_tag="2d"), threedim.py (data_tag="3d"), bc.py (data_tag="breast_cancer"), wis.py (data_tag="wis"), mnist.py (data_tag="mnist") correspoinding to 2D, 3D, Wis1992, Wis1995, MNIST data set mentioned in the article.

• Set the to_train option to be True otherwise it will just test on previous saved model.

• Then training on Wis1992 should be

  python bc.py

• One could change the data_tag to test other data sets.

  python base_approx.py
  

• The code for ECAN is in base_network.py. One could change the data_tag to these one mentioned in load_data.py

• Set the dimension operator to be used

  # dimension operator, set quadratic false to use ReLU and neural network (not fully connected)
  model = build_model_do(
      state_len, num_classes, 
      to_raise=True, to_reduce=False, 
      raise_quadratic=True, reduce_quadratic=True)
  
  # using fully connected neural networks as dimension operator
  # model = build_model_dnn(state_len, num_classes, to_raise=True, to_reduce=True)

• Train with AECA

    # vanilla
    # ECMM = EigenDist
    # vanilla
    # model.compile(loss=[categorical_bernoulli_crossentropy, categorical_bernoulli_crossentropy],
                  # loss_weights=[0.5, 0.5],
                  # optimizer=keras.optimizers.Adadelta(),
                  # metrics=['accuracy'])
  
    # approx
    ECMM = EigenDistApprox
    # model.compile(loss=keras.losses.categorical_crossentropy,
    # model.compile(loss=keras.losses.mean_squared_error,
    # model.compile(loss=categorical_bernoulli_crossentropy,
    # approx
    model.compile(loss=[categorical_crossentropy, categorical_crossentropy],
                  loss_weights=[0.5, 0.5],
                  optimizer=keras.optimizers.Adadelta(),
                  metrics=['accuracy'])

• Set to_train to be True and training on MNIST data set

  python base_network.py

• The only difference from training with AECA is in this block of code

  # vanilla
  ECMM = EigenDist
  # vanilla
  model.compile(loss=[categorical_bernoulli_crossentropy, categorical_bernoulli_crossentropy],
                loss_weights=[0.5, 0.5],
                optimizer=keras.optimizers.Adadelta(),
                metrics=['accuracy'])
  
  # approx
  # ECMM = EigenDistApprox
  # model.compile(loss=keras.losses.categorical_crossentropy,
  # model.compile(loss=keras.losses.mean_squared_error,
  # model.compile(loss=categorical_bernoulli_crossentropy,
  # approx
  # model.compile(loss=[categorical_crossentropy, categorical_crossentropy],
  #               loss_weights=[0.5, 0.5],
  #               optimizer=keras.optimizers.Adadelta(),
  #               metrics=['accuracy'])

In the history folder, with the corresponding MAGIC and WORK MAGIC code, we can find the obtained EFM P , ECMM LL. In the 2-fold ECAN, EFM and ECMM has suffix a number indicating the corresponding fold. The RaDO or ReDO all belong to the 1st fold. In ECAN, the identity operator is installed every other fold since two consecutive dimension operators in a row are trival.

LL = np.round(LL)

x_norm = np.linalg.norm(x, axis=1, keepdims=True)
x /= x_norm
x1 = np.matmul(x,P[:,np.sum(LL, axis=1)==1])

LL = np.round(LL)

x_norm = np.linalg.norm(x, axis=1, keepdims=True)
x /= x_norm
# change of basis
psi = np.matmul(x,P1)

# nonlinear dimension reduction
x1 = ReDO(redo, psi)

x1_norm = np.linalg.norm(x1, axis=1, keepdims=True)
x1 /= x1_norm
# linear dimension reduction
x2 = np.matmul(x1,P2[:,np.sum(LL2, axis=1)==1])

The reducing dimension operator (ReDO) is defined as

def ReDO(redo, psi, real_eca=True):
	x = np.concatenate([np.ones(psi.shape[0],), psi],axis=1)
	x = x * x
  
	# guarantee the square root being legal in real ECA
	if real_eca:
		redo = redo * redo 					
    
	return np.sqrt(np.matmul(x, redo))