ADL is a modular dictionary learning toolbox for plug-and-play sparse approximation and dictionary update steps.

ADL implements:

• several of the popular sparse approximation techniques
• many of the dictionary update steps
• overall full dictionary learning algorithms (like MOD, K-SVD, ITKrM etc.)

The main features of the toolbox are:

• Adaptive dictionary size
• Training of replacement atoms
• Several image/synthetic demos
• Generation of synthetic datasets
• Coherence constraints
• Structured dictionaries
• Adaptive sparsity
• Avoid fitting noisy atoms
• Several initialization techniques

Who could use the toolbox?

I. dictionary learning researchers:

• easily develop new ideas
• easily compare with previous work
• reproducibility of results

II. other researchers:

• apply dictionary learning to your application
• no/little parameter tuning
• easily compare several dictionary learning techniques and choose the best one

In the first demo blind run we just load some image data and then run the adaptive dictionary learning algorithm without setting any of its parameters. The toolbox falls to its default setting and for each choice it outputs a warning to let the user know about the choice that was made.

Then we implement two of the most popular dictionary learning algorithms: K-SVD and ITKrM.

In both cases, we fix the sparsity level and the size of the dictionary.

The first two demos are just implementations of well-known dictionary learning algorithm. This demo shows how the toolbox can be used to experiment with new ideas and combine old ones to produce new results. The hybrid demo combines the dictionary update step of K-SVD with the thresholding sparse approximation step used in ITKrM.

The purpose of this demo is to highlight how easy it is to use the toolbox to experiment with different combinations of sub-routines.

That that we have covered the basic examples, let's move to a more interesting case.

Consider that we want to construct a dictionary with a fixed number of atoms (say K = 80 atoms) such that the representation accuracy is improved over a given number of iterations. The atom replacement demo implements ideas of pruning old atoms and adding new atoms such that the representation error is improved.

A notion of importance (or ranking) is attached to each atom and the worse atoms are pruned while a new training procedure takes care of adding fresh blood to the dictionary.

The mutual coherence of a dictionary plays an important role in the theoretical understanding of sparse coding in that dictionary. It is therefore useful to have a dictionary learning method that caps the mutual coherence to a maximum imposed value. The mutual coherence demo deals with this problem by pruning highly coherent atoms.

In most dictionary learning algorithms there are two parameters that need to be chosen, or tunned: the size of the dictionary and the sparsity level of the representations.

The adaptive dictionary size demo keeps the sparsity level fixed and tries to find the dictionary of best size. Having few atoms we might expect bad results while increasing the number of atoms might leads us to fitting noise components. There is a sweet spot in the middle that we are searching for.

Finally, the full adaptive demo deals with both the sparsity and the dictionary size adaptively.

[1] Michal Aharon; Michael Elad; Alfred Bruckstein (2006), K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation, IEEE Transactions on Signal Processing, 54 (11): 4311–4322, doi:10.1109/TSP.2006.881199 http://www.cs.technion.ac.il/~elad/publications/journals/2004/32_KSVD_IEEE_TSP.pdf

[2] Karin Schnass (2018), Convergence radius and sample complexity of ITKM algorithms for dictionary learning, Applied and Computational Harmonic Analysis, 45 (1): 22-58
https://arxiv.org/pdf/1503.07027.pdf