Hierarchical modulation, also called layered modulation, is one of the techniques for multiplexing multiple data streams and modulating them into one single symbol stream, where the base-layer symbols and enhancement-layer symbols are superimposed before being transmitted. When hierarchically modulated signals are transmitted, users with good reception and advanced receivers may demodulate more than one layer of data streams. For a user with a conventional receiver or poor reception, it can only demodulate the data streams embedded in low layer(s), e.g, the base layer only. From a network operation perspective, a network operator can seamlessly target users with different services or QoS’s and this technique.

Further from an information-theoretical perspective, hierarchical modulation is one practical implementation of superposition precoding, which can help achieve the maximum sum rate of Gaussian broadcast channel with employing interference cancellation by receivers. However, from a signal processing perspective or an implementation stand-point, traditional hierarchical modulation suffers from interlayer interference (ILI) so that the achievable rates by low-layer data streams, e.g. the base-layer data stream, can be dented by the interference from high-layer signal(s).

For example, for the hierarchically modulated two-layer symbols comprising a 16QAM base layer and a QPSK enhancement layer, the base-layer throughput loss can be up to about 1.5bits/symbol with the total receive signal-to-noise ratio (SNR) of about 23dB. This means, due to ILI, there is about 1.5/4 = 37.5% loss of the base-layer achievable throughput with 23dB SNR. The demodulation error rate of both the base-layer and enhancement-layer symbols increases due to ILI.