In 802.11g, the channel bandwidth is 20MHz, but why the IQ Baseband Filter cutoff frequency is only at 9MHz or 10MHz? I thought the baseband IQ filter should be at 20MHz since the input signal to the IQ filter is 0Hz to 20MHz.
Remember that the baseband signal is complex (I and Q). For 802.11g, the “input signal to the IQ filter” is not 0-20MHz. The baseband signal consists of both I and Q, both of which are 0-10MHz. When upconverted, this corresponds to a 20MHz RF bandwidth.
Also note that although the channel bandwidth is 20MHz, the actual bandwidth required for the signal is less. The 802.11g spectral mask has an 18MHz wide central portion, therefore 9MHz filtering (at baseband) is appropriate.