Due to the way that tokens are merged together for arguments which require explicit whitespace, an
empty token could result in a fragment like:
{.type = 'I', .data = '/usr/include '}
Such a fragment would be treated differently than:
{.type = 'I', .data = '/usr/include'}
This difference causes the compiler to include a system include path as part of the additional includes, thus
breaking things like include path shadowing, resulting in random build failures. As such, we skip empty tokens
from the tokenizer as they do not have any relevance anyway.
William Pitcock