@lritter
I guess with "xterm based" you mean "outputs to a VT102 or similar" terminal, supporting the usual color escapes etc?
I think the only graphical terminal output SDL ever supported was libcaca and aalib in SDL1.2
But you most probably don't want pixels displayed as ASCII art, but something more text-focused. For SDL that wouldn't just be a backend, but an entirely new API, because it doesn't really support text output and doesn't have GUI abstraction (widgets or whatever)