divclassbutton, 5 months ago

@bobulous

Tbh I would keep the parser as UTF-8 only, do some checks beforehand if you can and convert before feeding in; thats what we do.

Modern JSON demands UTF-8, but it was allowed in the past to be UTF-16. YAML says you must accept UTF-16 or 32 for JSON compatibility.

https://yaml.org/spec/1.2.2/#52-character-encodings

bobulous, 5 months ago

@divclassbutton Well, the battle with dynamic dispatch in #Rust is currently going so badly that I might end up leaving it as UTF-8. But I'll keep fighting with it a while longer, seeing as it would be useful to have the software detect supported encodings automatically.

janriemer, 5 months ago

@bobulous I might miss some things here, but can't you use static dispatch like this?:

• one struct XmlParser for the consumer of your library
• multiple internal xml parsers as structs - for each encoding one struct => all implement same trait XmlParse

1. user calls XmlParser.parse(xml)
2. encoding is detected
3. depending on the encoding, an internal XmlParser is chosen

@divclassbutton

bobulous, 5 months ago

@janriemer @divclassbutton Maybe it's the way I've structured things, but because the encoding can't be determined until runtime, the compiler can't know which decoder to use. So, even though I'm already using a trait for the decoder, dynamic dispatch is needed to select the specific decoder at runtime.

I had almost no trouble with the borrow checker in eight weeks of working with static generic types, but after seven hours yesterday the dynamic code is wildly broken.

janriemer, 5 months ago

@bobulous Hm...but can't you just do something like this? (again, maybe I'm oversimplifying here):

fn parse(xml: String) {
match detect_encoding(&xml) {
Encoding::Utf8 => XmlUtf8Parser::parse(&xml),
Encoding::Utf16 => XmlUtf16Parser::parse(&xml)
// other encoding here...
}
}

the parse method at the top is an impl on your pub XmlParser, while the other XmlParsers for the encoding are internal only.

@divclassbutton

bobulous, 5 months ago

@janriemer @divclassbutton I got the dynamic dispatch versions working eventually (tip: hunt down every dyn Trait and replace it with dyn Trait + 'a with extreme prejudice).

Then I decided you were right, and created static functions which simply wrap the messy bit in a non-generic type so that the client code doesn't need to worry about it. This won't allow the decoder to be swapped mid-stream, but I'm no longer confident that's really needed.

#Rust #RustLang

janriemer, 5 months ago

@bobulous Yay, nice! Glad you found a solution to the problem.

Thank you for letting me know.🙂