kellogh, 9 months ago

@persagen are you aware of similar work but it labels text as being factual in nature — not correct or incorrect, simply the kind of statement that COULD be wrong or subject to hallucinations. This work seems to validate veracity, but what if I just want to figure out if veracity might be in question?

persagen, 9 months ago

@kellogh
1/2
Tough questions! 😅 w/o studying the paper, it seems to be a early yet imp. comparison of a biomed knowledge graph construction using synthetic data generated by ChatGPT vs. published literature.
Issues:

• ontology depth: can be too coarse/detailed (granularity);
  *GPT / ChatGPT generative/probabilistic; w/o self-supervised augmentation (fact checking) or prompting prone to hallucination/noise (only mentioned 2-3x)
• different GPT versions/subversions behave differently, retrograde

persagen, 9 months ago

@kellogh
2/4

• this work is thus more proof of concept effort
• biomed literature / domain complex: unknown level of this expertise among authors
• that said v. interesting PoC
• would be interesting to see LLM triple-based KG, auto fact-checked (parts-of-speech? coreference resolution the hard issue)
• will be nice to see how this plays out; imo, structured KG (ontological; topics) tremendously important/usefu !
• noise always an issue, as well (in health) transparency, explanable << NOTE

persagen, 9 months ago

@kellogh
3/4

Can large language models build causal graphs?
https://arxiv.org/abs/2303.05279

Establishing Trust in ChatGPT BioMedical Generated Text: An Ontology-Based Knowledge Graph to Validate Disease-Symptom Links
https://arxiv.org/abs/2307.01128

AI and the transformation of social science research
Careful bias management and data fidelity are key
https://www.science.org/doi/10.1126/science.adi1778

persagen, 9 months ago

@kellogh
4/4

Construction of Knowledge Graphs: State and Challenges
https://arxiv.org/abs/2302.11509

Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases
https://arxiv.org/abs/2009.11564

A Framework for Large Scale Synthetic Graph Dataset Generation
https://arxiv.org/abs/2210.01944

kellogh, 9 months ago

@persagen thank you so much. Solid set of links. There goes my evening!