I've been thinking about program understanding, and about how to encourage #languagemodels to do compositional/verifiable #reasoning on program text ("statically").
If, as some recent literature suggest, transformer-based LMs are not more expressive than regexps, this line of thinking is doomed, but at least it could be a valuable heuristic and complementary to rigorous #formalverification. #machinelearning#nlproc