The con "Regenerating the output of a test, or all tests, is way too easy." seems like a huge con to me. Large test suites are supposed to be useful for regression testing and verifying changes are only changing what they're supposed to. Having the golden files auto-regenerate to whatever the updated compiler spits out invalidates the tests. You mention validating the changes with Git, but a cursory check of the diff is very different than manually going in and changing files. The change described here seems to require more discipline by the human user to "do it right" vs. "regenerate and lgtm" which is often the wrong direction when dealing with test suites. Everyone, including me, loves doing easy shortcuts. Test suites are purposeful friction to make it easier to do things right than cheat.
Golden tests are a great idea in general for programming language projects :)
Markdown is certainly readable, but it does make the program slightly harder to run by hand. LLVM's lit lets you use a source file with comments, which is a nice approach too. You can open the file in an IDE as normal.
The con "Regenerating the output of a test, or all tests, is way too easy." seems like a huge con to me. Large test suites are supposed to be useful for regression testing and verifying changes are only changing what they're supposed to. Having the golden files auto-regenerate to whatever the updated compiler spits out invalidates the tests. You mention validating the changes with Git, but a cursory check of the diff is very different than manually going in and changing files. The change described here seems to require more discipline by the human user to "do it right" vs. "regenerate and lgtm" which is often the wrong direction when dealing with test suites. Everyone, including me, loves doing easy shortcuts. Test suites are purposeful friction to make it easier to do things right than cheat.
Golden tests are a great idea in general for programming language projects :)
Markdown is certainly readable, but it does make the program slightly harder to run by hand. LLVM's lit lets you use a source file with comments, which is a nice approach too. You can open the file in an IDE as normal.
So basically your gains are due to Golden Testing mainly