hot take: #cs academics should be expected not only to publish the full source of the programs that demonstrate their research (I can't believe this still needs to said), but they should also be expected to offer #ReproducibleBuilds of the environments so that the results may be reproduced at a later point in time. "it works on my machine" is no longer a legitimate problem and "I'm self-conscious about the code because its messy" both holds you to a lower standard than undergraduate CS students are held to for their assignments, while holding the reader responsible for reconstructing your research from some small set of source snippets, which amounts to imposing trust in the authority of university discourse, a kind of metaphysics.
taking a few weeks to learn how to use #guix for #ReproducibleResearch should not be too much to ask, knowledge production is mere discourse production when the results cannot be scrutinized beyond a peer review process without a good deal of effort, which we all know that few people have the time for. if you can't be bothered to get it working, then why should your readers be expected to assume that it works?
And while I welcome scientific CS papers, I am more worried that we take Big Pharma's word for their most profitable drugs to also be the most safe and effective ones available.
@alxlg@rml my favorite question out from the first "a.i." revolution back in the 2000s was "where is the web in the sw?" asked by one frank van harmelen in 2006. it's curious to reflect, then, how "a.i." was considered a symptom of what was "wrong" with the semantic web. considering the very nature of "a.i.", or at least what goes for discourse about it today, "reproducibility" is a curious criterion; "a.i." now is mostly, apparently, curated african labor.
@alxlg@rml "reproducible" according to what economic or political context, then? since apparently all this work is at once "socio-technological". to say the least, being forgiving isn't common to the echosystem that is "open source", like ever. and the tones are almost always and everywhere chauvinistic/universalistic. so, "make it reproducible and we won't think about how affordable that might be — for science marches on!" is just, well, upsetting for all future aims, i bet.
Just consider scientificity a spectrum where the more subjects (people, institutes, groups etc) can actually reproduce a result the more scientific it is.
@alxlg@rml bungists would probably consider that if a science is disjoint from all the sciences (however inconceivable that might seem) it fails inclusion as [interscience] (e.g., psychoanalysis, parapsychology). without a sense of values for cooperative ethos, collaborative effort, etc., "reproducibility for its own sake" is philosophically untenable. reproducibility alone, then, seems necessary but not sufficient. that's the point i am drawing on here.
True, I am not saying reproducibility solves science politics, I am just saying for it to be science in the first place its results must be realistically reproducible.
if reproducible builds became an academic standard, there would also be more investment into the development of tools for reproducible research. if you ran into difficulties, guix developers would have a better idea of what can be improved. it would become easier, and the value of reproducible builds not just for research but also for user's safety and functional software deployments would be clear. everything would improve, its a win/win :)
Imagine a mathematician publishing a paper claiming to find a solution for some algorithm, without publishing a complete expression of the solution. It would be unthinkable and the maths community would paint that person as seriously untrustworthy.
@vruz i commented elsewhere that much of computer science research amounts to programmer experience reports. which is ridiculous because with much of what we do is computational, you shouldnt need to trust anyway or even work through the entire source; just enough to understand nothing tricky is going on, and then you can reproduce the results yourself and even find edge cases.
On the topic of researchers not sharing the source of their research, @mattmight has an interesting solution: the CRAPL license, which holds that any reproduction of the source — for example in a blog post — won't include or make reference to implementation-specific quick fixes, gnarly glue code, etc, that isnt relevant to the outcome of the research (a more efficient implementation shouldnt effect the results, for examole), and as such the type of petty bullying that is all too common in the hacker sphere isn't tolerated.
@rml hot take indeed. I barely manage to compile some of the open-source dependencies for my work, not to mention any closed-source components. To get that stuff reproducible, I'd not only have to hire three more student employees, I'd have to outright scrap eveything that uses closed-source tools anywhere. I'm a firm believer in open source and scientific reproducibility, but I recognize it's often not even close to practical due to the ecosystem.
@rml The comparison with undergraduate course work doesn't hold water since research work is more complex by orders of magnitude. Again, I get your standpoint and I agree with your motivations, but I see a multitude of practical problems that will prevent us from getting there any time soon.
if you fought to get your dependencies to compile it sounds like you need reproducible builds, so that upstream takes care of them and you can primarily focus on your work. a system like guix provides a massive amount of python packages, most of what you need will already be packaged upstream. and if we opposed nihilism wrt to reproducibility, and all researchers were expected to produce a manifest for reproducing their research, then those of us who work on systems like guix would be able to make a business out of it, which more people want to do that there is demand at the moment; creating jobs that enhance scientific production is good.
what tools do you need that are essential to your research and don't have an open equivalent? I use all free software, and very rarely does it cause me trouble. most programming languages have open implementations. using proprietary software for scientific research should simply be disqualified from science, because they serve as an excuse to hamper verifiability.
but the idea that research work is too complex to demonstrate its veracity seems frankly wild. proofs sometimes run hundreds of pages. why do CS researchers get to just wave their hands like "trust me bro"?
the only answer seems to be "universities would have to hire a full time research dependency packaging engineer" and considering there is a jobs crisis that doesn't sound like a bad thing
@rml It's great that you can use all free software. I think you're making the mistake of projecting your use of software to all of CS. CS is a large field, and what you're doing might be very different from what other do. Banning non-free software would outright ban entire sub-fields of CS, which is why I"m saying it's not practical to ban it. Putting the burden on individual, generally underfunded research departments also cannot be the way to go. (1/3)
@rml On a more constructive note, I think the way forward in this dilemma would be two-fold. On one hand, significant public research funding has to be allocated towards tool development, aiming to replace dependencies on proprietary software over the next decades. On the other hand, academic venues such as conferences and journals should start valuing tooling work, allowing researchers to create better tooling instead of writing more papers without sabotaging their careers.
@rml In the meantime, I think blaming individual researchers, or even institutions for what is a systemic issue is the wrong approach. It's only going to alienate people, since there is a bunch of fields where no single researcher or even institution has the resources to actually replace all the proprietary or otherwise bad software in their stack.
@jaseg totally! I absolutely agree. thats the thing, I think institutions should be held accountable, and researchers should feel unsatisfied about their current conditions and have a strong case that the way things are is coming at the cost of science's legitimacy, and through that can organize change that could provide more jobs and producer better science and society as a result
I'm not blaming individual researchers, I think its an institutional problem, but I'm addressing individual researchers because I don't how institutional problems change except by recognizing that something is wrong that is unsatisfactory for meeting the criteria of science. But i have some thoughts on each top
We now have openUSD, which is the highest fidelity scene description format to date, which is incredibly generic, extensible and used in nearly every modern CAD software and will most CAM stuff is also integrating it;. openUSD Physics Schema can already describe most of what Houdini is capable of, which is already one of the most advanced suites for 3D simulation, and its at the heart of NVIDIA's Fabric continuous subdivision surface textures used to accurately simulate material physics in realtime, so I'm pretty sure it can handle most needs and software producers should be expected to support it. lots of researchers already target it and it should be standardized.
There are plenty of free FEM solvers. but I think that this another space where openUSD seeks to be a physics simulation interop solution. if they can't support a way to produce the results on an open platform, then it shouldn't be considered science
Again, I don't understand why scientists would need to use a proprietary HDL when there are so many free ones. if it's due to the demands of their commercial employer, we should be honest that they are doing product engineering research and not science.
4.that sounds like hardware problem. most FPGAs can be reasonably emulated, and SDR is also a hardware problem. but I'm assuming you have the hardware requirements, you want to validate the idea/software described in a paper.
Nvidia is the big bad guy here but they seem to even be open up their drivers. relying on closed cuda kernel technology shouldn't acceptable for science.
Is it a particular proprietary execution environment? then idk why research on that would be considered science.
again, if its science I think it should be demonstrable with an open solver.
yeah accessibility of hardware and reproducing physical experiments seems to me to be another issue; nevertheless, people with the equivalent hardware available to them shouldn't find it difficult too cumbersome to reproduce your results.
but I think simply accepting the reproducibility crisis as intractable when it isn't is unacceptable for science.
@jaseg take the double slit experiment. its easily reproduced in an introductory physics course. and software is virtual, it should be even easier to reproduce than the double slit experiment. but misprioritized expectations of researchers that place economic metrics (papers published, etc) over scientific standards has resulted in most CS research disregarding reproducibility altogether, and its been normalized to not even share the source code, so what we're usually left with is something like experience reports when the research isn't reducible to a functional pearl
Add comment