I have code which does IO operations (read and process) on large files (multi GB or sometimes TB). It is not practical for me to post real examples to stackexchange and I would not want to post code which requires downloads in order to reproduce - especially multi GB file downloads.
The fact that these files are large is crucial, otherwise the reviews would likely suggest to load the file into RAM, or the reviews may not be representative - an optimum process for a 10MB file may not be optimum on 10TB. However if I don't provide files to be operated on then my question could be flagged as off-topic as it won't run without the data. Even if the question was not flagged - without the files to be read it might make it harder for reviewers to provide constructive feedback at some of the obscure operations I have concocted to process these larger files.
I have been thinking of ways to generate pseudo-random data such that reviewers could generate relevant files of appropriate and variable size (10MB, 100MB, 1GB...) with an input flag, but I don't know that this would be sufficient as I recently had a question closed because "simplified versions... aren't the actual code".
What is the best way to post my code for review?
- Post the code knowing that no reviewer can meaningfully run it
- Post code with a representative generator, knowing that it will be a simplification
- Something else?