How do I post code which is reliant on large file IO operations?

Question

I have code which does IO operations (read and process) on large files (multi GB or sometimes TB). It is not practical for me to post real examples to stackexchange and I would not want to post code which requires downloads in order to reproduce - especially multi GB file downloads.

The fact that these files are large is crucial, otherwise the reviews would likely suggest to load the file into RAM, or the reviews may not be representative - an optimum process for a 10MB file may not be optimum on 10TB. However if I don't provide files to be operated on then my question could be flagged as off-topic as it won't run without the data. Even if the question was not flagged - without the files to be read it might make it harder for reviewers to provide constructive feedback at some of the obscure operations I have concocted to process these larger files.

I have been thinking of ways to generate pseudo-random data such that reviewers could generate relevant files of appropriate and variable size (10MB, 100MB, 1GB...) with an input flag, but I don't know that this would be sufficient as I recently had a question closed because "simplified versions... aren't the actual code".

What is the best way to post my code for review?

• Post the code knowing that no reviewer can meaningfully run it
• Post code with a representative generator, knowing that it will be a simplification
• Something else?

Answer 1

I don't think that the absence of data files would make your question off-topic, but I do applaud and appreciate your desire to help reviewers.

Your idea of including a data file generator is a good one (not necessarily using the same programming language), and I recommend you do that. If you want to emphasise that you're not looking for a review of the generator program, it may help to put that code in a blockquote section (prefixed with > ) so that it is easier to tell apart.

On the other hand, you might want to write the generator to production standards, and actually have that reviewed, too. It sounds like something you might want to keep around for your own use from time to time, particularly if it can produce a deterministic output.

I also encourage you to include unit tests of the main code that exercise the functional requirements with all the edge cases you can think of. Even if you're mainly looking for a performance review, these can help reviewers to understand how you've interpreted the requirements and to check any proposed improvements for correctness.

I'm looking forward to seeing your code for review! I just hope I'm able to answer...