Asker's problem was that, after a certain amount of time, the program inexplicably slows down. In this case, the Asker knew there was a problem, but, had no idea what, and gave very little information that would point to the issue.
The bug in this case is the subtle interaction between multiple levels of call hierarchy, and that what was considered a performance enhancement in initial versions of Java (when memory was very expensive) has become (in special cases) a hinderance.
By themselves, the following code processes are fine:
- having input with millions of lines
- reading lines 1 at a time
- parsing each line to locate a small substring of them using a StringTokenizer.
- saving that 'small' string for each line.
At face value there does not appear to be any problem, but, by knowing the internals of how String, StringTokenizer, and Java memory management work the bug becomes apparent: the code is saving the entire file in memory, instead of just a small portion.