Sorting large files

Good evening everyone, Hoping I might be able to gleam some info for you tonight. Best Answer. Gerard This person is a verified professional. Verify your account to enable IT peers to see that you are a professional. View this "Best Answer" in the replies below ». Popular Topics in Microsoft Office. Which of the following retains the information it's storing when the system power is turned off?

Submit ». Doughnut This person is a verified professional. SullyTech This person is a verified professional. Hi See this link for the full constraints of Excel Y Feb 7, at UTC. Doughnut Destroyer wrote: Technically you should be able to sort all of them. To expand on the original info I was given and with my testing.

It seems that the issues are connected specifically to filtering and NOT sorting. User is trying to look through VIN numbers and have noted that the majority of the entries would be non-recurring. I was able to get it to work albeit a bit slowly, but with so many different options for the filtering could that cause crashes? If each record has been written to the input file using ObjectOutputStream then we specify the java Serializer:.

This is a good option for many binary formats. Let's use a binary format with a person's name and a height in cm. We'll keep it unrealistically simple with a short field for the length of the persons name, the bytes of the name, and an integer for the height in cm:.

In that case make a type T that can be header or an item and have your serializer return that T object.

In your comparator ensure that the header is always sorted to the top and you are done. To fully do your own thing you need to implement the Serializer interface. Your large input file might have a lot of irrelevant stuff in it that you want to ignore or you might want to select only the columns from a csv line that you are interested in.

You can use the java. Stream API to modify the input or use direct methods filter , map , flatMap :. Having sorted to a file f , you can read from that file like so Reader is Iterable :. You might want to deal with the results of the sort immediately and be prepared to throw away the output file once read by a stream:. The interaction is a little bit clumsy because you need the stream to be auto-closed by the try-catch-with-resources block.

Note especially that a terminal operation like. When called, the close action of the stream deletes the file used as output. If you don't close the stream then you will accumulate final output files in the temp directory and possibly run out of disk. The fact that java. Stream has poor support for closing resources tempts the author to switch to a more appropriate functional library like kool. We'll see.

At the same time however, I began to consider what the solution could be for programmatic production level code that can run independently of other tools. How this can be done is similar to how the unix sort command was implemented: the unix sort command uses External Sorting , a method similar in concept to a merge sort. In this approach, the file to be sorted is read in chunks, each chunk of which is sorted independently of the next, and each written to its own temporary file. Afterwards, one is left with several files - each with its contents sorted - that must be spliced back together.

This can be accomplished by reading lines from each file into memory and merging the lines together in sorted order, writing out to the final file as the algorithm moves along through each temporary file. The following code is a java class which has the very basic implementation of this sorting algorithm. The method splitChunks performs the initial sorting into temporary files.

The method mergeChunks merges the temporary files into the final merged file. Another more extensive implementation of this algorithm is available as an open source External Sort library. Using perl , the header was removed and sorting column split into their own columns.