Moving a lot of files under Linux (aka messing with MS COCO)

Microsoft COCO is a recent annotated database for object detection, segmentation, tagging, etc. It’s one order of magnitude larger than previous similar databases available. Here a comparison of sizes:

So when working with COCO, it’s typical to end up having to handle the 123.287 files of results. At some point you may want to move some of them, for instance, those that have a certain pattern in the name, such as val2014.

In a Linux environment and with mat files as results, you would typically do something like:

mv *val2014*.mat destination

This worked flawlessly in Pascal, but in COCO you will get something like:

Argument list too long

What’s the problem? The number of elements that the shell can handle in the arguments is limited, and we may be trying to move a larger number of files. Luckily, we can find a workaround by using xargs:

ls | grep  val2014 | xargs mv -t destination

The ls command lists all files and grep filters the ones we want to move. Then, the list passes to xargs, which executes the mv command once per member of the list. The -t option (Note that the -t option works only on GNU systems) allows to exchange the order of the parameters of mv. More info here.