Bug/mistake/missing features in batch processing
Posted: 27.10.2017 16:06
Hi Support (meaning Hi Stas, of course),
now that I finally find the time...
I found three quite annoying issues trying to use batch processing with HF 6.7.1 (also tried 6.2.2, same results) on Macintosh, making this (originally neat) feature useless in many cases:
First thing: There is some strange behavior regarding the order* (ascending/descending) of the pictures processed:
When configuring a batch there is no way to choose which order to use - ok, but with or without user interaction, the application obviously will have to make a "decision" about this. So I would assume that the application uses the current setting chosen in the main window's dropdown menu - or, at least, the same setting for all stacks in the current batch. Unfortunately, neither is the case: No matter what is chosen in the dropdown, MOST stacks are performed in a descending order (starting with last photo in the stack, ending with first photo) while SOME are performed in an ascending order. This seems to happen randomly, approximately in an 80/20 relation (about 80% descending, 20% ascending).
My usual workflow (as I mainly shoot micropanos and VR/MoCo objects) is to shoot a series of stacks, let's say 20 stacks of 50 photos each, 1000 photos in total. Then I put all of them into one folder, load that folder into batch processing and then use the "Split stack" (by picture count) feature to divide the 1000 photo bunch back into 20 Stacks. When I push the "Render" button, HF correctly generates the 20 stacks in the output list of the main window and starts rendering the first job in the list. If I click on one of the outputs that has not been processed yet, the source file list shows the pictures in ascending order (which is what is also set in the dropdown), but as soon as processing starts, the order is - mostly - reversed and rendering is done in descending order. Mostly, but not always - for some reason I can't find, some stacks are not reversed and are rendered ascending.
As a partial workaround I have set the file name template to [SOURCE_FILE_NAME]([PARAMS]), so I can easily distinguish ascending and descending stacks and repeat the erroneous ones manually, but this is time consuming unnecessary work. Plus, when doing slabs (sub-stacks), I depend on using an ascending order, so I can't use batch processing at all.
I think there should be some possibility to set the order in the batch processing dialog.
Second thing: Currently, there is no way to select (and thereby, to use) a dust map in batch processing (at least I did not find any). Combined with the above issue this makes batch processing somewhat "inferior focus stacking" and unusable in many cases.
Third thing: This does not only affect the batch mode, but typically shows critical effects especially when batch rendering and therefore should be avoided in that case: HF caches a lot of temporary data when working on a stack (especially in RAW/DNG workflows when every photo is TIFFed first). There is no reserve in disk space - if HF needs that amount of memory, it will eat up available space up to the last byte and then stagnate. All the temporary files of an HF session will be preserved until HF is quit; even when a stack is deleted from the output list and is thereby unavailable for further work, its temp files are preserved unless you quit HF. While this hardly causes problems in normal operation, it most likely does in batch mode as the cache folder usually resides on the startup disc (of course this makes sense as the startup disc usually is the computer's fastest mass storage - but unfortunately also the smallest one), and if the startup disc is filled up to the edge - well, troubles may vary, but one can be sure to be in trouble then.
I think that three actions should be taken on this issue:
1. When a stack is deleted from the output list, it's temp files should be deleted as well.
2. If possible, a disk reserve should be defined, keeping HF from using every bit of disk space.
3. Some strategy should be applied to take away the limits from batch processing - for example, automatically delete outputs and their corresponding temp files after successfully saving the resulting stacked picture. Or, as a luxury variant, give the user the choice about whether outputs are to be deleted automatically or not.
What do you think about these issues? I have no idea about the programming efforts needed to follow my suggestions... but fact is that currently, at least for the Mac platform, we have nothing that we can call an adequate batch processing.
Kind regards
Stephan
*One maybe important annotation: I use HF in a german language version, and unfortunately i could not find a way to change the language inside the application. So I have to translate the german vocabulary used in the GUI back to what I _assume_ is in the english version. Errors may occur, but I hope to understandably express what I mean.
now that I finally find the time...
I found three quite annoying issues trying to use batch processing with HF 6.7.1 (also tried 6.2.2, same results) on Macintosh, making this (originally neat) feature useless in many cases:
First thing: There is some strange behavior regarding the order* (ascending/descending) of the pictures processed:
When configuring a batch there is no way to choose which order to use - ok, but with or without user interaction, the application obviously will have to make a "decision" about this. So I would assume that the application uses the current setting chosen in the main window's dropdown menu - or, at least, the same setting for all stacks in the current batch. Unfortunately, neither is the case: No matter what is chosen in the dropdown, MOST stacks are performed in a descending order (starting with last photo in the stack, ending with first photo) while SOME are performed in an ascending order. This seems to happen randomly, approximately in an 80/20 relation (about 80% descending, 20% ascending).
My usual workflow (as I mainly shoot micropanos and VR/MoCo objects) is to shoot a series of stacks, let's say 20 stacks of 50 photos each, 1000 photos in total. Then I put all of them into one folder, load that folder into batch processing and then use the "Split stack" (by picture count) feature to divide the 1000 photo bunch back into 20 Stacks. When I push the "Render" button, HF correctly generates the 20 stacks in the output list of the main window and starts rendering the first job in the list. If I click on one of the outputs that has not been processed yet, the source file list shows the pictures in ascending order (which is what is also set in the dropdown), but as soon as processing starts, the order is - mostly - reversed and rendering is done in descending order. Mostly, but not always - for some reason I can't find, some stacks are not reversed and are rendered ascending.
As a partial workaround I have set the file name template to [SOURCE_FILE_NAME]([PARAMS]), so I can easily distinguish ascending and descending stacks and repeat the erroneous ones manually, but this is time consuming unnecessary work. Plus, when doing slabs (sub-stacks), I depend on using an ascending order, so I can't use batch processing at all.
I think there should be some possibility to set the order in the batch processing dialog.
Second thing: Currently, there is no way to select (and thereby, to use) a dust map in batch processing (at least I did not find any). Combined with the above issue this makes batch processing somewhat "inferior focus stacking" and unusable in many cases.
Third thing: This does not only affect the batch mode, but typically shows critical effects especially when batch rendering and therefore should be avoided in that case: HF caches a lot of temporary data when working on a stack (especially in RAW/DNG workflows when every photo is TIFFed first). There is no reserve in disk space - if HF needs that amount of memory, it will eat up available space up to the last byte and then stagnate. All the temporary files of an HF session will be preserved until HF is quit; even when a stack is deleted from the output list and is thereby unavailable for further work, its temp files are preserved unless you quit HF. While this hardly causes problems in normal operation, it most likely does in batch mode as the cache folder usually resides on the startup disc (of course this makes sense as the startup disc usually is the computer's fastest mass storage - but unfortunately also the smallest one), and if the startup disc is filled up to the edge - well, troubles may vary, but one can be sure to be in trouble then.
I think that three actions should be taken on this issue:
1. When a stack is deleted from the output list, it's temp files should be deleted as well.
2. If possible, a disk reserve should be defined, keeping HF from using every bit of disk space.
3. Some strategy should be applied to take away the limits from batch processing - for example, automatically delete outputs and their corresponding temp files after successfully saving the resulting stacked picture. Or, as a luxury variant, give the user the choice about whether outputs are to be deleted automatically or not.
What do you think about these issues? I have no idea about the programming efforts needed to follow my suggestions... but fact is that currently, at least for the Mac platform, we have nothing that we can call an adequate batch processing.
Kind regards
Stephan
*One maybe important annotation: I use HF in a german language version, and unfortunately i could not find a way to change the language inside the application. So I have to translate the german vocabulary used in the GUI back to what I _assume_ is in the english version. Errors may occur, but I hope to understandably express what I mean.