Page 1 of 1

Bug/mistake/missing features in batch processing

Posted: 27.10.2017 16:06
by sbuerger
Hi Support (meaning Hi Stas, of course),
now that I finally find the time...
I found three quite annoying issues trying to use batch processing with HF 6.7.1 (also tried 6.2.2, same results) on Macintosh, making this (originally neat) feature useless in many cases:

First thing: There is some strange behavior regarding the order* (ascending/descending) of the pictures processed:
When configuring a batch there is no way to choose which order to use - ok, but with or without user interaction, the application obviously will have to make a "decision" about this. So I would assume that the application uses the current setting chosen in the main window's dropdown menu - or, at least, the same setting for all stacks in the current batch. Unfortunately, neither is the case: No matter what is chosen in the dropdown, MOST stacks are performed in a descending order (starting with last photo in the stack, ending with first photo) while SOME are performed in an ascending order. This seems to happen randomly, approximately in an 80/20 relation (about 80% descending, 20% ascending).
My usual workflow (as I mainly shoot micropanos and VR/MoCo objects) is to shoot a series of stacks, let's say 20 stacks of 50 photos each, 1000 photos in total. Then I put all of them into one folder, load that folder into batch processing and then use the "Split stack" (by picture count) feature to divide the 1000 photo bunch back into 20 Stacks. When I push the "Render" button, HF correctly generates the 20 stacks in the output list of the main window and starts rendering the first job in the list. If I click on one of the outputs that has not been processed yet, the source file list shows the pictures in ascending order (which is what is also set in the dropdown), but as soon as processing starts, the order is - mostly - reversed and rendering is done in descending order. Mostly, but not always - for some reason I can't find, some stacks are not reversed and are rendered ascending.
As a partial workaround I have set the file name template to [SOURCE_FILE_NAME]([PARAMS]), so I can easily distinguish ascending and descending stacks and repeat the erroneous ones manually, but this is time consuming unnecessary work. Plus, when doing slabs (sub-stacks), I depend on using an ascending order, so I can't use batch processing at all.
I think there should be some possibility to set the order in the batch processing dialog.

Second thing: Currently, there is no way to select (and thereby, to use) a dust map in batch processing (at least I did not find any). Combined with the above issue this makes batch processing somewhat "inferior focus stacking" and unusable in many cases.

Third thing: This does not only affect the batch mode, but typically shows critical effects especially when batch rendering and therefore should be avoided in that case: HF caches a lot of temporary data when working on a stack (especially in RAW/DNG workflows when every photo is TIFFed first). There is no reserve in disk space - if HF needs that amount of memory, it will eat up available space up to the last byte and then stagnate. All the temporary files of an HF session will be preserved until HF is quit; even when a stack is deleted from the output list and is thereby unavailable for further work, its temp files are preserved unless you quit HF. While this hardly causes problems in normal operation, it most likely does in batch mode as the cache folder usually resides on the startup disc (of course this makes sense as the startup disc usually is the computer's fastest mass storage - but unfortunately also the smallest one), and if the startup disc is filled up to the edge - well, troubles may vary, but one can be sure to be in trouble then.
I think that three actions should be taken on this issue:
1. When a stack is deleted from the output list, it's temp files should be deleted as well.
2. If possible, a disk reserve should be defined, keeping HF from using every bit of disk space.
3. Some strategy should be applied to take away the limits from batch processing - for example, automatically delete outputs and their corresponding temp files after successfully saving the resulting stacked picture. Or, as a luxury variant, give the user the choice about whether outputs are to be deleted automatically or not.

What do you think about these issues? I have no idea about the programming efforts needed to follow my suggestions... but fact is that currently, at least for the Mac platform, we have nothing that we can call an adequate batch processing.

Kind regards

*One maybe important annotation: I use HF in a german language version, and unfortunately i could not find a way to change the language inside the application. So I have to translate the german vocabulary used in the GUI back to what I _assume_ is in the english version. Errors may occur, but I hope to understandably express what I mean.

Re: Bug/mistake/missing features in batch processing

Posted: 27.10.2017 18:13
by sbuerger
Oops, didn't notice 6.8.0 is out already - got no notification from HF, either...
Nevertheless, just downloaded the new version and did a few quick tests. So, here's what I found:

Issue #1:
Stacks are still rendered in descending order, no way to influence this. But as yet, random appearance of ascending stacks has not occured in the about 30 stacks I did for a test.
Still, I would prefer to be able to set the order as needed.

Issue #2:
No changes in 6.8.0.

Issue #3:
Suggestion 1: practically solved - nearly all temp files are deleted with the output, only the depth map remains for whatever reason.
Suggestion 2: Couldn't test this (would take hours and be likely to freeze my computer).
Suggestion 3: No changes in 6.8.0.

So, most of my issues are still valid, but fortunately development seems to move in the right direction. ;)

Kind regards

Re: Bug/mistake/missing features in batch processing

Posted: 31.10.2017 17:18
by Stas Yatsenko
Hi Stephan,
Thanks for taking the time to write these detailed observations and suggestions.

#1: That's not how it's supposed to work. It should respect the currently selected (meaning, last used) option, be it ascending, descending, or automatic. We'll check it.

#2: Noted, we'll have to add that feature.

#3: We're aware this is an issue, especially for batch processing. We're aiming for a disk quota as opposed to an unlimited cache. It's not trivial to implement and requires significant internal changes in the program, but we're already working on it right now.

Re: Bug/mistake/missing features in batch processing

Posted: 02.11.2017 19:11
by sbuerger
Hi Stas,
that sounds great. I've been a HF user for about ten years now, and in that time I have seen a lot of useful features being added which today I wonder how I could ever live without. So, hopefully, the series will continue... ;)

#1: Is the user selection of stacking order respected correctly in the Windows version? This would offer a workaround (though Apple's denial to support NTFS and Microsoft's ignorance of HFS make file transport a little uncomfortable).

#2: Thanks a lot!

#3: A disk quota is basically a good idea anyway as it would prevent the computer from freezing in general, but what if the quota is used up? To prevent HF from getting stuck there would have to be some strategy to (manually/automatically/selectively) delete temp files that are not needed anymore. For my opinion, the most important step to that behalf is already done in HF 6.8.0, since in most cases (with batch processing as the only exception) I would definitely prefer to select and delete manually, thus keeping the ability to preserve stacks of my choice for comparison.
So, as long as I don't use the batch mode, things are fine for me now. Only when rendering a batch of stacks the problem comes up that, while HF is rendering a stack (especially when using RAW-to-DNG workflow), it hardly ever reacts to user (mouse) input. Meaning, I virtually don't get a chance to manually delete the first stack's temp files until the last stack is finished.
This issue could be solved by a pref setting like "Only keep the last XX stacks" or "Delete oldest stacks when disk quota is reached", however, I have no idea of how tough it may be to implement such a mechanism. Maybe it would be easier to do something completely different first: "just" put a pause/resume toggle button beside the render button in the main window (ok, I know there's much more to this than just putting a button into the GUI, but you know what I mean... ;) ). I neither have any idea if this would be any simpler or even harder, but if it would be possible, it would at the same time solve the problem (well, not really a problem, but sometimes a good occasion to be annoyed about one's own confusion) that, once rendering a stack is started, there is no way to cancel that but kill the application.

However, I think that issue 3 is least important as the universal solution to all related problems is obtaining additional disc space, which is as cheap as I would encourage you to prioritize #1 and 2. No need to mention that you can always count on me as a beta tester if needed.

Kind regards

Re: Bug/mistake/missing features in batch processing

Posted: 03.11.2017 11:44
by Stas Yatsenko
#1: No, it doesn't seem to work anywhere at the moment, we'll fix that.

#3: The process won't get stuck on behalf of the cache being full because the cache only holds the internal data that's used to speed up processing (this data can be re-generated, doing that simply takes time). The stacking results are not placed in the cache, hence they are not included into the quota. If your storage cannot hold all the results for the batch job that you're running, well, life is tough. You'll need to specify (or buy) another storage drive. The situation in which the result cannot be saved (a single image per stack, typically) doesn't sound normal.

We're also looking into the problem of unresponsive UI while rendering is in progress.