Guidance for massive batch processing

Announcement of new releases, bugs, support, suggestions
Post Reply
sheimend
Posts: 9
Joined: 17.07.2016 23:21

Guidance for massive batch processing

Post by sheimend » 17.07.2016 23:35

I love that Helicon has a Batch process feature, but I may be asking a little too much of it and I'm experiencing some issues in return.

I'm attempting to batch-stack 1500 folders, each of which contain 6 images to stack. In theory, the batch UI should be able to do this. In practice, here are the issues I'm encountering.

1) On one of my machines, the UI hangs and never recovers when I try to add these folders to the batch stack list. On another machine, the UI freezes up for a minute but finally recovers. If I divide the folders into sets of, say, 200, I can get it to work. But that kinda sucks.

2) Helicon keeps filling my hard drive and then failing. It looks like the cache is only disposed when Helicon is closed - not once a stack is complete. As a result, I ended up generating a few hundred GB in cached files, filled up c:\ and stopped. Unfortunately, the UI doesn't provide a good way to recover the stacks that failed. I have to figure out the last stacked image I got and go reload the job starting from there.

3) The Multiple Folder picker UI lets me select all (Ctrl + a) or lets me click-select multiple folders, but it doesn't let me shift-select a continuous range of folders. So, when I discover that folders 1-148 out of 1500 stacked correctly, I don't have an easy way to go select 149-1500 without changing my folder structure on the disk.

4) The job progress window is pretty bad. By default, it scrolls to the bottom, but the first job to run is at the top. It doesn't provide any overall progress bar of (Job 18 of 1500), time elapsed, or time remaining. If you were batch stacking 4 folders, this wouldn't be a problem. But, since my stacking jobs will routinely run for 8 hours or more, it'd be a nice feature to get some indication of time remaining.

Again, I'd assume I'm pushing the batch stacking functionality way past its intended use, but hopefully the feedback is helpful.

Ideally, I'd love to run batch stacking through a command-line interface that I can control via PowerShell or some other scripting interface. That way, I can control my workflow on top of the stacking process to, for example, automatically generate stacks with different stacking methods and file output types, automatically start stacking watch folders, notify me on errors, etc.

By any chance, does Helicon offer a type of command line interface like what I described?

Thanks in advance! Every time I use your software I'm still amazed by the magic of focus stacking! (and THANK YOU for DNG output!)

User avatar
Stas Yatsenko
Posts: 3850
Joined: 06.05.2009 14:05
Contact:

Re: Guidance for massive batch processing

Post by Stas Yatsenko » 18.07.2016 13:50

Thanks for the feedback. All good points.

0) It was intended for arbitrary number of stacks, but it wasn't tested properly, especially with relation to disk cache.

1) It scans the metadata of all the images when you add them. Most likely it didn't hang entirely; it was working and would un-freeze when done. Having said that, this shouldn't happen and we'll look into this.

2) Yeah, that's a major problem and we're already working on it.

3) I'll take a look, but most likely there's no way for us to fix this because we didn't implement the folder picker. We simply use it as is.

4) That's a consequese of trying to use the same main UI for batch mode without any significant changes from single-stack mode. On that note, I can think of one or two non-invasive ways to add a global progress indicator.

There is a command line interface but only for single-stack processing, not for batch: http://www.heliconsoft.com/focus/help/e ... F_COM_LINE

We might add an advanced batch processing command line support later, but for now you can write a script to iterate over a set of folders and invoke Focus for each of them one by one.

sheimend
Posts: 9
Joined: 17.07.2016 23:21

Re: Guidance for massive batch processing

Post by sheimend » 20.07.2016 22:52

Thank you so much for the feedback and guidance! I'm thrilled to see that these issues are on your radar already.

It looks like the command line path could be a great solution for me. However, I don't see tips on syntax for saving as DNG. Is this supported from the command utility and, if so, can you help with the syntax?

I'm cooking up an idea for parallel processing groups of stacks with multiple instances of Helicon running in the background - based on my testing, it's much faster than serial processing. If it works, I'll post the PowerShell script.

Thanks again! I have so much stacking in my workflow right now that any improvements to these scenarios make a huge impact on my work (and life!!).

Warmest regards,
Scott

User avatar
Stas Yatsenko
Posts: 3850
Joined: 06.05.2009 14:05
Contact:

Re: Guidance for massive batch processing

Post by Stas Yatsenko » 21.07.2016 15:50

Are you on Windows or Mac? We've fixed 3) and improved 1) (it will still freeze while scanning the file system for images, but now it completes 3-4 times as fast). We'd like to send you an updated Helicon Focus with these fixes.

Your performance research is very curious. Did you use Helicon Focus 6.7.x? If you did, it's extremely odd that multiple instances were more effective than one. That was the case until 6.7.0, but not anymore (or at least it shouldn't be).

You can indeed save to DNG in command line mode, just specify an output file name with .dng extension.

sheimend
Posts: 9
Joined: 17.07.2016 23:21

Re: Guidance for massive batch processing

Post by sheimend » 26.07.2016 01:12

Awesome! I'm on Windows and would love to try a version with the update.

Thanks for the tip on .DNG!

My research with performance so far has been limited to a few trials and some unscientific time measurement... but I'll make more formal performance recordings with my next batch-process and send your way.

Thanks again, such excellent support!

Post Reply