Clicky

edm81363
Hi there -

First off, thanks for making such a very helpful tool. Here's the short story;

  • I shoot with a Canon 7D in RAW
  • Camera and GPS are both set to GMT (I work in scientific / forensic fields)
  • Files are transferred from camera card to protected storage with an MD5 checksum verification
  • Files are renamed to include ISO8601 compliant timestamps in the actual file names, and these are generated from the file creation date. 
  • Files are geotagged in HoudahGeo as shown in the attached file "HOUDAH"
  • EXIFs are exported using the "Timestamp Settings" (see attached file).
  • When viewed in Finder, some of the files have had their timestamps changed to the time that they were geotagged, NOT created. See the attached BEFORE and AFTER files.
I suspect that there may be some issue with the fact that, in this case, the timestamp actually occurs in the future. I will run the tests again once that is not the case.

UPDATE: Running the tests after the GMT time has passed locally eliminates the issue. It appears that processing the images before the 8 hour time differential has passed causes HoudahGeo to change the file creation date on some files. 

(All files are correctly located in HoudahGeo, and the GPS was running continually during the field operation.)

FWIW, these files were created solely as a proof-of-concept for this new workflow. I'm open to alternative approaches, but it is important that the absolute file creation timestamp be retained, in both the filename and file timestamp. And the file needs to have GPS information recorded in it.

I think another option for me would be if HoudahGeo had an option to ONLY generate XMP sidecars. I have tested this workflow and I believe it will work. Exporting and then deleting undesired RAW files consumes a lot of hard drive space and takes a long time. (There are projects where we generate a LOT of images.)

This particular system I am designing needs to be as automated and reproducible as possible. It must also withstand vigorous review in court or in the scientific community.

Thanks in advance for your assistance.

Kind Regards

Ed

<edited to correct typo ISO8001 = ISO8601) Click image for larger version - Name: AFTER.png, Views: 16, Size: 30.76 KB Click image for larger version - Name: BEFORE.png, Views: 21, Size: 29.49 KB Click image for larger version - Name: Timestamp_Settings.png, Views: 21, Size: 48.24 KB Click image for larger version - Name: HOUDAH.png, Views: 16, Size: 37.90 KB
0 0
houdah
Hi!

I am sorry I did not see your message earlier.

Your workflow is sound.

By default, HoudahGeo tries not to modify the file modification date.
There however is a hidden option for the modification date to be updated during EXIF export.
This was requested by some customers.
This is off by default, and needs to be enabled at the command line in Terminal.app.

The "Timestamp" option in EXIF/XMP export settings refers to the date tags in EXIF/XMP.
Images taken by digital cameras usually have date and time tags, but lack time zone information.
With the "Timestamp" option checked, HoudahGeo would add the time zone information.
If would also correct the "clock error" you specified upon import.

During export you could check "Always write XMP sidecars". HoudahGeo then creates missing sidecar files.
It will write to the XMP sidecars and leave the actual image files untouched.

Best,
Pierre Bernard
Houdah Software s.à r.l.



Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
petrie
Hi Pierre,

I am a new to Houdah Geo and I am just trying to geotag my first images.

Like the TO I am wondering why approx. 1/3 of my files have the modification date changed, while for the rest of the files the modification date remains the same as the creation date.

I am actually tagging JPG and RAW couples, sometimes only one of them gets a changed modifications date, sometimes both, sometimes none of them. That is really confusing me. ;-)  

What I want is to have both, the creation date and the modification date, untouched. Is this possible somehow?

You wrote "By default, HoudahGeo tries not to modify the file modification date." - Could you please explain what you mean by "try" and why this seems to fail sometimes? 

Your help is very appreciated, thank you very much in advance.

Regards,
- petrie


0 0
houdah
Hi!

There briefly was a bug where date preservation did not work. That was fixed.

There are error cases where HoudahGeo needs to create copies of the image. E.g. when the image has inconsistent tag values.
If that is the problem, you should see it happen again if you tag the same unmodified file again.

Can you send me a file with which the problem can be reproduced?

Best,
Pierre Bernard
Houdah Software s.à r.l.
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
petrie
Hi Pierre,

I did some testing today. Here are the results.  :-)

1st test - Processing of 4 RAW/JPG couples (Timestamp correction + geotagging):
I repeated the test 8 times. 
Result: everything fine

2nd test - Processing the whole bunch of JPGs (261 pics, - timestamp correction only):
I repeated this test 4 times.
Result: no problems again. creation and modification date are not altered.

3rd test - Processing all 261 pics again (timestamp correction + geotagging):
I repeated this test 6 times.
     1. run: 214 okay / 47 not okay
     2. run: 215 okay / 46 not okay
     3. run: 261 okay / 0 not okay
     4. run: 261 okay / 0 not okay
     5. run: 239 okay / 22 not okay
     6. run: 261 okay / 0 not okay
Huh? ;-)

The pictures where preserving the original timestamp failed are completely different from each other. But it seems to me as if "it" only happens in the second half of the tagging process, not right at the beginning of it. - However the pictures are tagged correctly.

For this reason I think it does not have to do with the picture itself. Nevertheless, I put several results of one picture into a ZIP archive: "_3HH3039.jpg". You will find the ..._0.jpg version, - that is the original file out of the cam. The other versions are the corresponding results of the tests as mentioned above: No. 1,2,5: not okay, no. 3,4: okay.
You can download the ZIP file from my dropbox: https://www.dropbox.com/s/ev0geuxz2vgvdn2/_3HH3039.zip (48MB).

If you have further questions or should you need some other examples, please let me know. :-)
Thank you very much for your help!


Regards,
- petrie


0 0
houdah
Hi Petrie,

I have tried tagging your image files repeatedly. Timestamps never changed.

Do you have the file numbers right?
I see 2,3,4 and 6 having the same modification date as 1.
Not ok: 1 and 5

If you do not use "Spotlight comments", try disabling that option during export.
I did not see it make any differences, but it could. Writing those is handled by the Finder.

HoudahGeo relies on exiftool to write EXIF/XMP/IPTC tags. This uses temporary files to preserve modification dates.
I wonder if there could be anything interfering with that.
Since the problem occurs randomly, I could imagine some other tool running in parallel and on occasion getting to the temporary files before exiftool is done.
Is there any tool that watches the folder to create backups, sync to DropBox, …?

Best,
Pierre Bernard
Houdah Software s.à r.l.
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
petrie
Hi Pierre,

that´s odd...

I did a lot of testing again and I think, I can now say that this only happens when a large number of files is processed. If it´s only up to 300-400 pictures chances are good that everything will work fine. 

When tagging files stored on my internal SSD it happens later, if they are on my external HDD (FireWire) it happens sooner.

I turned every unused process off that might possibly be interfering with the tagging process, but the effect is the same. I am very clueless, now. :-/


Do you have any other idea?

Regards,
Helge


PS. Concerning the file numbers you are absolutely right. Sorry, I must have been confusing that... ;-)


0 0
houdah
Hi Helge,

I still have no idea what could cause this.

I wonder if the speed of the drives and the number of files in the export could be triggering this.

Depending on the number of CPU cores in your Mac, HoudahGeo may try to process up to 16 files simultaneously.
Having many more images queued up after those 16 should make no difference at all.

I will test with a batch of 300 images using different drives.

BTW are all images located in the same directory? Are there images with the "same name"? E.g. IMG1.jpg as well as IMG1.cr2 

Best,
Pierre Bernard
Houdah Software s.à r.l.
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
edm81363
Hi -

First of all, I have not processed images since the 2013 work season, so my experiences are NOT with the latest batch of software.

However, I ran into this issue when processing lots of images. It is possible for me to shoot 1,500 images, and then process them all at once. Sometimes I am using fast storage (four disk RAID 4 using eSATA), but sometimes they are on a single external USB drive. I have not tried to track the problem to which storage medium I was using, but I will in the future.

My next project starts April 9, so hopefully I will have more to report soon.

My workflow has changed slightly, as my computer is now also set to GMT timezone, so that will eliminate the "images shot in the future" aspect I had originally reported.

-Ed

(I use HoudahGeo to geotag images taken on research vessels by using the ships GPS log in NMEA format.)
0 0
houdah
Hi!

I was able to recreate the problem here by tagging 300+ images stored on a SSD.
Just as I was getting convinced this could be triggered by either the fast drive or the number of images, tagging a single file had its modification date changed. Same file had just been tagged 100+ times with no ill effect.

Still working on this.

Best,
Pierre Bernard
Houdah Software s.à r.l.
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
petrie
Hi Pierre,

sorry for my late answer.

"BTW are all images located in the same directory? Are there images with the "same name"? E.g. IMG1.jpg as well as IMG1.cr2?"

Yes. For each run every image file was located in its own directory. Under my photo folder I created 10 subfolders 1,2,3,... and I copied my image files into each of them, in order to always have a new set of images to be tagged.

At first I had RAW and JPG couples, for the following tests I only used the JPG files.
0 0
petrie
Hi Ed,

"However, I ran into this issue when processing lots of images."
Yes, I can confirm that. Okay be honest I just did 5 cycles with only a few number of images. The rest of my tests were much more images.

"I have not tried to track the problem to which storage medium I was using, but I will in the future." 
Looking forward to hearing from you, - and sorry for using your thread. Hope that is okay. ;-)

Regards,
Helge
0 0
petrie
Hi Pierre,

"I was able to recreate the problem here by tagging 300+ images stored on a SSD."
That´s good news. Well, for me.  ;-)

In my tests it was only slightly dependent on which kind of storage the image files were stored. In my tests it happened more often if the image files were stored on a external HD connected via FW. But with the internal SSD it also happened. 
I usually use GeoSetter (but it has far too many settings for me) and for testing purposes I repeated my tests with a Windows VM running with Parallels. In fact the image files are on the same storage. I ran the test a couple of times, but everything was okay.

This program also relies on ExifTool. Maybe the Mac implementation of ExifTool behaves different than the Windows version?

"Just as I was getting convinced this could be triggered by either the fast drive or the number of images, tagging a single file had its modification date changed. Same file had just been tagged 100+ times with no ill effect."
Although tagging only a few files always worked for me, I can confirm that "it" happens randomly. Files that worked before, suddenly were faulty, and vice versa.

"Still working on this."
Thanks a lot, your help is very welcome!

Regards,
Helge


/EDIT: Fixed some typos. One shouldn´t write Emails before going to work.  ;-)
0 0
houdah
Hi all,

I am still looking into this.

HoudahGeo calls upon exiftool to write the metadata. exiftool takes great care to preserve or restore the file modification dates.
I added code to HoudahGeo to check the dates before and after the update. At that point dates are still the same.

It seems like the change in modification dates is triggered by the tagging, but actually happing later.

Best,
Pierre Bernard
Houdah Software s.à r.l.
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
petrie
Hi Pierre,

there is no rush!

When you have new findings about this, please let me know if I can help you by re-running the tests with another (alpha/beta) version of HoudahGeo.

Best regards,
Helge
0 0
houdah
Hi Ed,
Hi Helge,

This looks more and more like an OS bug.
The dates get changed some 10 to 20 seconds after HoudahGeo is done with the file.
The delay seems to get longer if the system is busy.

I am not sure how to report such a randomly occurring bug to Apple. But I'll try.
I am also not sure yet how to work around the bug.

Currently HoudahGeo has exiftool modify the existing file in place. I.e. exiftool modifies the file and then restores the modification date. All other metadata (e.g. labels and tags) are left untouched by the process. With this procedure, the result looks fine for 10 seconds. Then the OS updates the modification date again.

An alternative procedure is to create a new file during tagging; copy modification dates and metadata from the original file to the new one; then delete the original file. This should for most purposes look the same.

I'll probably build a beta with that alternative approach.

Best,
Pierre Bernard
Houdah Software s.à r.l.






Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
houdah
Hi!

I have an experimental beta of HoudahGeo:

https://dl.dropboxusercontent.com/u/2381634/HoudahGeo/HoudahGeo3.6.5b1.zip

Rather than modifying the exiting images files, this version replaces the images files with copies and then restores the original modification dates to these copies.
I found this procedure to work reliably.

Best,
Pierre Bernard
Houdah Software s.à r.l.
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0