Clicky

PhilM
On macOS Catalina, HoudahSpot will thus no longer be able to find your Apple Mail messages.
What exactly does this mean for HS? Is this going to be permanent? Can we have some more detail please?
0 0
houdah
On macOS Catalina, third party applications no longer have access to the search index for Apple Mail messages. Even after you grant "Full Disk Access", HoudahSpot will not be able to find Mail.app messages.

It currently appears this is an intentional change by Apple. One that, as far as I can tell, makes little sense. There is no security / privacy benefit. Users can still grant third-party applications access to the files holding mail messages. While searching these is much slower than searching the Spotlight index, malicious apps could go about there business of extracting personal information. 

What does make the change seem unintentional is the fact that it does not seem to be in line with Apple's current efforts at privacy. These all defer to user consent. Private data is fenced off until the user explicitly grants access. I.e. when you want an application to work with your photos, you can grant that application access and get the task done. In the case of Mail, it appears that Apple has made the decision to block access without leaving the customer an option to override and allow third-party searches, automation, or integration with other tools.

We have reported this as a bug to Apple via https://feedbackassistant.apple.com
You can do the same or contact apple via https://www.apple.com/feedback/
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
PhilM
 That's helpful. So by the sounds of it, with enough pressure and/or time, Apple may well reverse this and allow the user to override. Am I right?
0 0
gilby
I have started to experiment (on my Catalina Mac) with a one-way synchronisation from ~/Library/Mail to a folder somewhere else - right now ~/Temp/Mail.  I am using ChronoSync to do this daily.  The emails then get indexed by Spotlight as files.  So HoudahSpot can see them.  I have changed my HS templates which include email.  This works, though it is not ideal.

I do wonder how long it will be before emails are stored in a database (likes Notes are) and not as files.  I hope never.
0 0
houdah
One problem with this solution is wasted disk space. Mail has the files and the index. Duplicating the messages may waste disk space. APFS might be able to avoid that. Duplicating the index will waste space.

It is not a good solution for application developers to pick up. Imagine several applications doing this. Your Mac would be kept busy duplicating and indexing mail messages.

Yet, I imagine power users and developers will take this route if Apple leaves no other choice. Then Apple would have achieved the exact opposite of what they aim for. Mail messages with potentially sensitive personal information would no longer be protected by "Full Disk Access" setting. Instead, the files would be spread all over the place for every malicious application to pick up.

I hope Apple changes course before the public release of Catalina.

Mail messages moving to a database would not be all that bad, provided that Apple allows third parties to search the "Core Spotlight" index. This is where Notes and now Mail messages are indexed.

Currently "Core Spotlight" is off-limits to HoudahSpot for 2 reasons:

- Apple simply does not allow third-party applications to search "Core Spotlight"
- Items in the "Core Spotlight" index do not necessarily map to actual files. E.g. Apple Notes are entries in a database. HoudahSpot, however, is designed to work with files (show a path, copy, move, rename, tag, …). This is not a problem with Mail messages on Catalina: those still exist as files.
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
chazzo
I've filed a feedback request with Apple and urge all HoudahSpot users to do the same, as long as you think it would be relatively easy to change this behaviour in Catalina. I find Mail's search functionality confusing and hard to use, and rely on HoudahSpot for anything remotely complicated.
0 0
robgendreau
Ugh. One of the major reasons I bought HS was to search Mail. It's own searching is lame, and indeed has often been broken on my machine.

For other reasons I have already done some email copying, although I switched to Email Archiver Pro a while ago. Basically it batch prints to PDF, and cleverly embeds the email metadata as tiny text (for various legal reasons I need something closer to hard copy email). I can still use HS to search that, but then replying, etc is a pain.

And I've used other applications that will be affected as well, like Leap, also by Ironic Software. It's great because it can be used like HS to say find both PDFs, Word docs, and emails with say the same person's name in them for quick work on a project you're discussing. Why Apple would abandon this utility makes no sense.

But a question: can HS still search the applications that can import and store mail messages? like EagleFiler or Devonthink? Devonthink indexes some files optionally (leaves them in places and hence sees changes), but I don't think it can index in the Mail folder without maybe a permission change (on my laptop I get that red dot with a minus indicating no access; is there a way to change that?).
0 0
houdah
I have yet to install DEVONthink on Catalina. I do expect it to work just the same as on Mojave.

DEVONthink supports Spotlight indexing and HoudahSpot can search mail messages imported into DEVONthink.
I don't think DEVONthink extends Spotlight indexing to items it has not moved to its own catalog.

While you can find archived messages in DEVONthink using HoudahSpot, there are 2 important restrictions:
- Quick Look preview will not be able to show you the contents of the message. We will try to work around that limitation. Text Preview works fine.
- Replying to a message will take extra steps. Opening the result from HoudahSpot will take you to DEVONthink, not Apple Mail
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
CraigP
If a third-party email application (Outlook, Newton, Spark, etc.) is used, does this override the mail.app issues?  If not, how do these applications perform email searches?
0 0
houdah
The Apple Mail application saves its messages as .emlx files in the ~/Library/Mail folder. These files are picked up by Spotlight, run through an importer plug-in that can extract metadata and text from .emlx files, this is saved to the Spotlight index where HoudahSpot can find it. Starting with macOS Mojave, the ~/Library/Mail folder is protected. You need to grant HoudahSpot "Full Disk Access" permission for it to be able to find files from that folder and show these to you.

In macOS 10.15 Catalina, the indexing of Apple Mail messages has move to a newer area of Spotlight: Core Spotlight. Here third-party applications can only search for data they submitted themselves. I.e. HoudahSpot cannot go and search for data submitted by Apple Mail. Only Apple's own Spotlight window is allowed to access data submitted by all applications.

Third-party mail applications will not be affected in the same way as Apple Mail. That does not necessarily mean that you will be able to find messages using HoudahSpot. For HoudahSpot to be able to find messages owned by a third-party mail application:
  • the application needs to save messages as individual files. i.e. one file per message
  • the application needs to allow Spotlight indexing (not Core Spotlight)
  • the application needs to provide a Spotlight importer plug-in to read its proprietary file format
Last I checked, this applied to Outlook and MailMate. It does not apply to AirMail. I don't know about Newton nor Spark.
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
mny212a
Is there a way to replace the ~/Library/Mail folder tree with a symbolic link or hard link, and then, move the actual ~/Library/Mail contents to a directory that Spotlight will fully index such as ~/Documents, so Houdah can access it?

I've tried MailMate and Outlook -- HoudahSpot seems to works well with both, but there are other complications that make both inferior to Apple Mail + HS for my purposes. (My company uses an Exchange server and after much trial and error I got DavMail's Outlook Web Access gateway to work, but it is slow and sometimes crashes. MailMate does not display emails as nicely as Apple Mail, and Outlook does not preserve Apple Mail flag colors which are part of my filing system on iOS Mail.)
0 0
gilby
When I did that (use a symbolic link), Mail threw an exception on startup:
Application Specific Information:
dyld3 mode
*** Terminating app due to exception while holding database lock 'NSInternalInconsistencyException', reason: 'Couldn't get database connection in performBlock', userInfo: (null)
abort() called
You may have more success.
And it is not really a solution to go moving mail and creating links.
0 0
mny212a
I tried older online directions for moving mail, deleting ~/Library/Mail, and creating a symbolic link to ~/Library/Mail with sudo ln -s (both using /Volumes/"Macintosh HD"/Users/user-acct/Documents/Mail" and without -- on an internal volume, several ways. With ls it appears to create a valid symbolic link, but Mail app just re-created a new Mail hierarchy in the original ~/Library/Mail after launching the app.

So that seems to fail.

Outlook 2019 appears to be able to relocate mail folders to a searchable location that works with HoudahSpot.

But I will probably switch from Apple Mail to Freron.com's MailMate, which works with DavMail Gateway for an Exchange server via Outlook Web Access with my work email and also works fine with HoudahSpot.

Not perfect: MailMate doesn't preview the first sentence of every message in the 3-panel view, but the developer answered many questions for me before buying to help me get it working with HoudahSpot and DavMail, and the smart mailboxes in MailMate are very flexible, and the program is fast and small disk size compared to Microsoft Outlook.

And good luck trying to get help from Microsoft, months ago I paid for Office 2019, not Office 365 subscription but the pay once and done mac Office 2019, and for 3 months couldn't get the license to register properly on my Mac, was elevated to Level 3 tech who took over 4 weeks to resolve the issue.
0 0
houdah
The root of the problem is that Apple has moved Mail indexing from Spotlight to Core Spotlight. While doing so they have excluded ~/Library/Mail/ from Spotlight indexing. 

Spotlight indexes files. Whenever a file is modified, the Spotlight engine calls upon the appropriate importer plug-in to read metadata and text content from the file. That data is then indexed for searching.

Core Spotlight is a more recent addition. Core Spotlight does not watch for data or files to appear. Instead, applications actively submit data to Core Spotlight for indexing. This reversal of roles allows Core Spotlight to index any kind of data.

The problem we are facing with Mail on Catalina is two-fold:
- There is no obvious reason for indexing to have moved from Spotlight to Core Spotlight
- Apple does not allow third-party applications to search Core Spotlight. This is a privilege it reserves to its own Spotlight window


Replacing ~/Library/Mail/ with a symbolic link may allow bypassing Apple's "patch" of excluding that folder from Spotlight indexing. I call this a patch as I assume the reason for excluding the folder from Spotlight is to avoid duplicates in the Spotlight window search results. Too bad, Apple Mail does not accept the symbolic link.

A hard link would have been a better solution. Unfortunately, APFS does not support hard links.


One possible workaround is to create a copy of the ~/Library/Mail/ folder and use synchronization tools to keep this copy up-to-date. This has the obvious drawback of wasting disk space. Not only will the index be duplicated, but the whole email storage will also exist twice on the Mac. This may be alleviated by using APFS's "copy on write" feature where the copy takes up no actual space until it is modified to differ from the original.

A similar workaround would be to use a second email client (e.g. MailMate) that does support Spotlight. There would still be two copies of all messages on the system. You could use Apple Mail for its ability to work with rich-text messages. HoudahSpot would find MailMate messages and send you there when you need to check on an old message.
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0
houdah
More details: https://blog.houdah.com/2019/11/mail-search-on-macos-catalina/

Automation and application integration – formerly a hallmark of the Mac – are pushed out the door as more and more user data is hidden away in iOS-style data silos. It is time for a change of course that once again favors productivity.

The solution is obvious: Apple needs to again allow third-party applications full access to the search engines it builds into macOS. This includes both Spotlight and Core Spotlight.

Do submit feedback to Apple if I feel this situation needs correcting. 
Houdah Software s. à r. l.
https://www.houdah.com

HoudahGeo: One-stop photo geocoding
HoudahSpot: Advanced file search utility
Tembo: Easy and effective file search
0 0