The case of the recursive symlink: an iTunes outlook
On a regular Friday afternoon, I pop in my iPhone to do the regular iTunes backup (like any sane person does). It turns out iTunes is instantly crashing when I try to do a backup.
That's not good news.
Trying a couple of times, but still failing, crashes right before I try to do a backup.
Now to give some context on my setup, since I'm using a 128GB iPhone, it's not the best use of storage to backup this mammoth on the primary SSD. I use a symlink approach to point my Backup folder to an external HDD.
First thing I try: as they always say, have you tried turning it off and on again?. I restart the Mac, start clean, no open apps and let's see.
Crash report analysis
Still crashes. Looking at the crash report doesn't give you a hint right away.
Process: iTunes [2360]
Path: /Applications/iTunes.app/Contents/MacOS/iTunes
Identifier: com.apple.iTunes
Version: 12.9.4 (12.9.4)
Build Info: iTunes-1200012009004094~15
Code Type: X86-64 (Native)
Parent Process: ??? [1]
Responsible: iTunes [2360]
User ID: 502
Date/Time: 2019-07-12 23:47:10.284 +0100
OS Version: Mac OS X 10.14.4 (18E226)
Report Version: 12
Anonymous UUID: <MASKED>
Time Awake Since Boot: 840 seconds
System Integrity Protection: enabled
Crashed Thread: 21
Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x000070000e218ff8
Exception Note: EXC_CORPSE_NOTIFY
Termination Signal: Bus error: 10
Termination Reason: Namespace SIGNAL, Code 0xa
Terminating Process: exc handler [2360]
Seems that iTunes is trying to access something it's not supposed to?
First thing comes to suspicion is the symlink. Maybe some folder structure changed or the symlink broke for some reason (drive name change, etc..).
I delete the /Library/Application Support/MobileSync/Backup symlink just to be sure. Guess what? Still crashes.
Looking further down through the crash report,
Thread 21 Crashed:
0 libsystem_platform.dylib 0x00007fff69140271 _platform_strlcpy + 23
1 libsystem_c.dylib 0x00007fff6902223e realpath$DARWIN_EXTSN + 291
2 com.apple.iTunes 0x000000010f23eb8e 0x10e906000 + 9669518
3 com.apple.iTunes 0x000000010f23dbd0 0x10e906000 + 9665488
...
Now _platform_strlcpy is darwin's strlcpy C function which (you guessed it) copies a string. Now what is iTunes trying to copy and failing?
My suspicion starts to go to any recently installed apps that might conflict with iTunes, I have recently installed MalwareBytes after the busy month of June Malware festival that was happening. Could it be that MalwareBytes is attempting to "scan" iTunes and causes it to crash?
Spolier alert, uninstalling MalwareBytes does not solve the issue.
Upgrading MacOS
Glancing through hoakley's MacOS 10.14.4's known bug list, it seems there are some that have issues with the APFS file system. Maybe that's it?
Upgrading to Mojave 10.14.5 does not solve the iTunes crash. So maybe 10.14.4 corrupted the filesystem somehow, let's run First Aid.
First Aid reports some directory failed checks that it is not able to solve.
Checking snapshot 1 of 6.
error: directory valence check: directory (oid 0x13): nchildren (1) does not match drec count (0)
warning: snapshot fsroot tree corruptions are not repaired; they'll go away once the snapshot is deleted
error: directory valence check: directory (oid 0x150045): nchildren (2) does not match drec count (0)
error: directory valence check: directory (oid 0x150063): nchildren (2) does not match drec count (0)
...
Now I've dealt with corrupted disks before and trust me, it ain't fun. Literally anything and everything can go wrong with a corrupted filesystem and it becomes near impossible to debug.
So before giving up and reformatting, I decided to give it another shot, what else can go wrong?
To eliminate variables as much as possible, I try loading up iTunes on a different User Account. Guess what? It works perfectly fine.
We got a lead!
It now seems that my user has some kind of corrupted settings that iTunes cannot access. Now let's get rid of anything user related that iTunes would access and it should work, right?
Still crashing, iTunes recreates these settings files and crashes shortly after navigating to the iOS device tab.
At this point I was on the verge of giving up and just popping in the Time Machine and going back to when it was working, but that is rather a destructive solution that would not solve the root cause (which will probably return and haunt me again).
Another less-destructive solution would be to just create a new user and continue life there, but still the bug might return.
Dig deeper, debugging tools
Xcode
Debugging in Xcode was quite interesting, iTunes did not crash when attached to the debugger. This is probably due to Xcode cutting the process some slack when it comes to stack allocations (as we will see below)
fs_usage
A very useful debugging tool I use is fs_usage, it watches the process manipulate files in the filesystem and enables you to track down what's really going on. It can be very useful if you are suspicious of a process and want to know what it's doing behind the scenes.
sudo fs_usage iTunes
LO AND BEHOLD!
Right before iTunes crashes, it's trying to access these files:
Now what does iTunes have to do with this directory? Why is it trying to access PokeMMO's Application Support?
PokeMMO is an Unofficial Online Multiplayer Pokemon game that I was really hesitant when installing. It's closed source and seemed shady. (Turns out I was right, kinda?)
As you can see in the screenshot, it's trying to access the same directory, over and over again, around 200,000+ times. This is absurd!
Right before this massive recursion or loop:
/Users/ahmedshehata 0.000004 iTunes.184583
14:42:26.798987 getattrlist /Users/ahmedshehata/Pictures 0.000004 iTunes.184583
14:42:26.798992 getattrlist /Users/ahmedshehata/Pictures/PokeMMO Screenshots
It looks like iTunes is accessing the /Pictures directory (probably to get sync data for photos synced with the iOS Device).
Looking at the /Pictures directory and this is where we finally find our culprit. A rouge recursive symlink that has been causing iTunes to crash.
The filesystem calls to list the Pictures directory was probably causing the stack to overflow when listing through this recursive symlink.
Now recursive symlinks are no fun, they can break apps randomly, cause the SSD to wear out and cause lots of horrible things to happen.
Deleting the symlink solves the issue as expected.
I hope you have learned some useful debugging skills that can help you find the root cause of some undefined behaviour of a program without resorting to extreme measures (such as re-installing Operating System or recovering).
TLDR: PokeMMO creates a recursive symlink in
/Picturesdirectory that caused iTunes to behave abnormally and crash.
Thanks for reading and let me know in the comments if you have faced a similar situation and used some cool debugging tools yourself!
