Featured image of post My DeApple Journey
Tutorials

My DeApple Journey

In this post I would like to go through my journey of de appled (is that a word?) life.

Introduction

I have an iPhone 13 Mini and I use it extensively. Some time back I used to have MacBook Pro 13 courtesy of my workplace. As I had best products from Apple eco system I used them extensively to the length the OS supported. Even before that I used to have a iPhone 4 way back and later I had Android. Later Samsung Note 4 and Samsung S6 and so on. So basically my digital life is switching between Android and Apple eco (closed) systems . After MacBook Pro I’m still using Thinkpads having Kubuntu and I’m a Linux enthusiast for over a decade now.

Problem of switching eco systems

Whenever I was in Apple eco system all my data is in cute apps and extraction of data is quite difficult if we want an advanced approach, for example Notes, Reminders, Wallet passes etc.,. I know that there are lot of third party apps but they are not open source or behind a paid license. Apple allows files and photos to be exported but there in a quirk in photos regarding folder structure and naming conventions. Basically they make leaving Apple eco system as much hard as possible. Entering eco system feels good but when we have to exit there is always some sort of data loss and lot of pain involved.

Solutions

On the hindsight both eco systems save their data in proprietery formats and easy exporting is difficult. Haven’t tried Google but it is not our focus so we leave it here. In my opinion Companies should use open formats so that we don’t get stuck with their tools. Some formats like JSON, XML gives a good way for interoperability. If the data is complex then have it in a sqlite database and the document the hell out of it

There are couple of solutions for switching/extracting data out of eco systems in the order of least to highest friction.

  1. Leave everything behind and start fresh
  2. Export all the necessary data manually (If the data is very less)
  3. Using first party apps to export data (Move to iOS)
  4. Using third party apps to export data (Usually they support more apps and output formats)
  5. Apple Takeout
  6. Backup raw data (databases and files) from a working Mac
    • Then trying to export the data we want as we like it.
    • Setting up a dev environment
    • Then developing up a software/script to export the data in intended format.
    • Testing if the output format is viable for next set of tools.
    • So basically we have to open can of worms just for exporting data as we want it.

In the above list first 2 are self explanatory.

Coming to third we have some Apple supported apps to transfer from iOS to Android and vice versa but they are compatible only with them. So what if we want data out of the system to migrate into a completely different eco system? Usually this is not supported or partially supported like

  • Contacts in vcf
  • Calendars in ics
  • Photos in original formats but organization is skewered
  • Files in original format via iCloud
  • Notes, no industry standard format to export
  • Reminders, no industry standard format to export
  • Wallet pases, no industry standard format to export
  • Messages, no industry standard format to export
  • More apps maybe, but you get the gist

In the 4th option we have to install third party apps and probably they are best at what they are doing. They are doing the job of Apple because Apple won’t do it due to licensing/technical difficulties (aka money) If we have enough money to pay all the apps we need to export then this option would be best but still limited in the output formats. There might be some format which we like but if it doesn’t supported by the app then we are still stuck.

Coming to 5th option, Apple takeout seems like a boon when it comes to exporting data thanks to EU for enforcing laws for consumer benefit. So I tried this route and exported all necessary data (Notes, Reminders, Wallet, Bookmarks) and it took like a week to process this request (Seriously Apple, does exporting a 40mb of data take one week? This made me think their priorities). It was advertised that the takeout will include all possible formats.

I thought I will have more than one format to work with and importing elsewhere will be breeze.

Guess what, Notes was in single format and that is in simple text and I expected a json or xml format for more compatability. Ok I can work with it but when I was comparing the takeout and data from iPhone there are discrepancies. In some cases whole notes are missing and for some notes file is present but no data.
I got a JSON file for reminders and I was rejoiced but it was short lived as some reminders were missing comparing with my iPhone. So again a bust.
I didn’t even tried the other apps data because this proved to me that this is not reliable export.
Seriously I didn’t expect this from a trillion dollar company. A very poor quality of effort from Apple.

Now we are entering the Developer realm. Extract the data from existing Mac and then write up a software to export. Everything in this approach is a challenge right from extraction to deployment to another eco system

Below is the journey I went through for this step

  • In order to extract raw data we need to have a Mac.
  • If we have a Mac we won’t be having this discussion because we would be using the Apple provided apps and get deeper into that eco system.
  • Buy a cheapest Mac possible just for this purpose (I think I’ll pass).
  • Next option is to borrow a Mac from friends/family for a day or two and setting up iCloud and getting the data. I don’t think this is happening easily
  • Next option is to install MacOS in a virtual machine or do a Hackintosh.
  • Looked at Hackintosh and it looks more complex than the whole problem of data extraction so that is no go.
  • Virtual Machine seems easy option
    • The MaxBook Pro which I used from work was on Monterey so I’m trying to get that version so that everything might work.
    • Downloaded multiple ISO’s and VMDK files (almost 50gb of bandwidth used), tried to install MacOS from all of them. Half of them didn’t install and it took me almsot 2 days just to get to a bootable MacOS. Some ISO’s are either corrupt or not meant to be bootable (That is new for me).
    • Successfully installed Mojave, Catalina and BigSur
    • Now the moment of truth. Setting up iCloud in all those systems
    • Mojave worked with iCloud setup without any issues but Notes didn’t sync fully and Reminders didnt sync at all.
    • I have like 600+ notes but it synced only 300+ notes and even after logging out of iCloud and restarting Notes app and VM, nothing worked. Maybe version mismatch in database and it doesn’t support older versions.
    • Tried with BigSur but iCloud cannot be setup because it needs a valid physical Mac serial number which I don’t have.
    • I tried with mac serials script, SMBIOS and virtualbox extra parameters and stuff. Slowly it is entering into Hacintosh realm and that is not worth it (Mostly because I lack the knowledge to see it through, believe me dealing with Linux is very easy compared to MacOS)
    • Later I tried Catalina where iCloud works but here also Notes didn’t sync all the way but luckily in sqlite database all records were present.
    • So here at-last we have the raw data.
    • While I was in Catalina I tried to install older version of Obsidian and tried to export Apple notes into Obsidian.
    • For 600+ notes the expected time was more than 2 hours and it has lot of failures. It exited mid way leaving with half baked import which I cannot use. So I have to manually extract data and process it.
    • I installed virtualbox guest additions in Catalina and it still shared folders didn’t work. I don’t want to fight with it.
    • So I used scp to uplaod the compressed folder to my other laptop running Linux and called it a day.

So here we are after 2 days with our raw data from MacOS. No ones want to go this route but Apple makes us do it. Thanks Apple. This is a example why we should not get exited to get into your walled garden in the first place.

Conclusion

Pretty much all easy options failed or not adequate and our only option is to recover from the raw data. I don’t guarantee that this will work entirely as I will be working with no inside information of how the database is setup and what the data means.

Stay tuned for second part where I will process raw data.