Add a progress percentage/bar/log for library syncs instead of the spinning indicator, even if it’s only a rough estimate.
Problem solved:
People with large libraries like me (almost 800.000 songs and over 20TB in Navidrome) cannot see that the sync is actually still doing something. I had to enable debug logging and check out a 200MB log file with 350.000 lines to see that the sync is in fact not hanging after no visible change occurred over multiple hours.
There’s plans to improve on that, but Subsonic is the typical example of it’s not possible
The API suxx for fast sync all I can do is query artists X by X until the returned data is < X, then the same for albums then the same for songs.
I never know how many items are left or the time it will take or anything.
With that said even for so much data is should not take multiple hours there’s something wrong.
I suspect the search3 endpoint hack I use might be slow on that amount of data on Navidrome side and maybe I should check with @deluan if there’s something to improve with indexes or something.
I’ve got my library in Navidrome, Plex and Jellyfin, Navidrome being the newest addition.
It took Navidrome a bit under 19 hours to scan my library and except for the missing support for .lrc files and occasional browser freezes when accessing huge artists with hundreds of albums I am happy with the performance.
I was looking forward to see how Symfonium handles my library as I was not happy with Ultrasonic and substreamer, which I’ve tried first.
By now the symfonium log file is almost 2GB in size. Should I let the sync run and see where it ends up/if it finishes at all?
The fastest is Kodi, then I’d say Plex then the rest, but Navidrome is quite fast usually.
I’d need just a part of the logs that contains at least 2 or 3 search3.view calls to see the time Navidrome takes to answer and the time to insert on db on Symfonium side.
If you enable logs during sync, on low/mid range device the sync is wayyy slower due to all the writes to storage.
I’ve disabled logging now. The device is a OnePlus 7T in high performance mode connected to the same local network that the Navidrome server is in.
Considering that the sync is currently adding songs starting with “B” this might take quite a while more to finish.
2023-07-10 16:44:50.177 Verbose/Subsonic: Insert Song: Big Time
If you’re interested in other parts of the log let me know, I’ve just serched for \d+.?\d+ms in the lines with search3.view and tried to keep it short. All other times with ms are in the sub 1-10ms range.
At the start when it was doing artists:
29.964 to 30.035
30.169 to 30.243
30.382 to 30.451
At the end with songs:
44.050 to 44.241
45.551 to 45.803
47.268 to 47.875
So it started out fairly consistent with 50-80ish ms and now it’s inconsistent and vastly slower.
Not yet. I wanted to see how things run locally before I make it accessible over the internet.
Update:
I just re-enabled logging for 23 seconds and the search3.view query time is up to 1.450ms on average.
It also created 4MB worth of logs (4.7million characters) in that time.
Call 1449ms
2023-07-10 18:05:20.017 Verbose/Subsonic: Insert Song: Fried Neckbones and Home Fries
..
2023-07-10 18:05:20.112 Verbose/Subsonic: Insert Song: Frodo Disappears
Call 1450ms
2023-07-10 18:05:21.680 Verbose/Subsonic: Insert Song: Frodos Entscheidung
..
2023-07-10 18:05:21.798 Verbose/Subsonic: Insert Song: From The Beginning
Call 1346ms
2023-07-10 18:05:23.256 Verbose/Subsonic: Insert Song: From The Beginning
..
2023-07-10 18:05:23.352 Verbose/Subsonic: Insert Song: Frozen Fountain
The sync did eventually finish after taking 2 days and draining my phones battery 2 times. There’s some room for improvement I guess. The last time I checked the log it took over 3.1 seconds for each call to navidrome when it was adding songs starting with “p”.
Yes don’t mix stuff in issues else it’s not usable for the others.
I’m not sure any provider properly handle external lyrics. But there’s plenty of tools to automatically add the lyrics to the file tags and they Symfonium can handle them in all cases.
And yes artist splitting is provider specific I only use the data the provider sends. Navidrome does not support multiple artists for now. (Plex also don’t).
Kodi/Emby/Jellyfin (And Symfonium local device parser) are the only ones that properly handle multiple artists.
I’ll create a new post then. I’ve edited my post to keep this issue clean, feel free to delete your response and this one by me as well to clean up the issue.
Hey @655321, I just merged a change in Navidrome that should speed up this sync process. It will be part of the next release, but it will be available in the develop build later today, if you want to give it a try.
