Progress percentage, bar or log for library sync

Feature description:

Add a progress percentage/bar/log for library syncs instead of the spinning indicator, even if it’s only a rough estimate.

Problem solved:

People with large libraries like me (almost 800.000 songs and over 20TB in Navidrome) cannot see that the sync is actually still doing something. I had to enable debug logging and check out a 200MB log file with 350.000 lines to see that the sync is in fact not hanging after no visible change occurred over multiple hours.

Brought benefits:

More information is usually useful.

There’s plans to improve on that, but Subsonic is the typical example of it’s not possible :slight_smile:

The API suxx for fast sync all I can do is query artists X by X until the returned data is < X, then the same for albums then the same for songs.

I never know how many items are left or the time it will take or anything.

With that said even for so much data is should not take multiple hours there’s something wrong.
I suspect the search3 endpoint hack I use might be slow on that amount of data on Navidrome side and maybe I should check with @deluan if there’s something to improve with indexes or something.

That’s a shame.

Would syncing Plex or Jellyfin be faster?

I’ve got my library in Navidrome, Plex and Jellyfin, Navidrome being the newest addition.
It took Navidrome a bit under 19 hours to scan my library and except for the missing support for .lrc files and occasional browser freezes when accessing huge artists with hundreds of albums I am happy with the performance.

I was looking forward to see how Symfonium handles my library as I was not happy with Ultrasonic and substreamer, which I’ve tried first.

By now the symfonium log file is almost 2GB in size. Should I let the sync run and see where it ends up/if it finishes at all?

The fastest is Kodi, then I’d say Plex then the rest, but Navidrome is quite fast usually.

I’d need just a part of the logs that contains at least 2 or 3 search3.view calls to see the time Navidrome takes to answer and the time to insert on db on Symfonium side.

If you enable logs during sync, on low/mid range device the sync is wayyy slower due to all the writes to storage.

It started out with 50ish ms on average early on in the sync

12:35:34.418 Verbose/SubsonicLogger: <-- [961] 200 ... v=1.13.0&c=Symfonium&f=json (55ms, unknown-length body)

now it’s around 350ms on average

2023-07-10 16:44:03.476 Verbose/SubsonicLogger: <-- [790] 200 ... v=1.13.0&c=Symfonium&f=json (352ms, unknown-length body)

I’ve disabled logging now. The device is a OnePlus 7T in high performance mode connected to the same local network that the Navidrome server is in.

Considering that the sync is currently adding songs starting with “B” this might take quite a while more to finish.

2023-07-10 16:44:50.177 Verbose/Subsonic: Insert Song: Big Time

If you’re interested in other parts of the log let me know, I’ve just serched for \d+.?\d+ms in the lines with search3.view and tried to keep it short. All other times with ms are in the sub 1-10ms range.

Ok so Navidrome is probably using limits and not windowing so the bigger the start index the longer the query to drop not needed data :frowning:

What I need is also the logs between 2 of those calls to see the Symfonium insertion delay.

The way the query is done the songs should not be sorted, so probably Navidrome that add a name sort and not an row index sort.

@deluan If you are around, probably some things we can improve here.

After the first call (line 3.352.607):

2023-07-10 16:44:13.402 Verbose/Subsonic: Insert Song: Bacon In The Skillet

Before the next call (line 3.353.106):

2023-07-10 16:44:13.457 Verbose/Subsonic: Insert Song: Bad Game (3AM)


Yes 50ms is slow :frowning:

Is it consistent or has it increased?

Is your server remotely accessible so I could try to repro with that large library?

I just noticed that there are some lines in the log that are obscenely long.

2023-07-10 16:44:49.294 Verbose/SubsonicLogger: <-- [251] {"subsonic-response":{"status":"ok","version":"1.16.1","type":"navidrome","serverVersion":"0.49.3

for example is 297.776 characters in length.

At the start when it was doing artists:
29.964 to 30.035
30.169 to 30.243
30.382 to 30.451

At the end with songs:
44.050 to 44.241
45.551 to 45.803
47.268 to 47.875

So it started out fairly consistent with 50-80ish ms and now it’s inconsistent and vastly slower.

Not yet. I wanted to see how things run locally before I make it accessible over the internet.

I just re-enabled logging for 23 seconds and the search3.view query time is up to 1.450ms on average.
It also created 4MB worth of logs (4.7million characters) in that time.

Call 1449ms
2023-07-10 18:05:20.017 Verbose/Subsonic: Insert Song: Fried Neckbones and Home Fries
2023-07-10 18:05:20.112 Verbose/Subsonic: Insert Song: Frodo Disappears
Call 1450ms
2023-07-10 18:05:21.680 Verbose/Subsonic: Insert Song: Frodos Entscheidung
2023-07-10 18:05:21.798 Verbose/Subsonic: Insert Song: From The Beginning
Call 1346ms
2023-07-10 18:05:23.256 Verbose/Subsonic: Insert Song: From The Beginning
2023-07-10 18:05:23.352 Verbose/Subsonic: Insert Song: Frozen Fountain

Logs have everything so yes they are large.

And so the 100ms is for all the songs between calls? So normal time on Symfonium.

And not normal time on Navidrome. Need to see with @deluan how to optimize this.

Yes, I tried to symbolize that with the … in the lines between.

Alright, I’ll monitor how the response times evolve over time and report back if something noteworthy happens.

Just to be sure what Navidrome version ?

So quick look to Navidrome code and yes it force sort by title and use limits so that will be exponentially slow with the increased startOffset.

via this docker image

The sync did eventually finish after taking 2 days and draining my phones battery 2 times. There’s some room for improvement I guess. The last time I checked the log it took over 3.1 seconds for each call to navidrome when it was adding songs starting with “p”.

Yes I’ll need to convince @deluan to add a special optimized path in Navidrome.

Too bad I know nothing at Go to make a PR.

I synced the same library with jellyfin and that took less than 2 hours.

Yes don’t mix stuff in issues else it’s not usable for the others.

I’m not sure any provider properly handle external lyrics. But there’s plenty of tools to automatically add the lyrics to the file tags and they Symfonium can handle them in all cases.

And yes artist splitting is provider specific I only use the data the provider sends. Navidrome does not support multiple artists for now. (Plex also don’t).

Kodi/Emby/Jellyfin (And Symfonium local device parser) are the only ones that properly handle multiple artists.

I’ll create a new post then. I’ve edited my post to keep this issue clean, feel free to delete your response and this one by me as well to clean up the issue.

Hey @655321, I just merged a change in Navidrome that should speed up this sync process. It will be part of the next release, but it will be available in the develop build later today, if you want to give it a try.

1 Like