Seems it’s hard to find so let’s expose this more.
In all screens there’s a filter button that will expose filters specific to each screens. ([Wiki] How to sort, filter and configure display modes in library views)
To hide small artists (So artists with less than 5 songs) just enable the filter in the artist view
And the exact same thing for albums
If you do not like the default values you can always use smart filters to configure the value you want.