addSelect images in Admin module/serendipity_traversePath slowness

Discussion corner for Developers of Serendipity.
Post Reply
FishNiX
Regular
Posts: 40
Joined: Sun Sep 02, 2007 6:32 pm

addSelect images in Admin module/serendipity_traversePath slowness

Post by FishNiX » Fri Jul 31, 2020 2:12 pm

Good morning!

We have been happy s9y users for a long while now and we have a fairly large uploads directory with many subdirs. The content editors are having a hard time adding and selecting media because it takes so long when you click the "Media" button in a "New Entry" and also after that frame is finally loaded, when you click the "Add Media" button.

I initially thought this was a database problem, but the the database queries, while time consuming, are not terrible (around 5s total). I think I've tracked the problem to the directory traversal which I think is populating the folder lists. It looks like its calling serendipity_traversePath() which walks over all 55,000(!) images we have in the uploads directory.

If there were a hook for this add select or for that serendipity_traversePath() function, I could add some caching there but I don't want to muck with core functionality. Is there any option you can think of to make this faster? I think those directories exist in the database already, can we somehow skip the filesystem walk?

Thanks!

User avatar
onli
Regular
Posts: 2362
Joined: Tue Sep 09, 2008 10:04 pm
Contact:

Re: addSelect images in Admin module/serendipity_traversePath slowness

Post by onli » Fri Jul 31, 2020 3:29 pm

Hi!
I had a look at the code, hoping we just do something stupid. But no, the traversePath function looks reasonable and needed.

So we can try caching. I'm not sure it will work, because:

1. Whenever something is modified in the media library we will have to invalidate the cache.
2. We so far have just one cache, mixed with the entries, so new entries will also invalidate the cache.
3. The caching could make the function slower whenever the cache is empty.

But despite all that it's worth a shot, and it can be improved if it looks promising.

Is it okay for you to test this in your environment? Without such an upload folder it's hard to test this. I added caching to the current functions_images.inc.php and uploaded it here: https://gist.github.com/onli/01ba94c2a3 ... 28d000a65d

Could you backup your current include/functions_images.inc.php, replace it with the modified version and see whether it helps?

FishNiX
Regular
Posts: 40
Joined: Sun Sep 02, 2007 6:32 pm

Re: addSelect images in Admin module/serendipity_traversePath slowness

Post by FishNiX » Fri Jul 31, 2020 6:10 pm

Hi onli -

Thank you for your reply and your work on the platform!

I've replaced my functions_images.inc.php with your modified version but I don't see a difference (it takes about 51s to load the media library). I don't see any content in the database cache... should this show in the filesystem cache or the DB?

I also added

Code: Select all

$serendipity['useInternalCache'] = true;
to my serendipity_config_local.inc.php although I expect it's already enabled (?).

Cheers!

FishNiX
Regular
Posts: 40
Joined: Sun Sep 02, 2007 6:32 pm

Re: addSelect images in Admin module/serendipity_traversePath slowness

Post by FishNiX » Fri Jul 31, 2020 6:18 pm

I'm not sure if it's possible, but if there was a hook in the addSelect function (and maybe the next one thats slow after we make this faster.... :lol: ) I could do my own caching of the folder list.

Of course, it might be more beneficial to serendipity as a platform to allow a hook in the traversePath function, I don't know. (and I don't know if that's even possible).

FWIW, we have a simple directory structure of YYYY/YYYYMM with lots and lots of images (going back to 2007!).

User avatar
onli
Regular
Posts: 2362
Joined: Tue Sep 09, 2008 10:04 pm
Contact:

Re: addSelect images in Admin module/serendipity_traversePath slowness

Post by onli » Fri Jul 31, 2020 7:16 pm

It loads equally long when you enter the Media Library (or go to "Add Media") immediately after visiting that page once? It's 51 seconds always, with the old version it was equally slow?

Are you on shared hosting, or do you control the server completely?

I don't see how to provide a sane hook that would enable the caching better than this modification, I think that belongs in the core. The performance has to be good enough to support setups such as yours. And I think we can at least improve it.

The option to enable the cache is under "Configuration -> General Settings -> Enable caching", it's the last one in that submenu.

FishNiX
Regular
Posts: 40
Joined: Sun Sep 02, 2007 6:32 pm

Re: addSelect images in Admin module/serendipity_traversePath slowness

Post by FishNiX » Fri Jul 31, 2020 10:08 pm

Ah-ha! Yes, caching was disabled. With your changes and the media pages cached, the response is much better.

The second load of the media library (from the left panel of the admin) only took 5.2 seconds. Clicking the "Media" button in a new entry after the cache has been populated took about the same (6s). Clicking the "Add media" button seems to take the full 50s. the first time and then about 5s the second time.

I am hosting the site on a server that I control but testing this in a local docker development environment. Interestingly, I have the cache disabled on the main site as well (although I make heavy use of varnish).

User avatar
onli
Regular
Posts: 2362
Joined: Tue Sep 09, 2008 10:04 pm
Contact:

Re: addSelect images in Admin module/serendipity_traversePath slowness

Post by onli » Sat Aug 01, 2020 9:57 am

Nice :) Now just the question remains whethere the occasional long loading time is acceptable for your editors. If not we have to think about a different approach to creating that file list - taking it from the database should be way faster for example, and we already have the data in the database, I'm not sure why it's not done like that already.

Since you control the server: File system speed will have a big influence here. If you use ext4 for this partition, make sure it's mounted with relatime or noatime. Plus lazytime as well if the kernel is not ancient, https://lwn.net/Articles/621046/.

FishNiX
Regular
Posts: 40
Joined: Sun Sep 02, 2007 6:32 pm

Re: addSelect images in Admin module/serendipity_traversePath slowness

Post by FishNiX » Sat Aug 01, 2020 1:07 pm

onli wrote:
Sat Aug 01, 2020 9:57 am
Nice :) Now just the question remains whethere the occasional long loading time is acceptable for your editors. If not we have to think about a different approach to creating that file list - taking it from the database should be way faster for example, and we already have the data in the database, I'm not sure why it's not done like that already.
It's definitely an improvement over the current state, but it would be better if the 5s was the norm, especially since we have the data already in the database. I can explain it to the editors (there are only a few) but they would be confused as to why it was slow sometimes and fast sometimes.
onli wrote:
Sat Aug 01, 2020 9:57 am
Since you control the server: File system speed will have a big influence here. If you use ext4 for this partition, make sure it's mounted with relatime or noatime. Plus lazytime as well if the kernel is not ancient, https://lwn.net/Articles/621046/.
Thanks, I'll check it out!

As a side note... I do remember looking into simple_cache a year or so ago (maybe longer?) and I'm not sure why it's off on my instance. Can you imagine any downside to just turning it on?

Thanks again for the help!

User avatar
onli
Regular
Posts: 2362
Joined: Tue Sep 09, 2008 10:04 pm
Contact:

Re: addSelect images in Admin module/serendipity_traversePath slowness

Post by onli » Sat Aug 01, 2020 8:34 pm

Some plugins don't work when the cache is enabled, maybe because of that. It was also broken for a bit and it got introduced with its default set to off, so older installations that changes settings until the default changed won't have it enabled if they didn't change the setting manually.

For improving the performance in general, I will enable the cache for now and hope to either pick it up later, or maybe someone else tries improving how that that file list is generated in general.

FishNiX
Regular
Posts: 40
Joined: Sun Sep 02, 2007 6:32 pm

Re: addSelect images in Admin module/serendipity_traversePath slowness

Post by FishNiX » Fri Aug 07, 2020 4:02 pm

Sounds good thanks. I've enabled the cache for our site. I'll monitor to make sure everything is good. I do vaguely remember some issue with a plugin or other functionality back when it was first added.

Once the cache stuff is merged/released, I'll try to find some free cycles to look at that directory list... maybe we can read it from the database? I assume the reason we don't do that now is because there aren't a lot of opportunities to refresh that list of directories and this is probably one of them. Perhaps for users like me who don't ever add directories/files directly to the filesystem (only through the UI), there can be a manual refresh option.

Thanks again!

User avatar
onli
Regular
Posts: 2362
Joined: Tue Sep 09, 2008 10:04 pm
Contact:

Re: addSelect images in Admin module/serendipity_traversePath slowness

Post by onli » Fri Aug 07, 2020 4:20 pm

Yes, I assume that by not relying too much on the database here s9y avoided the filesystem and the database being out of sync. But I think we already have everything to change that, to read the data from the database and make the synchronisation a button in the maintenance menu. Would be great if you could work on that!

Post Reply