New RSS Parser for RSS Aggregator

[oliof]

Hi,
being frustrated by the fact that both Onyx and Magpie are a couple of years out of maintenance, I invested a couple hours tonight to patch in SimplePie support for the RSS Aggregator plugin.

I'm not 100% convinced that it works flawlessly, but I'm 100% convinced it's working faster and better than the other two parsers.

If anyone is interested, I can provide a patch vs. RSS Aggregator v0.23.

[garvinhicking]

Hi!

That would be great, did you maybe make it optional like the current toggle for Onyx and Magpie, for a third option? I'd love to commit your patch!

Regards,
Garvin

[oliof]

Yes, I made it a third option to toggle between Onyx, Magpie and SimplePie. I will check if I need to clean up stuff (my PHP is not worth mentioning really, I just mixed and matched snippets until they worked).

SimplePie is available under a BSD license, just like Onyx.

I still see some (to all) articles from some feeds parsed and found in the cache, but not listed on the planet. I guess that's due to unexpected interaction with the cache.

How would you like to get the patch?

[garvinhicking]

Hi!

That's fine, I can also try to help with the cleanup.

If you can attach the complete plugin as a ZIP here, that would be fine.

Regards
,Garvin

[oliof]

Hi,

I cannot attach the zip file here. You can download it from http://blog.mausdompteur.de/uploads/aggregator.zip

The patch assumes simplepie 1.2 to be present in bundled-libs/

Cheers,
Harald

PS: I found the solution to my caching problems. I cleared out s9y_entries earlier, but forgot to also clear out s9y_aggegator_md5 …

[Don Chambers]

Sounds exciting Harald!!! Can you elaborate on your caching problem? you said you "cleared out" the entries table? Does this mean you deleted all aggregated entries, then forced re-aggregation?

Garvin - if you can review this code carefully, I might be willing to experiment with it on a high volume aggregated site.... but the code/functionality needs to be clean/reliable.

[garvinhicking]

Hi!

Great, I just committed your patch. Works fine here on my machine; I bundled the simplepie libraries inside the plugin, though.

Best regards,
Garvin

[oliof]

Hi Don,

I had problems with UTF support with both Onyx and MagpieRSS, so I wanted a modern, maintained RSS Parser. A quick trip to Google sent me around to simplepie.

Can you elaborate on your caching problem? you said you "cleared out" the entries table? Does this mean you deleted all aggregated entries, then forced re-aggregation?

Exactly. When switching between the parsers, I wanted to avoid articles parsed by the former parser(s) to be mixed up with ones parsed by the current one, so I deleted all aggregated entries (on the db level), then re-aggregated. Deleting aggregated entries on the db level does not trigger deletion from the associated md5 sum that's stored in another table, so the caching mechanism saw the md5 sum and decided not to store the entry.

Garvin - if you can review this code carefully, I might be willing to experiment with it on a high volume aggregated site.... but the code/functionality needs to be clean/reliable.

I'm aggregating ~100 sites with 50 updates/week. The three days I use simplepie, it has been working fine.

The code is relatively simple, less than 30lines IIRC. It involves an addition to the parser selector, a new else tree for using simplepie in the library load logic, and a new else tree for actually having simplepie read the feeds and stuffing them into the hash that the rss aggregator uses to store entries.

Since I lifted most of the logic from the aggregator itself (except for the simplepie initialization, which I lifted from the simplepie demo), I hope it's not too unclean. As Garvin already committed the patch, I guess it's good enough.

PS: moving simplepie to the plugin itself is splendid; but I don't know enough about the inner working of s9y, so I just copied the way Onyx is included. Thanks for cleaning that mess up!

Cheers,
Harald