Best way to keep a dynamic (large) list of "stuff"

[FishNiX]

If I had the need to keep a (potentially) large list (10k - 100k) of relatively short (300 characters-ish) strings what's the best way to do that in a plugin? Is there a "standard"/"best practices" way?

Text file -- slow and disk intensive
DB -- but where/how?
BerkleyDB -- more complexity
Memcache -- even more complexity

Thanks!

[FishNiX]

BTW -- I'm planning to implement a couple of these, but what should be canonical?

[garvinhicking]

Hi!

Good point. I think if you need lookup features, i.e. fetch certain strings depending on conditions, the DB is the only proper way to deal with. Make the plugin create its own database (see staticpag eplugin, look for "setup" method and how its called, andhow it uses "CREATE TABLE..." sql strings.

BerkeleyDB and memcache have too many dependencies so they should be avoided by default and maybe only added as an option.

I would use a textfile, if you always load the full set of data/strings though. This is faster than passing data from the SQL server to your application through a socket.

Regards,
Garvin

[FishNiX]

Yeah -- I'm thinking at some point in the future I will want to query the list one way or another. I'll shoot for the DB. Thanks!