Automated functional testing

Discussion corner for Developers of Serendipity.
Post Reply
peter.kehl
Regular
Posts: 5
Joined: Tue Apr 01, 2014 8:18 am

Automated functional testing

Post by peter.kehl » Tue Apr 01, 2014 8:49 am

If you'd like automated browser-based functional/regression testing of Serendipity, you may be interested in SeLite. It extends Selenium IDE. It allows your tests to access (read and write to) a test DB (isolated from the DB of the tested application). It's open source.

It could work very well for you, especially since Serendipity can have its data in an SQLite DB. Since SeLite uses a (separate) SQLite DB for test data, your test data lifecycle would be very easy. However, you can benefit from SeLite even if your test instance of Serendipity uses other type of DB.

Creating tests in Selenium IDE is convenient and highly productive. It doesn't require low-level programming skills, so even users/designers could contribute new tests. Tests can be maintained in SVN. See https://code.google.com/p/selite/wiki/ProjectHome.

I could create a test framework and provide some basic examples of tests. Would there be any interest on side of Serendipity developers?

User avatar
garvinhicking
Core Developer
Posts: 30020
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Automated functional testing

Post by garvinhicking » Tue Apr 01, 2014 11:00 am

Hi Peter!

I must admit I currently not very much knowledge about Selenium and tests like these. But I would definitely be interested in some examples to get me "kickstarted", yes. Many thanks for contacting us!

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

peter.kehl
Regular
Posts: 5
Joined: Tue Apr 01, 2014 8:18 am

Re: Automated functional testing

Post by peter.kehl » Thu Apr 17, 2014 9:02 am

Hi Garvin and others,

I'm working on the test framework. It's still a work in progress. The API and the framework are likely to change. If you'd like to get a feel of it, follow https://code.google.com/p/selite/wiki/S ... yFramework.

Currently it can login/logout as a specified user. It posts an entry (with random title, body and extended). It can verify title, but not body and extended yet (more XPath/Javascript functionality to come).

-Peter Kehl

User avatar
garvinhicking
Core Developer
Posts: 30020
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Automated functional testing

Post by garvinhicking » Thu Apr 17, 2014 10:26 am

Hey,

wow. Ok. Phew. This sounds like an awful lot of work to grep through Class hierarchies. I wonder, because we offer different templates, how it can best be achieved to test the "core" of serendipity rather than the template itself. But I suppose for now, all would needs to be tested with the default template.

For Serendipity 2.0 however we greatly reworked the HTML output of the backend (new HTML5) so maybe it would be better to start working on this with serendipity 2.0 in mind - because all the old HTML structure of the backend will be different after 1.7...

Thanks for your help,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

mattsches
Regular
Posts: 440
Joined: Sat Nov 05, 2005 10:35 pm
Location: Wiesbaden, Germany
Contact:

Re: Automated functional testing

Post by mattsches » Sun Apr 20, 2014 11:04 am

Hi Peter,

I will definitely take a look at it. I created a setup for unit testing Serendipity plugins that works quite nicely for me. I also have some experience in functional testing with Selenium and other tools

My setup also contains a small Sqlite db to create simple "fixtures" of data required to bootstrap Serendipity. Maybe we can join forces in some way or the other to increase test coverage in the future.

I'll let you know once I found time to install your tool. And I agree with Garvin: If possible, adjust your test suite to cover the new Serendipity 2.0-beta versions. This way, it will be a lot more helpful for the dev community here.

Thanks for your work!
mattsches

peter.kehl
Regular
Posts: 5
Joined: Tue Apr 01, 2014 8:18 am

Re: Automated functional testing

Post by peter.kehl » Mon Apr 21, 2014 4:24 am

Hi Garvin and Mattsches,

I'll update the framework and docs for Serendipity 2.0. I'll try to make it honour configuration of permalinks and various paths (as per Configuration - http://www.s9y.org/66.html).

I see that there are themes in Spartacus. As Garvin suggested, those can modify HTML of the forms, so I can't target elements by XPath based on CSS classes or element IDs. Can I assume that plugins won't modify/hijack
- the standard page flow - e.g. a page form for a new entry will still be submitted to the same URL to post the entry
- 'name' for input/textarea elements for fields of new entry form etc. (This should be a fair assumption, since the backend depends on input names.)
- standard Javascript validation - that way I can distinguish the Preview and Submit buttons (in Serendipity 1.7)

Another way of making tests independent of HTML structure and CSS is to use UI Mapping, as per
http://www.seleniumhq.org/docs/06_test_ ... ations.jsp > UI Mapping and
https://selenium.googlecode.com/git/jav ... i-doc.html There could be one set of mappings per theme (or per set of themes).

User avatar
garvinhicking
Core Developer
Posts: 30020
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Automated functional testing

Post by garvinhicking » Thu Apr 24, 2014 1:53 pm

Hi!

You should only care about the HTML layout of our 2k14 default admin template. This will be the only template that we will internally support and use for functional testing. Any other templates could possible modify HTML form, but that is beyond our scope to be able to support.

So the HTML classes and workflow you find in 2.0 should IMO be the ones used to fixture on.

HTH,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

User avatar
yellowled
Regular
Posts: 7078
Joined: Fri Jan 13, 2006 12:46 pm
Location: Eutin, Germany
Contact:

Re: Automated functional testing

Post by yellowled » Thu Apr 24, 2014 3:05 pm

garvinhicking wrote:You should only care about the HTML layout of our 2k14 default admin template.
2k11. Or did I miss something? :wink:

YL
amazon Wishlist - Serendipity-Podcast (German only, sorry)

peter.kehl
Regular
Posts: 5
Joined: Tue Apr 01, 2014 8:18 am

Escaping and treatment of permalinks

Post by peter.kehl » Fri Apr 25, 2014 9:48 am

When Serendipity generates permalink to an entry (with standard permalinkStructure archives/%id%-%title%.html), it replaces/removes some characters from 'title' field. What I've found so far:
  • replace space with a hyphen -
  • remove & ' " / \ <> ` * $ £ ? = # : | ^ ( ) [ ] { }
  • url-escape %
Are there any other replacements/escapes/removals? Or what's the PHP function that does it?

(I see that it doesn't matter too much when retrieving the entry, because then the system uses the ID and it seems to ignore the rest of the URL. However, it matters when I want to verify a link generated on the success page - after the author posted it, and on the list of entries).

Also, are these escape/replace/remove rules same for any fields in permalinks and for all permalinks?

User avatar
garvinhicking
Core Developer
Posts: 30020
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Escaping and treatment of permalinks

Post by garvinhicking » Fri Apr 25, 2014 10:51 am

Hi!
Are there any other replacements/escapes/removals? Or what's the PHP function that does it?
Yes, depending on the visitor language, the language's include file can define a "i18n_filename" array that contains replacements.

The PHP function serendipity_makeFilename() performs the replacements.
Also, are these escape/replace/remove rules same for any fields in permalinks and for all permalinks?
Not sure what you mean by this, but the function is used for every permalink that is generated by the system, i.e. article links, category links, author links...?

HTH,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

peter.kehl
Regular
Posts: 5
Joined: Tue Apr 01, 2014 8:18 am

Re: Automated functional testing

Post by peter.kehl » Mon May 05, 2014 8:27 am

I’ve updated the test framework for Serendipity 2.0. I’ve added some more tests. It only performs optimistic tests - no attempt of SQL injection etc.
- login/logout as a specified user
- create a user with a given access (editor, chief or admin) with
-- random realname, username, password and
-- some random settings (wysiwyg, language etc. - only radio and dropdown ones for now)
- log in as that newly created user
- post an entry (with random title, body and extended)
- verify an entry: title, body, extended, author link
- get and change global preferences (only freetype ones for now - e.g. permalinkStructure)
See https://code.google.com/p/selite/wiki/S ... yFramework.

In case you've installed Firefox extensions form SeLite, you'll need to install SelBlocksGlobal 1.01. You won't get it automatically updated yet, because it wasn't reviewed by Mozilla yet (but it was accepted by their automated validation). You need to download it from https://addons.mozilla.org/en-US/firefo ... /versions/

I’m curious about your feedback.

User avatar
garvinhicking
Core Developer
Posts: 30020
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Automated functional testing

Post by garvinhicking » Mon May 05, 2014 10:49 am

Hi!

That sounds great! I'll try to find time to check this out in the next days, and setup the framework.

Thanks,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

Post Reply