Missing GROUP BY clause or aggregate function

Found a bug? Tell us!!
ads
Regular
Posts: 93
Joined: Sun Oct 29, 2006 11:39 am

Missing GROUP BY clause or aggregate function

Post by ads »

The following query raises an SQL error:

Code: Select all

SELECT DISTINCT ep_sticky.value AS orderkey,
         ep_cache_extended.value AS ep_cache_extended,              
        ep_cache_body.value AS ep_cache_body,
                            e.id,
                            e.title,
                            e.timestamp,
                            e.comments, 
                            e.exflag,   
                            e.authorid, 
                            e.trackbacks,
                            e.isdraft,   
                            e.allow_comments,
                            e.last_modified, 
                            a.realname AS author,
                            a.username AS loginname,
                            a.email
                        FROM
                            serendipity_entries AS e
                            LEFT JOIN serendipity_authors a
                                ON e.authorid = a.authorid
                            LEFT JOIN serendipity_entrycat ec
                                ON e.id = ec.entryid
                            LEFT JOIN serendipity_category c
                                ON ec.categoryid = c.categoryid
                            INNER JOIN serendipity_entrytags AS entrytags ON e.id = entrytags.entryid  LEFT OUTER JOIN serendipity_entryproperties ep_cache_extended
                                                          ON (e.id = ep_cache_extended.entryid AND ep_cache_extended.property = 'ep_cache_extended')
         LEFT OUTER JOIN serendipity_entryproperties ep_cache_body
                                                          ON (e.id = ep_cache_body.entryid AND ep_cache_body.property = 'ep_cache_body')
         LEFT OUTER JOIN serendipity_entryproperties ep_access
ON (e.id = ep_access.entryid AND ep_access.property = 'ep_access')
         LEFT JOIN serendipity_entryproperties ep_sticky
                                                    ON (e.id = ep_sticky.entryid AND ep_sticky.property = 'ep_is_sticky') LEFT JOIN serendipity_authorgroups AS acl_a
                                           ON acl_a.authorid = 0
                                    LEFT JOIN serendipity_access AS acl_acc
                                           ON (    acl_acc.artifact_mode = 'read'
                                               AND acl_acc.artifact_type = 'category'
                                               AND acl_acc.artifact_id   = c.categoryid
                                              )
                             WHERE e.timestamp >= 1238194800 AND e.timestamp <= 1238277599 AND isdraft = 'false' AND e.timestamp <= 1269812100 AND (entrytags.tag IN ('blog','categories','46-Kino'))  AND  (ep_access.property IS NULL OR ep_access.value = 'public')  AND     (
                                         c.categoryid IS NULL
                                         OR ( acl_acc.groupid = 0)
                                         OR ( acl_acc.artifact_id IS NULL
                                            )
                                       )
                              HAVING count(entrytags.tag) = 3
                             ORDER BY orderkey ASC, timestamp DESC
                              LIMIT 3
PostgreSQL is way more picky about correct and repeatable query results than MySQL. All result columns must either be GROUPed or appear in an aggregate function (DISTINCT as example).
ads
Regular
Posts: 93
Joined: Sun Oct 29, 2006 11:39 am

Re: Missing GROUP BY clause or aggregate function

Post by ads »

Ah, forgot the SQL error:
2010-03-28 23:32:23 CEST ERROR: column "ep_sticky.value" must appear in the GROUP BY clause or be used in an aggregate function
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Missing GROUP BY clause or aggregate function

Post by garvinhicking »

Hi!

I hate this pickyness about postgresql, because it's pendantic. If I don't group by the statement, at least mysql properly realizes that due to other join/where conditions, the selected key cannot possibly contain something else than a specific row.

I don't really know how to rewrite this query for postgresql, and I hate dealing with this specific pickyness*. Input appreciated. :-)

Best regards,
Garvin

* I had to deal with this on another project a few days ago and it drove me insane, so I'm not really keen to dive into this yet once again on my own ;-)
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
ads
Regular
Posts: 93
Joined: Sun Oct 29, 2006 11:39 am

Re: Missing GROUP BY clause or aggregate function

Post by ads »

garvinhicking wrote:I hate this pickyness about postgresql, because it's pendantic.
It's SQL-standard and it provides predictable results ;-)

http://stackoverflow.com/questions/1023 ... n-group-by


If I don't group by the statement, at least mysql properly realizes that due to other join/where conditions, the selected key cannot possibly contain something else than a specific row.
MySQL also allows queries where the result can differ between two calls of the same query on the same data.


I don't really know how to rewrite this query for postgresql, and I hate dealing with this specific pickyness*. Input appreciated. :-)
DISTINCT all columns maybe?


* I had to deal with this on another project a few days ago and it drove me insane, so I'm not really keen to dive into this yet once again on my own ;-)
If you need PostgreSQL-support, just ask me ;-)
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Missing GROUP BY clause or aggregate function

Post by garvinhicking »

Hi!

I haven't properly grasped the concept of Distinct, especially because AFAIR it differs on MySQL and PostgRESL a lot in the outcome, and I'd hate to fork one of the most central s9y queries for abstraction, as it could also impact a lot of plugins...

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
ads
Regular
Posts: 93
Joined: Sun Oct 29, 2006 11:39 am

Re: Missing GROUP BY clause or aggregate function

Post by ads »

If the result differs across two databases then one database is doing it wrong. Guess which one ;-)
By the way, if the MySQL database is running with Ansi-SQL enabled, this query will probably fail too.

Ok, to move forward: where can i find this query and what is the expected result? Since the current query might return ambiguous results, can you specify the expected results in some words? I will try to rewrite the query.
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Missing GROUP BY clause or aggregate function

Post by garvinhicking »

Hi!

Okay, the query above should do this:

Return a list of entries from serendipity_entries that only contains an entryid once. Each entry that is returned needs to match the requirements:

- timestamp inside the valid range
- not a draft

Several tables are joined to restrict the list of possible entries further:

- is associated with a specific list of possible tags, with at least one of those possible tags matching (through DB table serendipity_entrytags)

- serendipity_entrycat (1:n) & serendipity_category (1:1) for the list of possible categories (even though the complete category list is fetched at a later time, this join is only used for ACL privilege checks)

- serendipity_entryrpoperties, serendipity_access for the ACL privileges restriction based on any category that is part of the subjoin above

Other joins performed to get additional metadata for the entry:

- serendipity_entryproperties for the "sticky" attribute of an entry
- serendipity_authors for the author of an entry (1:1)
- serendipity_entryproperties for a cached version of the entry body
- serendipity_entryproperties for a cached version of the entry extended body

The list of possible entries is restricted by only allowing those that have exactly 3 tags assigned (matching the list of possible tags)

Maybe it is sufficient to use:

Code: Select all

SELECT DISTINCT ep_sticky.value AS orderkey,
         ep_cache_extended.value AS ep_cache_extended,             
        ep_cache_body.value AS ep_cache_body,
                            e.id,
instead of:

Code: Select all

SELECT DISTINCT e.id,
         ep_sticky.value AS orderkey,
         ep_cache_extended.value AS ep_cache_extended,             
         ep_cache_body.value AS ep_cache_body,
to accomodate the error situation? Because what's needing to be distinct is the entryid, not the sticky value?

The query can be found in include/functions_entries.inc.php at around line 356, in the serendipity_fetchEntries() function.

Changing the code would be a change frmo

Code: Select all

      $select_key = "{$cond['distinct']}
                    {$cond['addkey']}

                    e.id,
to

Code: Select all

      $select_key = "{$cond['addkey']}
                    {$cond['distinct']} e.id,
...?

I'm not a 100% sure what kind of side-effects this might have on other situations, so you might want to test if after this change sticky-entries, cached entries and read-privileges are still properly evaluated on the frontend?

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
ads
Regular
Posts: 93
Joined: Sun Oct 29, 2006 11:39 am

Re: Missing GROUP BY clause or aggregate function

Post by ads »

The attached patch should fix the issue. You probably want to use this patch for all non-mysql drivers.

I checked the output of my blog with and without the patch and both are identical.
Attachments
s9y-group_by_2010-04-12.patch.gz
(420 Bytes) Downloaded 442 times
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Missing GROUP BY clause or aggregate function

Post by garvinhicking »

Hi!

Hm, the problem is that keys inside the SELECT key statement can be modified by any plugins. They could list more or less than the current hardocded list; especially, if the entryproperties plugin is not set, the ep_sticky key isn't even there.

Were you maybe able to test what I suggested earlier, that a reordering of the DISTINCT'ed keys might help? I don't have a pgsql installation to test this with currently...

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
ads
Regular
Posts: 93
Joined: Sun Oct 29, 2006 11:39 am

Re: Missing GROUP BY clause or aggregate function

Post by ads »

*hrmpf*

No, i did not test your idea, basically because you will run into the same problems: all columns must be aggregated or grouped.
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Missing GROUP BY clause or aggregate function

Post by garvinhicking »

Hi!

Hm, that really sucks. I mean, "entry.id" is already distinct. Why do we need all those columns? There can't ever be a dataset that has the same entry.id but a different entry.title...?!

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
ads
Regular
Posts: 93
Joined: Sun Oct 29, 2006 11:39 am

Re: Missing GROUP BY clause or aggregate function

Post by ads »

e.id is DISTINCT, yes. My fault. The DISTINCT is not the real problem, the HAVING() is.
You can use DISTINCT without GROUP BY.

Can you rewrite the HAVING into a subselect with COUNT() inside? Something like:

Code: Select all

WHERE (SELECT COUNT(e2.id)
          FROM serendipity_entries AS e2
          LEFT JOIN serendipity_authors a2
    ON e2.authorid = a2.authorid
  LEFT JOIN serendipity_entryproperties ep_sticky2
    ON (e2.id = ep_sticky2.entryid AND ep_sticky2.property = 'ep_is_sticky')) = 3
(Don't know if i got all required tabled for the subselect, you probably know better.)
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Missing GROUP BY clause or aggregate function

Post by garvinhicking »

Hi!

Ah, somewhere in this thread I forgot about the HAVING subpart. Hm, problem is I did not write that exact extension to the freetag plugin that performs this query, so I don't even really know what that query is used for exactly. I'm wondering if we can't put this thing into the WHERE part already?!

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
ads
Regular
Posts: 93
Joined: Sun Oct 29, 2006 11:39 am

Re: Missing GROUP BY clause or aggregate function

Post by ads »

My WHERE demo should be a good start.
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Missing GROUP BY clause or aggregate function

Post by garvinhicking »

Hi!

Serendipity tries to not use Subqueries at all, to be as compatible to any systems not supporting subqueries, like MySQL 4.0, and I believe also SQLite has some subquery-issues.

It might be that some parsing of queries (especially for "unlimited queries" for pagination etc.) might break, when subqueries are introduced at this central point of the s9y core.

So maybe it can be coded in a way that would only be triggered when the freetag plugin is executed AND the DB is postgresql. Sadly I currently don't have the time to inspect possible subqueries, but I'll try to keep this on my list to investigate as soon as I find the time to do some digging!

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
Post Reply