Page MenuHomePhabricator
Create Task
Maniphest T98706

internal_api_error_DBQueryError: Database query error while (mass) deleting file over api
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

I get such error since a few days when doing mass deletions:

API request failed (internal_api_error_DBQueryError): [0b608d20] Database query error

Details

SubjectRepoBranchLines +/-
Reduce RC table contention for file deletionmediawiki/coremaster+19 -7
Fixed $flags bit operation precedence fail in User::loadFromDatabase()mediawiki/corewmf/1.26wmf8+1 -1
Fixed $flags bit operation precedence fail in User::loadFromDatabase()mediawiki/coremaster+1 -1
Reduce page deletion lock contentionmediawiki/coremaster+27 -2
Customize query in gerrit

Related Objects

Event Timeline

Steinsplitter created this task.May 10 2015, 4:54 PM
Steinsplitter raised the priority of this task from to Medium.
Steinsplitter updated the task description. (Show Details)
Steinsplitter added projects: acl*sre-team, Commons.
Steinsplitter subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 10 2015, 4:54 PM
Steinsplitter updated the task description. (Show Details)May 10 2015, 4:55 PM
Steinsplitter set Security to None.
Glaisher added a project: Wikimedia-production-error.May 10 2015, 4:56 PM
Glaisher subscribed.
Steinsplitter added a comment.May 10 2015, 4:56 PM

And sometimes file links remaining blue, after purging by hand they become red. Not sure if this is related.

Steinsplitter moved this task from Incoming to Backlog on the Commons board.May 10 2015, 4:57 PM
hoo subscribed.May 10 2015, 5:00 PM

Backtrace (page/ revision id removed):

2015-05-10 16:52:01 mw1208 commonswiki exception INFO: [0b608d20] /w/api.php   DBQueryError from line 1276 of /srv/mediawiki/php-1.26wmf4/includes/db/Database.php: A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: SELECT  rev_id,rev_page,rev_text_id,rev_timestamp,rev_comment,rev_user_text,rev_user,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,rev_sha1,rev_content_format,rev_content_model,page_namespace,page_title,page_id,page_latest,page_is_redirect,page_len,user_name  FROM `revision` INNER JOIN `page` ON ((page_id = rev_page)) LEFT JOIN `user` ON ((rev_user != 0) AND (user_id = rev_user))  WHERE page_id = '...' AND rev_id = '....'  LIMIT 1   FOR UPDATE
Function: Revision::fetchFromConds
Error: 1205 Lock wait timeout exceeded; try restarting transaction (10.64.16.29)

#0 /srv/mediawiki/php-1.26wmf4/includes/db/Database.php(1231): DatabaseBase->reportQueryError()
#1 /srv/mediawiki/php-1.26wmf4/includes/db/Database.php(1756): DatabaseBase->query()
#2 /srv/mediawiki/php-1.26wmf4/includes/Revision.php(400): DatabaseBase->select()
#3 /srv/mediawiki/php-1.26wmf4/includes/Revision.php(342): Revision::fetchFromConds()
#4 /srv/mediawiki/php-1.26wmf4/includes/Revision.php(312): Revision::loadFromConds()
#5 /srv/mediawiki/php-1.26wmf4/includes/Revision.php(151): Revision::newFromConds()
#6 /srv/mediawiki/php-1.26wmf4/includes/page/WikiPage.php(620): Revision::newFromPageId()
#7 /srv/mediawiki/php-1.26wmf4/includes/page/WikiPage.php(661): WikiPage->loadLastEdit()
#8 /srv/mediawiki/php-1.26wmf4/includes/page/WikiPage.php(2796): WikiPage->getContent()
#9 /srv/mediawiki/php-1.26wmf4/includes/FileDeleteForm.php(193): WikiPage->doDeleteArticleReal()
#10 /srv/mediawiki/php-1.26wmf4/includes/api/ApiDelete.php(174): FileDeleteForm::doDelete()
#11 /srv/mediawiki/php-1.26wmf4/includes/api/ApiDelete.php(61): ApiDelete::deleteFile()
#12 /srv/mediawiki/php-1.26wmf4/includes/api/ApiMain.php(1131): ApiDelete->execute()
#13 /srv/mediawiki/php-1.26wmf4/includes/api/ApiMain.php(469): ApiMain->executeAction()
#14 /srv/mediawiki/php-1.26wmf4/includes/api/ApiMain.php(442): ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/php-1.26wmf4/api.php(90): ApiMain->execute()
#16 /srv/mediawiki/w/api.php(3): include()
#17 {main} {"private":false}
Steinsplitter raised the priority of this task from Medium to High.May 11 2015, 10:31 AM

API request failed (internal_api_error_DBQueryError): [e6308580] Database query error
API request failed (internal_api_error_DBQueryError): [c88d0938] Database query error
API request failed (internal_api_error_DBQueryError): [a3914641] Database query error
API request failed (internal_api_error_DBQueryError): [c6308edf] Database query error
API request failed (internal_api_error_DBQueryError): [f765ae82] Database query error
(.....)

Aklapper added a project: MediaWiki-libs-Rdbms.May 19 2015, 12:55 PM
Jdforrester-WMF added a project: Multimedia.May 28 2015, 9:10 PM
Umherirrender renamed this task from internal_api_error_DBQueryError: Database query error to internal_api_error_DBQueryError: Database query error while (mass) deleting file over api.May 29 2015, 6:11 PM
Umherirrender added subscribers: aaron, • Springle.
gerritbot subscribed.May 29 2015, 7:34 PM

Change 214684 had a related patch set uploaded (by Aaron Schulz):
Reduce page deletion lock contention

https://gerrit.wikimedia.org/r/214684

gerritbot added a project: Patch-For-Review.May 29 2015, 7:34 PM
gerritbot added a comment.Jun 2 2015, 1:20 PM

Change 214684 merged by jenkins-bot:
Reduce page deletion lock contention

https://gerrit.wikimedia.org/r/214684

aaron mentioned this in rMW7b59841f43b9: Reduce page deletion lock contention.Jun 2 2015, 1:20 PM
• chasemp assigned this task to aaron.Jun 2 2015, 5:22 PM
• chasemp subscribed.

@aaron, did your patch address this successfully?

I'm not sure where this issue stands, can you throw some commentary in here?

• chasemp edited projects, added DBA; removed Patch-For-Review, acl*sre-team.Jun 2 2015, 5:22 PM
aaron added a comment.Jun 2 2015, 5:30 PM

The query mentioned by hoo will no longer be "FOR UPDATE", so that error can't happen after the change is deployed. It would still be good to keep an eye on any page table related locks.

aaron closed this task as Resolved.Jun 2 2015, 5:31 PM
• Forrestbot added projects: MW-1.26-release, WMF-deploy-2015-06-09_(1.26wmf9).Jun 2 2015, 6:31 PM
• Forrestbot subscribed.
gerritbot added a comment.Jun 2 2015, 6:39 PM

Change 215379 had a related patch set uploaded (by Aaron Schulz):
Fixed $flags bit operation precedence fail in User::loadFromDatabase()

https://gerrit.wikimedia.org/r/215379

gerritbot added a project: Patch-For-Review.Jun 2 2015, 6:39 PM
gerritbot added a comment.Jun 2 2015, 6:49 PM

Change 215379 merged by jenkins-bot:
Fixed $flags bit operation precedence fail in User::loadFromDatabase()

https://gerrit.wikimedia.org/r/215379

aaron mentioned this in rMW833bdbab37cd: Fixed $flags bit operation precedence fail in User::loadFromDatabase().Jun 2 2015, 6:49 PM
gerritbot added a comment.Jun 2 2015, 6:57 PM

Change 215388 had a related patch set uploaded (by Aaron Schulz):
Fixed $flags bit operation precedence fail in User::loadFromDatabase()

https://gerrit.wikimedia.org/r/215388

hashar moved this task from Untriaged to Mar 2019 / 1.33wmf20-23 on the Wikimedia-production-error board.Jun 2 2015, 7:49 PM
gerritbot added a comment.Jun 3 2015, 12:18 AM

Change 215388 merged by jenkins-bot:
Fixed $flags bit operation precedence fail in User::loadFromDatabase()

https://gerrit.wikimedia.org/r/215388

aaron mentioned this in rMW69814403017b: Fixed $flags bit operation precedence fail in User::loadFromDatabase().Jun 3 2015, 12:18 AM
hashar moved this task from Mar 2019 / 1.33wmf20-23 to Resolved on the Wikimedia-production-error board.Jun 5 2015, 9:45 AM
Jdforrester-WMF added a project: WMF-deploy-2015-05-27_(1.26wmf8).Jun 10 2015, 2:11 AM
Steinsplitter reopened this task as Open.Jun 13 2015, 4:58 PM

https://commons.wikimedia.org/w/index.php?title=Commons:Administrators%27_noticeboard&oldid=163300665#Errors.2C_errors.2C_and_more_errors

Krenair subscribed.Jun 13 2015, 5:06 PM
In T98706#1362985, @Steinsplitter wrote:

https://commons.wikimedia.org/w/index.php?title=Commons:Administrators%27_noticeboard&oldid=163300665#Errors.2C_errors.2C_and_more_errors

2015-06-13 01:01:16 mw1223 commonswiki exception INFO: [5fb8a4d0] /w/api.php   DBQueryError from line 1276 of /srv/mediawiki/php-1.26wmf9/includes/db/Database.php: A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: INSERT  INTO `recentchanges` (rc_timestamp,rc_namespace,rc_title,rc_type,rc_source,rc_minor,rc_cur_id,rc_user,rc_user_text,rc_comment,rc_this_oldid,rc_last_oldid,rc_bot,rc_ip,rc_patrolled,rc_new,rc_old_len,rc_new_len,rc_deleted,rc_logid,rc_log_type,rc_log_action,rc_params,rc_id) VALUES (redacted)
Function: RecentChange::save
Error: 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.16.29)

#0 /srv/mediawiki/php-1.26wmf9/includes/db/Database.php(1231): DatabaseBase->reportQueryError()
#1 /srv/mediawiki/php-1.26wmf9/includes/db/Database.php(2117): DatabaseBase->query()
#2 /srv/mediawiki/php-1.26wmf9/includes/changes/RecentChange.php(306): DatabaseBase->insert()
#3 /srv/mediawiki/php-1.26wmf9/includes/logging/LogEntry.php(629): RecentChange->save()
#4 /srv/mediawiki/php-1.26wmf9/includes/page/WikiPage.php(2867): ManualLogEntry->publish()
#5 /srv/mediawiki/php-1.26wmf9/includes/db/Database.php(3515): Closure$WikiPage::doDeleteArticleReal()
#6 /srv/mediawiki/php-1.26wmf9/includes/db/Database.php(3724): DatabaseBase->runOnTransactionPreCommitCallbacks()
#7 /srv/mediawiki/php-1.26wmf9/includes/FileDeleteForm.php(199): DatabaseBase->commit()
#8 /srv/mediawiki/php-1.26wmf9/includes/api/ApiDelete.php(174): FileDeleteForm::doDelete()
#9 /srv/mediawiki/php-1.26wmf9/includes/api/ApiDelete.php(61): ApiDelete::deleteFile()
#10 /srv/mediawiki/php-1.26wmf9/includes/api/ApiMain.php(1130): ApiDelete->execute()
#11 /srv/mediawiki/php-1.26wmf9/includes/api/ApiMain.php(469): ApiMain->executeAction()
#12 /srv/mediawiki/php-1.26wmf9/includes/api/ApiMain.php(442): ApiMain->executeActionWithErrorHandling()
#13 /srv/mediawiki/php-1.26wmf9/api.php(88): ApiMain->execute()
#14 /srv/mediawiki/w/api.php(3): include()
#15 {main} {"private":false}
McZusatz subscribed.Jun 15 2015, 9:09 AM

Translations always trigger two edits at the same time. Just now one of them failed with

Saving the translation failed: [42491262] Database query error

leaving the translation in some "undefined" state.

gerritbot added a comment.Jun 15 2015, 11:58 PM

Change 218557 had a related patch set uploaded (by Aaron Schulz):
Reduce RC table contention for file deletion

https://gerrit.wikimedia.org/r/218557

gerritbot added a comment.Jun 16 2015, 12:12 AM

Change 218557 merged by jenkins-bot:
Reduce RC table contention for file deletion

https://gerrit.wikimedia.org/r/218557

aaron mentioned this in rMW4a3002063c94: Reduce RC table contention for file deletion.Jun 16 2015, 12:12 AM
• Forrestbot added a project: WMF-deploy-2015-06-16_(1.26wmf10).Jun 16 2015, 1:00 AM
PiRSquared17 awarded a token.Jun 18 2015, 5:32 AM
Aldnonymous subscribed.Jun 18 2015, 5:33 AM
Ankry subscribed.Jun 18 2015, 5:33 AM
• jcrespo moved this task from Triage to Blocked external/Not db team on the DBA board.Jun 26 2015, 10:43 AM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptJun 26 2015, 10:43 AM
TTO subscribed.Jun 27 2015, 12:18 PM

Fixed? Can close?

Legoktm removed a subscriber: • Forrestbot.Jun 29 2015, 5:49 PM
Wieralee subscribed.Jun 29 2015, 8:41 PM
aaron closed this task as Resolved.Jul 2 2015, 11:37 PM
Steinsplitter added a comment.Jul 5 2015, 9:35 AM

Error while moving the page.
A detailed description of the error is shown below:
API request failed (internal_api_error_DBQueryError): [e96e342a] Database query error <i>at Sun, 05 Jul 2015 09:34:10 GMT</i> <u>served by mw1189</u>

Is this related?

hoo added a comment.Jul 5 2015, 10:44 AM
In T98706#1427117, @Steinsplitter wrote:

Error while moving the page.
A detailed description of the error is shown below:
API request failed (internal_api_error_DBQueryError): [e96e342a] Database query error <i>at Sun, 05 Jul 2015 09:34:10 GMT</i> <u>served by mw1189</u>

Is this related?

No, it's not, that was about the image table… you can open a separate bug for that, if you want.

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits