In T364690#9898810, @Soda wrote:

I don't think the code here is duplicating any work, every time a page is saved or a related page is updated, there is a chance that the file does not exist or has changed and the sequence that was previously constructed is invalid. This needs to be accounted for by ProofreadPage, which it does by checking for the existence of the file, computing the list of pages, and then re-rendering the sequence (or displaying an error if the sequence is invalid).

This is the functionality of a separate <pagelist/> tag, launched when saving the Index page.
The <pages/> tag only needs existing pages and a pre-compiled <pagelist/> list of pages from the database. Running book indexing and file scanning in the <pages/> tag does nothing, but pointless duplicate re-compilations, broken pages (as discussed) and load on the server.

In T364690#9896615, @Soda wrote:

In T364690#9896515, @Vladis13 wrote:

And why, on each rendering, does the code recompile the index (mapping), although it does not change, and scan the pdf to Commons, which is not needed, since only text from the database is used? And besides the optimization problem, this leads to the discussed breakdown of the mass of pages.

As I said before the compilation does not happen on each render. It happens when a template transcluded/related to the page changes OR when the page is purged OR when the page is edited (basically ProofreadPage and the underlying parser treats the <pages /> tag like a series of templates and updates are triggered whenever mediawiki decides that the templates need updating).

In T364690#9896511, @Soda wrote:

In T364690#9896454, @Vladis13 wrote:

In T364690#9895464, @Vladis13 wrote:

In T364690#9798210, @Soda wrote:

In T364690#9793607, @Vladis13 wrote:

In T364690#9793548, @Uzume wrote:

In T364690#9793303, @Vladis13 wrote:

This shouldn't happen because the <pages/> mechanism works with text in Page and Index NS. Whereas the mediafile is in File NS. These are independent spaces.

But they aren't independent because the Index and Page pages depend on the File (and the Page pages also depend on the Index).

No, <pages/> only uses existing text in Page NS, file is absolutely not needed for this and is not used, why. Page and Index NS only use file during index page creation and visually assist users in proofreading. So, <pages/> should continue to display that text without breaking pages when the file is deleted.

That's incorrect, ProofreadPage depends on the File namespace to build the pagelist, figure out valid Page: namespace pages and determine the transclusion order of the pages.

That's what I said. The file is needed only at the modes of creating the pagelist (index) and proofreading.

The <pages /> tag should only take the text of pages from the local Wikisource database, according to a list of a pre-generated index. Why would it access an image file in the Commons repository?.. Sounds like nonsense. If this is indeed the case, then this is a major non-optimization of the code.

The index is computed based on the image file, or through manually linked pages as a fallback. I do not see why this is a "major non-optimization" or "nonsense".

You state that with EVERY reading of a page of a work by site readers, the index is re-generated, scanning the file in the Commons repository. This is a time-consuming and resource-intensive operation.

Then it should be like this:

• Only when saving the index page (Index NS) does the extension analyze the arguments of the <pagelist /> tag (which defines the mapping: pdf page number = book page number or literal label), creates an index (mapping, it’s short, banal "[{1: 'image'}, {2: 4}, {3: 5}, etc. according to the number of pdf pages]"), and is saves in the database. This is all.
• When a reader opens a page of a work, the <pages/> tag works only with database 1) takes the pages mapping from the database, takes the range of page numbers according to the arguments in the tag, 2) takes the text of the pages from the database, cuts out the sections specified in the arguments, renders html: gluing the texts of the pages, inserting between them the page numbers of the book with links to the pages of the Page NS, according to the mapping (index). This is all.

Compare the resource costs of generating an index from scratch for each read, and one only when saving the index page. I think this is obvious.

The recompilation happens on page save or link update on related pages, not on every render.

And why, on each rendering, does the code recompile the index (mapping), although it does not change, and scan the pdf to Commons, which is not needed, since only text from the database is used? And besides the optimization problem, this leads to the discussed breakdown of the mass of pages.

In T364690#9895464, @Vladis13 wrote:

In T364690#9798210, @Soda wrote:

In T364690#9793607, @Vladis13 wrote:

In T364690#9793548, @Uzume wrote:

In T364690#9793303, @Vladis13 wrote:

This shouldn't happen because the <pages/> mechanism works with text in Page and Index NS. Whereas the mediafile is in File NS. These are independent spaces.

But they aren't independent because the Index and Page pages depend on the File (and the Page pages also depend on the Index).

No, <pages/> only uses existing text in Page NS, file is absolutely not needed for this and is not used, why. Page and Index NS only use file during index page creation and visually assist users in proofreading. So, <pages/> should continue to display that text without breaking pages when the file is deleted.

That's incorrect, ProofreadPage depends on the File namespace to build the pagelist, figure out valid Page: namespace pages and determine the transclusion order of the pages.

That's what I said. The file is needed only at the modes of creating the pagelist (index) and proofreading.

The <pages /> tag should only take the text of pages from the local Wikisource database, according to a list of a pre-generated index. Why would it access an image file in the Commons repository?.. Sounds like nonsense. If this is indeed the case, then this is a major non-optimization of the code.

The index is computed based on the image file, or through manually linked pages as a fallback. I do not see why this is a "major non-optimization" or "nonsense".

You state that with EVERY reading of a page of a work by site readers, the index is re-generated, scanning the file in the Commons repository. This is a time-consuming and resource-intensive operation.

In T364690#9798210, @Soda wrote:

In T364690#9793607, @Vladis13 wrote:

In T364690#9793548, @Uzume wrote:

In T364690#9793303, @Vladis13 wrote:

This shouldn't happen because the <pages/> mechanism works with text in Page and Index NS. Whereas the mediafile is in File NS. These are independent spaces.

But they aren't independent because the Index and Page pages depend on the File (and the Page pages also depend on the Index).

No, <pages/> only uses existing text in Page NS, file is absolutely not needed for this and is not used, why. Page and Index NS only use file during index page creation and visually assist users in proofreading. So, <pages/> should continue to display that text without breaking pages when the file is deleted.

That's incorrect, ProofreadPage depends on the File namespace to build the pagelist, figure out valid Page: namespace pages and determine the transclusion order of the pages.

That's what I said. The file is needed only at the modes of creating the pagelist (index) and proofreading.

In T364690#9798244, @Uzume wrote:

https://gerrit.wikimedia.org/r/1031631

It seems like that patch, when merged might solve the issue with finding <pagelist/> usage when the File media cannot be properly processed

The patch has not been accepted so far. Am I wrong, or will the broken pages and their tracking never be fixed?

In T275319#9814384, @cscott wrote:

This discussion risks going in circles. As I wrote previously in T275319#6884320:

zhwiki for example should have 4x the character limit if this is to be the new rule. Unlike what is claimed above, many of the performance metrics *do* scale with bytes rather than characters -- most wikitext processing is at some point regexp-based, and that works on bytes (unicode characters are desugared to the appropriate byte sequences), and of course network bandwidth, database storage size, database column limits, etc, all scale with bytes not characters. We should be careful before bumping the limit that we're not going to run into problems with database schema, etc.

And as @Reedy stated in T275319#9057445:

Timo's comment in T275319#7947012 is still relevant.

And at the same time, no one has declined this task, or said it won't ever be done. It requires planning, effort, testing and actual prioritisation by the people doing those activities etc.

It seems you have ignored my replies to this: T275319#7977747 and T275319#9057620. If don't notice the answers, the discussion really risks going in circles.

In T364690#9793548, @Uzume wrote:

In T364690#9793303, @Vladis13 wrote:

This shouldn't happen because the <pages/> mechanism works with text in Page and Index NS. Whereas the mediafile is in File NS. These are independent spaces.

But they aren't independent because the Index and Page pages depend on the File (and the Page pages also depend on the Index).

No, <pages/> only uses existing text in Page NS, file is absolutely not needed for this and is not used, why. Page and Index NS only use file during index page creation and visually assist users in proofreading. So, <pages/> should continue to display that text without breaking pages when the file is deleted.

In T364690#9788743, @Soda wrote:

This is expected if there is no file backing a index file, transclusions won't work.

This shouldn't happen because the <pages/> mechanism works with text in Page and Index NS. Whereas the mediafile is in File NS. These are independent spaces.

Example and addition: T343630

The bot made 4 new items, seems to work. I close the issue, reopen if the problem is not solved. Thanks for the answer.

ID is not returned in about 20-30% of cases, as I noticed.

In T275319#9057445, @Reedy wrote:

None of this is helping move the discussion forward.

Timo's comment in T275319#7947012 is still relevant.

And at the same time, no one has declined this task, or said it won't ever be done. It requires planning, effort, testing and actual prioritisation by the people doing those activities etc.

Let me answer the complexities listed in the T275319#7947012.

In T275319#9057297, @Alexey_Skripnik wrote:

Readers don't care directly about the weight of a webpage's HTML. What they care about is how quickly they can access the document in which they're interested. Fortunately, most (if not all) contemporary browsers support webpage streaming, which allows them to begin rendering the webpage before its HTML has completely downloaded. Therefore, the time to Largest Contentful Paint for a webpage, whether it's 0.23 Mb or 2.3 Mb, should be more or less the same. Or not?

In T275319#9057235, @stjn wrote:

(@Vladis13 please keep in mind https://www.mediawiki.org/wiki/Bug_management/Phabricator_etiquette as you are resorting to personal attacks and I have said nothing against your Wikisource which follows the Russian law on ‘extremist content’ to a tee.)

You call it a personal attack, although I only repeated your words "just your opinion". In addition, you reproach me personally with the rule of the Russian Wikisource, based on the policy of the Terms of Use of Wikimedia Foundation. Yes, this is all a personal attack, only yours.

In T275319#9054277, @stjn wrote:

Wikisource editors can absolutely split pages into smaller ones, since those longer pages have sections and subsections. The community choice might be not to split them, but that’s not necessarily a technical problem.

It is obvious from your words that you have not read the discussion and arguments above. Including that all the users who spoke here from different language projects of Wikisource spoke in favor of expanding the limit. Due to big problems in splitting of pages for editors and usability for readers.
The expansion of the limit will not cause any technical problems.
Your opinion is the only your opinion as a Wikipedia user. This topic doesn't concern you at all.

In T337996#8897836, @Xover wrote:

I also do not see any good reason why section editing would not work here.

I provided a link to the error.

In T337996#8897504, @Soda wrote:

@Vladis13 Did this work at some previous date ? (trying to figure out if this a recent regression or something that has been broken for a long time)

This was an issue well before 2018.

T321505

Now the script says "intersection" even when I use only one category:

pwb.py category -site:wikisource:ru -cat:"Дистрикт Галиция" remove -from:"Дивизия СС «Галичина»"  -simulate
Retrieving intersection of generators.
Retrieving 4 pages from wikisource:ru.

How to remove pages from a category by generator? The only way is to use the replace.py script with the -regex argument?
pwb.py replace -family:wikisource -lang:ru -cat:"ОУН-УПА" -regex '\[\[Категория:Украина\]\]\n?' ''

There is also confusion with the name of the argument.
The script (by page generator) execute for each page the remove -from:category_name command. In human terms, this means: remove this page from the category.
But it turns out that it completely clears the category, removing all pages, with one command without restrictions and the ability to filter the process. Then this command should be called as "clear category".
But the script already has a clean command, which has no description. What it does? Is this a duplicate of remove?

Hm. The category.py help at https://doc.wikimedia.org/pywikibot/stable/scripts/main.html#module-scripts.category also says "This script supports use of pywikibot.pagegenerators arguments."

All of these scripts are based on pywikibot.pagegenerators and pywikibot.bot, and use common generator arguments.
If this script is somehow strange, could you remove the list of generator commands from the script's help (pwb.py category -help) and from its page https://www.mediawiki.org/wiki/Manual:Pywikibot/category.py#Generators_and_filters_available ?

How to disable -always for other bulk edit commands?

I use the remove command, examples are given here T318239.

@Samwilson About listing participants. Personally, I do not look at such lists in e-book files at all, and skip the title pages when I read them.
Moreover, usually the users who performed OCR, importing and proofreading indicate themselves under meaningless pseudonyms. In this case, a link to the wikipage with its edit history is sufficient.

Thanks, I added class="ws-noexport" in the tag of this image of the page layout, so now its uploader is excluded from the list.

In T275319#6884320, @cscott wrote:

database storage size, database column limits, etc, all scale with bytes not characters. We should be careful before bumping the limit that we're not going to run into problems with database schema, etc.

The database schema do not require modification. The database text column is of type binary MEDIUMBLOB, GZIPed diffs are written there. The MEDIUMBLOB type can store up to 16Mb of data per entry, i.e. this is the diff size limit for any page edit (revision).
The number of page edits is not limited, all diffs are collected together only when the page is rendered. If we recall the words of the documentation "The compression ratio achieved on Wikimedia sites nears 98%." (DB doc), then it can be argued that the current database schema can store pages of infinite size.

Seems, yes. Thank you very much!
I checked the places where the bug could appear. It was massive, if the bug remained somewhere I didn't notice, I'll let you know.

@Ladsgroup, I closed it up there, was 2 weeks of the poll. All administrators, bureaucrat and active users voted. Consensus.

Here is the community consensus.

It seems to me that this option should not cause side problems. All important transclusions are on the users watchlists.

@Tacsipacsi, thanks. But you answered an old example from 2019. At the time, I fixed it, bypassing the problem by moving all inclusions to Template NS. Several times I moved the page through different namespaces, tried patrolling and bouncing back, changed the settings for page stabilization and transclusions in different ways... it didn't help.

We probably have 30% of the entire Wikisource (!!!) with warnings that pages need to be patrolled, or it will appear after any editing of the page. Because our main template for texts in Main NS is dependent with json settings and modules in the Mediawiki NS that can be edited.

Sure, calendar algorithms, like Julian day as well as Unix time are beyond the scope of ISO.
It seems to be correct to keep the date as given in the printed source (let it be Julian "44-03-15" BC), without the Gregorian conversion variations. E. g. Wikidata birthdays: d:Q1048, d:Q5592. Will the date format have a code (like "J") about the calendar being used? Current #time supports codes for multiple calendars, excluding Julian.

Is the date format will has a mark to indicate which the calendar used? As a kind of wiki extension to the EDTF format. Like, Julian date "J44-03-15".
Or will need to keep specifying an additional tag about the date calendar? Like a string description in the text next to the date, an additional parameter in templates, a switch / qualifier in the Wikidata properties.

The community of Python language hasn't heard of EDTF either. stackoverflow, docs.python.org, supported format-codes. The size of this community is almost larger than that of Javascript and Php (on which the Wiki is written). Pywikibot is written in Python.
So, can be sure that the generally programming languages will not adopt the standard in the coming years.

Thanks you.

Was Pwb 3.0.dev0 (this version number did not change until this year for several years, although Pwb itself was regularly updated, probably it was a bug) → 6.0.1
PyMySQL 0.9.3 → 1.0.2
Python 3.7.

I updated pywikibot and pymysql and it worked.
I think, need to add check for the version of pymysql to pywikibot/data/mysql.py, and revert the code for support version <1.0.0. Or at least a notification that bots developers need to update pymysql.

By the whois, the domain was paid until 2023, maybe the hosting too. It is a pity that its admins have not responding for about a year, there is no way to fix the site or find out the situation.

On Wikidata and https://en.wikisource.org/wiki/Help:Wikilivres written that the Wikilivres.org was closed in August 2019.

And this page was created by bureaucrat of ruWikisource today. https://ru.wikisource.org/wiki/%D0%91%D0%A1%D0%AD1/%D0%91%D0%B5%D0%BB%D0%BE%D1%80%D1%83%D1%81%D1%81%D0%B8%D1%8F How you can see, this also have the mark "unreviewed pages" with link to a page in NS Project that which technically can't be reviewed.

@Aklapper, yes this still wanted, But I don't understand why you write to me. Because I am not its developer, I have not even seen its code, and moreover, I have not used PHP for 5 years having switched to Python, and don't write MW-extentions.
The author abandoned support of the extension 3 years ago. This is all the information I can help.

Enabled. Although seems that the extension don't work ... But there is no time for testing and no support, and formally this is another task.

Seems, it's a bug in code. How can resolve or get around this issue?
Which code show the revision requirement for this NS?

@Reedy, could you resolve this bug or disable the revision requirement for these NS?

T226054 The problem is that FlaggedRevs is already included in MediaWiki and you need to confirm revisions on a mass of pages. But this is not possible because the confirmation button is disabled. Can to approve this message from one page, but it will appear again and again, since Mediawiki pages cannot be patrolled.
During the year there was no solution to this problem.

Also with NS MediaWiki.
I am administrator and created the MediaWiki:Encyclopedias_settings.json and MediaWiki:Wikiprojects settings.json pages, which are included in most ruWikisource pages of Main NS. Now, after edits on pages (seems, only new users' edits), appear the requirement to review these MediaWiki pages, but this imposible to do. Can to remove this message from one page, but it will appear again and again, since Mediawiki pages cannot be patrolled.

The hotfix works on broken Russian Wikisource images. https://ru.wikisource.org/wiki/%D0%A1%D1%82%D1%80%D0%B0%D0%BD%D0%B8%D1%86%D0%B0:%D0%95%D0%B2%D1%82%D0%B8%D0%B4%D0%B5%D0%BC_(%D0%9F%D0%BB%D0%B0%D1%82%D0%BE%D0%BD,_1878).pdf/57, https://ru.wikisource.org/w/index.php?title=%D0%A1%D1%82%D1%80%D0%B0%D0%BD%D0%B8%D1%86%D0%B0:%D0%A2%D0%B8%D0%BC%D0%B5%D0%B9_%D0%B8_%D0%9A%D1%80%D0%B8%D1%82%D0%B8%D0%B9_(%D0%9F%D0%BB%D0%B0%D1%82%D0%BE%D0%BD,_%D0%9C%D0%B0%D0%BB%D0%B5%D0%B2%D0%B0%D0%BD%D1%81%D0%BA%D0%B8%D0%B9).pdf/308

As far as I understand, this extension is not active and abandoned, because the author did not saw interest of community. I don't understand why my suggestion to call interest of community to it caused your questions.

@freephile, could you add your extention on https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2020/Wikisource?

I removed Wikipedia from the commit, remains only Wikisources there.

It do, I think. This problem was not until was disabled FlaggedRevs for NS "Wikisourse". And there don't using [[{{LOCALDAY}}. {{LOCALMONTHNAME}}]] [[{{LOCALYEAR}}]] {{#time}} unlike T119366.
Also, the unupdated version shows as stabilized: https://ru.wikisource.org/w/index.php?title=Заглавная_страница&stable=1. I tried to disable stabilisation for Main page, but problem still, because a transcludion can' be reviewed, as I wrote in this task.

On the Main page of https://ru.wikisource.org for unlogged readers still showing old News. It doesn't updating. (They placed in NS "Wikisource" and can't be revieved for show.) What to do?