Change merged, give it some 30minutes to propagate everwhere. Resolving this, feel free to reopen in case of things not functioning as expected. Thanks everyone!

Change merged, give it some 30minutes to propagate everwhere. Resolving this, feel free to reopen in case of things not functioning as expected. Thanks everyone!

I 've also performed a NOOP deployment from deploy1003 today, worked slowly (20minutes) due to having to build the images, but otherwise OK)

No significant changes in the last month per

The situation is indeed known, see also T309772, T357950. Some efforts did happen to modernize the codebase, however, as far as I know, they haven't materialized into something yet (at least that can be recommended). I should note that service-template-node and service-runner are related but also distinct. service-template-node was meant to be what the name suggests. Just a template on how to structure code to use the service-runner framework. @sbassett is correct in characterizing it as dated and unmaintained. However, it isn't in itself plagued by security issues, it's the outdated packages defined in package.json (and their dependency trees) that are. The actual template code is barely 700 lines and it's meant as a template with best practices (well, not anymore given it's dated and unmaintained) in using service-runner.

Good point, do we have some more information why the automatic rollback didn't happen/failed?

In T370118#10016683, @Novem_Linguae wrote:

There's only 18 bots on the list at https://radar.cloudflare.com/traffic/verified-bots. Hopefully that isn't a sign of a slow or difficult application process.

Move to this server from deploy1002 scheduled for Monday 2024-07-29 09:00 UTC

Switching to low as all we can do now is wait.

OK, I 've group them into 1, named it Wikimedia Citation Bot, submitting the two different User-Agent headers from the links above. as well as a 2 match patterns that would match always, that is ZoteroTranslationServer/WMF and Citoid/WMF. I 've also provided a link to https://www.mediawiki.org/wiki/Citoid, copying to a short description field the first sentence.

Just armed keyholder, everything looks ok right now. I 'll send a notification to wikitech-l and engineering in slack for a deployment server move. Not much different from what we do for the switchover.

In T369898#9990749, @Ottomata wrote:

The number of resource_change and resource_purge events can get extremely high, spiking at 10k req/sec at times

I'm curious about the the problem that this causes. Too many jobs inserted for job queue to handle quickly enough? Too many purge requests at once?

@ppelberg, @DLynch @zoe. The verified bot form requires entering some input we need your help on.

OK, thanks I can see that too now thanks.

Adding some more info, I 've went to https://dash.cloudflare.com/?to=/:account/:zone/security/bots with a personal free account I have and of course there is no section to tell them about my bot as the blog suggests. Maybe an account with more privileges than a free account is required.

There is already T370118 for this and discussion is ongoing, I suggest to close this as a duplicate of that task and continue there.

Couple of notes here:

Moving from SRE to serviceops-radar and subscribing the people that can approve this (same as in T339102). On the SRE side, it's not difficult to implement this.

Moving from SRE to serviceops-radar and subscribing the people that can approve this (same as in T339102). On the SRE side, it's not difficult to implement this.

What @Legoktm suggsted. If you have already a JSON input for that command and expect back an SVG (it looks this way judging from https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Chart/+/refs/heads/master/cli/src/command.ts), it's way better architecturally to expose an HTTP endpoint via a nodejs service, feeding the JSON via an HTTP POST and get back the SVG and insert in whatever content you were planning to insert it to. It can probably even be done async via the JobQueue if you don't want it in the parsing/rendering request hot path. So, my suggest would be to keep the CLI part for quick dev/debugging, but also add an express dependency (or even better service-runner, with the caveat that effort is undergoing to modernize it) and expose the functionality over an HTTP route, enable the pipeline and get a proper nodejs service that can be monitored and reasoned with all the standard tooling we have.

T361724 is a followup (and I think the only one) of this one, so I guess it's best to not resolve yet.

In T368892#9961399, @Jdforrester-WMF wrote:

In T368892#9955701, @akosiaris wrote:

I am gonna be bold and lower this to "High".

UBN per https://www.mediawiki.org/wiki/Phabricator/Project_management is

Unbreak Now! – Something is broken and needs to be fixed immediately, setting anything else aside. This should meet the requirements for issues that hold the train.

Per https://grafana.wikimedia.org/d/FEkiKFqVk/wikifunctions, v1/evaluate sees traffic on the order of 0.1 to 0.2 requests per second and apparently per the description and comments in this tasks only a, currently not well estimated/unknown (correct me if I am wrong), ratio of those is affected. This doesn't look like something that needs to be fixed immediately, settings anything else aside. Nor is it holding the train of course.

Something like 80% of the functionality of Wikifunctions has been offline for over a week now. We don't understand what is actually broken (it seems to be between MW and the k8s cluster, or in handling the request at the boundary, or otherwise), and its errors are challenging to parse. I think this definitely counts as UBN still.

In T364400#9926454, @Joe wrote:

In T364400#9780622, @BBlack wrote:

In T364400#9779996, @hnowlan wrote:

Could we implement this remapping at the ATS layer rather than the Apache one, in a manner that would mean that when we need to cache we only need to store each effective URL once?

It would be preferable to not do so. The caching gains would be minimal, but more importantly: we hope to minimize the details of the application layer that are spread into the cache configuration (there will always be necessary cases, but the more we avoid it, the easier things are in the future).

Even more to @BBlack's comment, I would just have apache funnel anything under /api it receives to an endpoint in mediawiki, and do routing there.

All fluentbit images have (once more) been delete from the registry using https://wikitech.wikimedia.org/wiki/Docker-registry#Deleting_images

My git grep above was wrongly also matching the deploy-service user that is being indeed used in a number of cases, not just the group. The merged patch drops just the group and it's correct. I 'll resolve this.

In T360403#9953720, @hashar wrote:

The sync-prod-k8s step went from ~ 420 seconds to 180 seconds at some point between May 27th and May 29th:

While it's a bit early to gauge this:

In T368238#9955846, @elukey wrote:

To keep archives happy - this is the result after some days:

I'd like to lower down the concurrency again to see if we get more benefits.

I am gonna be bold and lower this to "High".

For posterity's sake, a summary follows:

Hasn't this already been done in T355020 ?

Apologies, I failed to anticipate that consequence, I 've merged a change to remove deploy1003 from the list of scap masters.

I am resolving the task given comments from 4 years ago. However, repeating that the functionality added in the course of this task 4 years ago is going to be removed since it's unused and causes maintenance burden.

Summarizing from a discussion in #wikimedia-tracing for posterity's sake.

4 years later, we don't see any data flowing in the kafka topic created back then. This feature apparently has never been used. But it is costing us in maintenance efforts as the image is on buster and we wanna to remove those images from the registry. Hence, after some discussions in #wikimedia-serviceops IRC channel, we have decided to disable the functionality from api-gateway and delete the fluentbit docker image from our repo as this pipeline is the only user of it. If anyone ever reaches this task and comment and is interested in the functionality implemented during work on this task, it can always be resurrected, assuming it's properly resourced.

Host is imaged, rest of the work is ongoing in T364417

• python3-imagecatalog published and gerrit repo updated
• php72 component made conditional

I 've applied the role and now working through packaging python3-imagecatalog for bullseye

For posterity's sake

In T361728#9934556, @Mvolz wrote:

Latest graph o fail: https://grafana-rw.wikimedia.org/d/NJkCVermz/citoid?orgId=1&refresh=5m&from=1719399913415&to=1719401713415&forceLogin=&var-dc=codfw+prometheus%2Fk8s&var-service=citoid&viewPanel=46

In T328036#9930685, @Jgiannelos wrote:

After some back and forth with kiwix folks it looks like end of July is reasonable to keep the MCS endpoints available for mw-offliner.
They have already migrated to other endpoints and they just finish some details.

T352956 is related (possibly a duplicate) and I 've mulling over it for a few months now. I think we need to have a larger in person discussion regarding this. There's some things I wanna understand on the kubernetes side before we move forward. I 'll send invites.

Just to point out that this is probably not from the network. We don't have networking rate limiting in either of these machines (nor actually anywhere) and 5MB/s is less than 5% of the capacity of a 1Gbps link, which is the lowest common denominator in our infrastructure.

In T364126#9915798, @Platonides wrote:

The description describes CP3 as used to «automatically prefetch top-ranked search results when the user views a Google search result page»

while https://developer.chrome.com/blog/private-prefetch-proxy/ states

Note: At this moment, to allow other sites to preload navigations through Google servers, users need to select the "Extended preloading" mode in Chrome's preload settings. We are looking for interested parties as a catalyst for further improvements to this initial approach.

Thanks for this and thanks for documenting the selection process in T362999. It's probably worth it to update the summary of that task with a quick note about the conclusion and chosen solution.

Patch merged and deployed.

In T367510#9902362, @dcausse wrote:

In T367510#9902307, @akosiaris wrote:

I do have one question though. Why 4 weeks retention? Is there some business reason or could it be dropped to a smaller duration?

we need 4 weeks to be able to backfill after an import, from the time the wikidata dump process starts, the time required to shuffle the data around (compression, hdfs-rsync to hdfs) and til the end of the import into blazegraph, see the initial lag column in T241128 for past import times, perhaps 3weeks would be manageable but we went to 4 weeks to have extra room.

Yeah, I confirm. The older hosts in the clusters, kafka-main[12]00[0-5] have 2TB free space left, so 100GB isn't an issue. The newer hosts have smaller disks (budget reasons) but they aren't in service yet.

I 've removed

In T367544#9898401, @JMeybohm wrote:

builder-envoy-03.packaging.eqiad1.wikimedia.cloud

Any objections to just remove the VM since we moved to (re-)packaging upstream (https://wikitech.wikimedia.org/wiki/Envoy#Building_envoy_for_WMF)?

In T367544#9897657, @Jelto wrote:

packager02.packaging.eqiad1.wikimedia.cloud

According to the etherpad upgrade docs this host is used to build the etherpad Debian package. I also used the host in the past to build the etherpad package. The dedicated host is used because "etherpad builds fetches npm modules during the build time".

In T364900#9884119, @Mvolz wrote:

I deployed a change today for this. It is not everything we hoped, alas :).

We don't actually seem to log info, only warns. However, it may look like we were because of the presence of NOTICE levels in logstash. Which brings me to a second issues, which is: these are fact NOTICES caused by parsing failure of what are actually WARNS. Some of them are just total garbage: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2024.06.12?id=TmlhDJAB_bxzc-Pm7TGI. Some of these NOTICES are a single line of a stack trace. Guessing from this one I am thinking... the inclusion of the html of the response in the message of the error might be the cause of this. It's maybe actually not sanitising it and it's escaping it some how? Yikes :P And then we're getting like 20 NOTICEs for parts of the response, each separately. I.e. https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2024.06.12?id=TdVkDJABDD1VxFBSzVVI

In Kubernetes Special Interest Group, we recently re-evaluated the approach of running Docker outside of Kubernetes. The findings has been documented at https://wikitech.wikimedia.org/wiki/Docker, the discussion is at T363558.

Given the above, I am gonna close this as Declined in the interest of not having lingering tasks, but feel free to reopen.

This is a first. We never had to implement something like that in the past. Historically, we did have some ingress ratelimiting functionality in RESTBase (it ended up abandonware) and we very recently did add some rate limiting functionality to the service-mesh, but it's internal requests only. Egressing rate-limiting functionality from our infrastructure has never been implemented, to my knowledge at least.

Thanks for the historical aspect @Dzahn. Given that perspective and the fact the double redirect isn't apparently considered a problem, my opinion is that it is probably prudent to keep the redirect as it was asked in T264797.

And I am still not sure.

I am not sure what this task asks to be honest. Care to add a bit more information as to what the problem is?

In T365265#9858663, @Clement_Goubert wrote:

In T365265#9858506, @akosiaris wrote:

In T365265#9858015, @Joe wrote:

A few thoughts on this:

1. I think using daemonsets is a better option than a deployment. It will ensure UDP fire-and-forget communication (which is in itself scary in k8s with all the layers of indirection) happens locally to the physical node.

Just to note that we don't need this if we want to keep the current status quo. We are in a fire and forget situation right now. Everything UDP fires-and-forgets to statsd.eqiad.wmnet (with hardcoded IP to avoid DNS requests). If we want to increase the reliability of that pathway, we should at least have a conversation as to why that is needed (e.g. are we experiencing issues with missing metrics currently?)

I am still unclear on when metrics are sent directly to statsd.eqiad.wmnet or to the prometheus-statsd-exporter through envoy. Is this a transitional state towards using prometheus to collect metrics and both are required for now, or are some metrics sent through one path and others through the second? I think the scope of this task is to solve the second path only (MediaWiki -> statsd-exporter -> prometheus).

In T365265#9858015, @Joe wrote:

A few thoughts on this:

1. I think using daemonsets is a better option than a deployment. It will ensure UDP fire-and-forget communication (which is in itself scary in k8s with all the layers of indirection) happens locally to the physical node.

kafka-main1010, after 2 rounds of imaging (1 with the normal recipe and 1 with the reuse recipe) imaged successfully. I am resolving this. Thanks for all the work everyone!

Hmm, this complicates things.

In T364900#9838429, @Mvolz wrote:

Splitting from the main thread, to inquire about this specifically:

In T362379#9807771, @akosiaris wrote:

In T362379#9805889, @akosiaris wrote:

I 'll have a deeper look tomorrow.

urldownloaders don't have the visibility needed to look at URLs since most sites are accessed over HTTPS. So the only thing they do see is URL domains, not paths. We need Citoid to log out what requests it sees.

In T362379#9805685, @Mvolz wrote:

I'm implementing logging in citoid now, but - don't we mostly need to know the domain, not the path, anyway? As it lets us sort by hosts that are blocking us? I don't think we actually need the full path...

In T364253#9849840, @Sfaci wrote:

I have never seen 301 when requesting anything to any AQS services. I have no idea about why that happened. In my case, all request responses include envoy as the server value even when a there was a hit on the cache.

kafka-main1009 is successfully imaged fully

In T363212#9842874, @akosiaris wrote:

The fail for kafka-main1009 is expected with the current recipe btw. Let me have a quick look.

The fail for kafka-main1009 is expected with the current recipe btw. Let me have a quick look.

In T366094#9841984, @CDanis wrote:

In T366094#9841558, @Stashbot wrote:

Mentioned in SAL (#wikimedia-sre) [2024-05-29T11:23:04Z] <akosiaris> T366094 re-undeploy otel-collector, it being around increased traffic to the API >50%

FWIW I think this traffic increase will not be an issue once we move to the new three control plane nodes

In T366094#9840665, @CDanis wrote:

I 've gone ahead and created the following dashboard today T366094

In T363212#9839469, @Dzahn wrote:

@akosiaris re: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1035769/1/modules/profile/data/profile/installserver/preseed.yaml

I think since that is Bash globbing and not regex you can't use the round brackets as in "kafka-main10(09|10)". Or at least it's the only line in preseed.yaml that does that.

In T363212#9829434, @Jclark-ctr wrote:

I was able to correct kafka-main1010 issue for dhcp but image fails still

@akosiaris did you have this issue with other servers?

This no longer needed. Thanks to the work of @Joe and others, copy by url is now asynchronous and no longer suffers from this.