Follow-up from IRC: Wikimedia uses the Hosted RPKI, but we assume the ARIN portal just doesn't support anything else than ROAs. There is an ASPA record for AS11358, whose ASN is controlled by ARIN, and we think they either use Hybrid RPKI, where ARIN still hosts the RPKI objects through their Repository Publication Service. Technically, Krill could both act as the CA and create ASPA records, and it is possible that AS11358 and others are doing this.

sessionstorage04 is no longer.

Couldn't upgrade Buster to 4.x, because there are no packages in buster-wikimedia. Installing Cassandra was a rather interesting process.

I didn't get a response in -sre, but Andrew has provided me with extra information.

Puppet fails to install the Cassandra instance:

Error: 'install -o cassandra -g cassandra -m 750 -d /var/lib/cassandra/data' returned 1 instead of one of [0]
Error: /Stage[main]/Cassandra/Cassandra::Instance[default]/Exec[install-/var/lib/cassandra/data]/returns: change from 'notrun' to ['0'] failed: 'install -o cassandra -g cassandra -m 750 -d /var/lib/cassandra/data' returned 1 instead of one of [0] (corrective)
Error: 'install -o cassandra -g cassandra -m 750 -d /var/lib/cassandra/data' returned 1 instead of one of [0]
Error: /Stage[main]/Cassandra/Cassandra::Instance[default]/Exec[install-/var/lib/cassandra/data]/returns: change from 'notrun' to ['0'] failed: 'install -o cassandra -g cassandra -m 750 -d /var/lib/cassandra/data' returned 1 instead of one of [0] (corrective)

Had to delete sessionstorage05 (bookworm) due to T357791, will replace with a bullseye instance for Cassandra

Great, can't ssh into my new instance:

$ ssh deployment-sessionstore05.deployment-prep.eqiad1.wikimedia.cloud
Connection closed by UNKNOWN port 65535

@Jgiannelos hey! Is deployment-restbase-bullseye (created by you last year) ready to take over the work from restbase04? Other than changing the references to restbase04 in Horizon hiera and LabsServices.php, and in the changeprop Chart (deployment--charts), it should be possible to switch, although the restbase service is not listening to port 7231 on -bullseye - any idea what's wrong?

@hnowlan I see you have created deployment-maps-master02. Other than possibly replacing the old master in https://github.com/wikimedia/maps-kartotherian-deploy/blob/master/scap/environments/beta/targets, is there anything needed before deleting master01?

As soon as the above changes have been merged, urldownloader03 can be deleted.

@BTullis I see you have created deployment-snapshot05 (Bullseye), although this new host was not part of https://gerrit.wikimedia.org/r/c/operations/dumps/scap/+/1008451, and neither is it part of the mediawiki-installation dsh group. Do we have to add snapshot05 to your 'scap' repository, as well or is it fine to just add it to the dsh group?

Given that Puppet does not have a flag to stop the periodic MediaWiki jobs, I had to disable Puppet on mwmaint03 and kill the jobs myself (just like the DC switchover cookbooks do). Can be re-enabled as soon as mwmaint02 is gone (deleted + removed from dsh groups).

It looks like the shellbox container is broken on the new host:

root@deployment-shellbox01:~# /usr/bin/docker run --rm=true  --env-file /etc/shellbox/env -p 8081:8081 -v shellbox:/etc/shellbox  -v /run/shared:/run/shared  -v /srv/shellbox/config/:/srv/app/config  -v /srv/shellbox/src:/srv/app/src --name spftest docker-registry.wikimedia.org/wikimedia/mediawiki-libs-shellbox:2024-06-13-133425-video --nodaemonize
[20-Jul-2024 14:43:13] ERROR: unable to bind listening socket for address '/run/shared/fpm-www.sock': Permission denied (13)
[20-Jul-2024 14:43:13] ERROR: unable to bind listening socket for address '/run/shared/fpm-www.sock': Permission denied (13)
[20-Jul-2024 14:43:13] ERROR: FPM initialization failed
[20-Jul-2024 14:43:13] ERROR: FPM initialization failed

deployment-parsoid14 has been installed with a Bullseye image.

Upgrade to bullseye/bookworm blocked due to T332015. @MoritzMuehlenhoff, can I help you to get poolcounter-prometheus-exporter imported to bullseye and/or bookworm (preferably both)?

^ after merging this change, deployment-push-notifications01 can be replaced with a Bookworm instance.

mediawiki11 and mediawiki12 are no longer in use, but still receive scap deployments. As soon as the two changes above have been merged, we can delete these instances.

@Jgiannelos I see you have created mobileapps02 with a Bullseye image. Is mobileapps01 ready for removal?

Done (volume has been deleted as well)

Done :)

deployment-jobrunner04 has been shut down. As soon as https://gerrit.wikimedia.org/r/1055394 and https://gerrit.wikimedia.org/r/1055412 are merged, we can delete that instance.

Instance is offline, seems to be superseded by deployment-changeprop-1.deployment-prep.eqiad1.wikimedia.cloud per T357476#9540192. @Urbanecm_WMF, do you agree we can delete deployment-docker-cpjobqueue01?

Instance does not exist anymore?

After merging https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1055306, the irc.beta.wmflabs.org RRset can be removed, and the floating IP can be removed from deployment-ircd03 as well; if you want to test the IRC server, you can run irssi on a Cloud VPS instance.

Loop fixed by setting profile::base::remove_python2_on_bullseye: false on prefix level, also done in production.

role::mw_rc_irc seems to work fine on a Bullseye box, except for a loop.

Relevant: T127717#9671526 (i.e. can be deleted without issues, but preferably only before other instances are deleted)

Thanks for your help, Riccardo! Given current time constraints, I'm afraid most of this work will take multiple months, but nevertheless, to see whether kea_python still works with the Kea packages provided by Debian, I felt it was time to bite the bullet and build the bindings manually.

Thank you for your reply! My comments:

In T351418#9703216, @Volans wrote:

[...]

1. The current setup that by default doesn't offer an IP to DHCP requests is by design, in response to data loss on servers that rebooted into PXE by error (the force PXE bit didn't get cleared by the BIOS or similar), see T251416 for context. This is easily fixable having a Netbox field that is empty by default and the reimage/dhcp cookbooks will set it to the PXE image to be installed next and will reset it after the DHCP step is done. The automation should surely not give PXE fields if that field is not set, probably not even an IP for safety reasons (to be discussed).

As I understand it, no server in production VLANs (that is: starting with {analytics,private,public} - excluding frack infrastructure?) should rely on DHCP for any purpose other than reimaging, because the IPv4 address will be set statically in d-i. For that reason, I can see why we would like to refuse DHCP requests if no syslinux path is provided by NetBox. I wouldn't classify it as a security measure against malevolent administrators, but rather as a failsafe to mitigate the impact of operator error.

@ayounsi and I have discussed my first findings, and we thought it made sense to share them here.

In T127717#9671526, @Andrew wrote:

In T127717#9671489, @Southparkfan wrote:

In T127717#9671034, @Andrew wrote:

@Southparkfan We're trying to reduce use of Buster in cloud-vps, and two servers in 'auditlogging' are running Buster: syslog-server-04 and syslog-client04. My recollection is that they're redundant now that server-05 and client05 exist (and are running bookworm) -- is that right? Can the 04 VMs be removed?

The purpose on having syslog servers on multiple operating systems is to verify compatibility. As you might have seen, sometimes, rsyslog requires OS-specific changes to work properly.

If you don't mind potentially breaking Buster compatibility in the future, or if we should remove support right away, then these servers are OK to go.

That's a good point. We'll save these for a bit later in the Buster deprecation cycle. Thanks!

In T127717#9671034, @Andrew wrote:

@Southparkfan We're trying to reduce use of Buster in cloud-vps, and two servers in 'auditlogging' are running Buster: syslog-server-04 and syslog-client04. My recollection is that they're redundant now that server-05 and client05 exist (and are running bookworm) -- is that right? Can the 04 VMs be removed?

Haven't made a lot of progress on this, unfortunately. Scheduled for April.

I'll work on this.

Production migration from the gnutls driver to the openssl driver can be tracked in T324623.

Alternative to consider: injecting REDIRECTs for traffic meant for a VIP. See the second section at http://www.linuxvirtualserver.org/docs/arp.html. I haven't tested it and it requires some sort of Netfilter implementation on the realservers, but it avoids MTU-related issues (when tunneling traffic). Nevermind, ARP problem is solved at Wikimedia by not annoucing ARP. MTU is a challenge when using any type of encapsulation (in this case IPIP), but that's a different issue :)

In T336428#8844099, @cmooney wrote:

Ok I think I see what the issue is. Looking at the kernel docs they state that "the max value from conf/{all,interface}/rp_filter is used when doing source validation on the {interface}."

This effectively means that this setting:

net.ipv4.conf.all.rp_filter = 1

Nullifies the per-interface setting on eno12399np0:

net.ipv4.conf.eno12399np0.rp_filter = 0

(...)

I have expanded https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Auth_logging. The 'known limitations' section shows there is enough work to do, but to avoid a never ending task, I am fine with resolving this task when T127717#8505600 has been applied to Cloud VPS. I find the lack of monitoring to be a blocker too, though.

Standalone puppetmasters are also affected by this Git update:

$ git push -f project_puppetmaster HEAD:production
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote: fatal: detected dubious ownership in repository at '/var/lib/git/operations/puppet'
remote: To add an exception for this directory, call:
remote: 
remote: 	git config --global --add safe.directory /var/lib/git/operations/puppet

I have tested https://gerrit.wikimedia.org/r/c/operations/puppet/+/865731 by using rsyslog-openssl on one syslog client and one syslog server running buster + one syslog client and one syslog server running bullseye. All works as expected.

Status: we chose #3 (Let's Encrypt via acme-chief). We've gotten stuck on a bug in the gnutls driver for rsyslog: T324623

Background: for T127717, we went with Let's Encrypt certificates. Unlike the rather simple chain of trust for the Puppet CA (leaf certificate -> root certificate (Puppet CA)), Let's Encrypt certificates have an intermediate certificate in between. 'Because TLS' (certificates are terrible, I know), the clients need to receive all certificates but the root certificate (because that is in /etc/ssl/certs/ca-certificates.crt).

@Andrew and I have spent this evening on the initial set up of two WMCS-wide syslog servers. Those work fine. However, this setup is broken for all Cloud VPS instances that do not use the central puppetmaster.

In T36738#8106918, @Krinkle wrote:

@Southparkfan Thanks. Your patch avoids use of $wgRequest which unblocks removal of that variable from core. However, this task represents the work for T32956 and would not be resolved unless other forms of global state are avoided as well. Whether we obtain WebRequest from $wgRequest or RequestContext::getMain() is a minor detail from this task's perspective.

I understand, eliminating $wgRequest was low-hanging fruit here. DI is still better than either using ::getMain().

I'm happy to accept it, but keep the task and checkbox open after this lands.

If you'd like to resolve the use of global WebRequest in ::inDebugMode(), I can recommend two approaches to try.

Now that we're at it, I'll try to get a durable solution; after all, I would like to reduce technical debt, not move it somewhere else. Assistance is needed to get me starting here, though :-). Your advice is welcome.

1. Perhaps create a setDebugMode() method marked @internal that we'd call from load.php and possibly OutputPage.php. It could take WebRequest and Config as parameter. Doing this would actually highlight an issue which is that we appear to be reading the cookie even when on load.php which is potentially a problem even today. The cookie should instruct OutputPage to make load.php?debug=true requests, there is no need for it to read it directly. If it does, it might actually poison the cache.

Assumptions

• load.php (RL\Context) only needs to know if ?debug=<value> exists; cookies and config don't matter here.
• index.php emits HTML elements (sourcing resources from load.php) that may or may not contain ?debug, depending on the sixth argument of ResourceLoader::makeLoaderQuery(). This entry point is not interested in the presence of a ?debug parameter - inverse of load.php.

In T127717#8092122, @Andrew wrote:

Sorry for the slow response here. I also don't see a clear way to provision those certs, so I think relying on source IP is probably good for this pass. It's already the case that we can't fully 'trust' log messages originating from within cloud-vps projects; I suspect that the risk of a ddos attack is already present even if we have certified logs.

Please lmk if I'm missing a more obvious threat.

Cool! Not sure what the relationship with a DDoS is, though :). If you have time, you can review the patch above. I wasn't too sure about the locations of the default hiera: some things have to be defined in cloud.yaml, others in common/, ...

(wrong task)

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/815776/ would alleviate the 'usage of $wgRequest global' concern, but a second pair of eyes is needed:

• Since [\MediaWiki\ResourceLoader\ResourceLoader]::inDebugMode() is a static function with neither WebRequest exposed via $this->getRequest() or [\MediaWiki\ResourceLoader\Context]->getRequest() available. Unless extensions can/should fetch a 'ResourceLoader object' (and therefore convert this function into a non-static one), I'm not sure how to refrain from this "last restort".
• Apart from OutputPage and OutputPageTest, [\MediaWiki\ResourceLoader\ResourceLoader]::makeCombinedStyles() is the only caller for OutputPage::transformCssMedia(), hence I couldn't refrain from using RequestContext::getMain() here either.
• According to T165176, RequestContext::getMain() can cause side-effects? Will that affect the ResourceLoader Context too?