Page MenuHomePhabricator
Create Task
Maniphest T365998

Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad
Open, MediumPublic
Actions

Description

Moved forward one week: 2024-07-23, 15:00 UTC
on this rack: https://netbox.wikimedia.org/dcim/racks/91/

  • db1195 - s1
  • db1202 - s7
  • db1203 - s8
  • db1205 - backup
  • moss-be1003
  • ms-be1075
  • an-presto1015
  • an-worker1156
  • an-worker1146
  • kafka-jumbo1015
  • kafka-stretch1002
  • elastic1100
  • elastic1101
  • elastic1102
  • wdqs1016
  • dse-k8s-worker1008
  • ml-serve1008
  • kubernetes1025
  • kubernetes1026
  • kubernetes1052
  • kubernetes1053
  • kubernetes1054
  • kubernetes1055
  • kubernetes1056
  • mw1496

Teams Involved: Data Persistence, Data Platform, Search, Machine Learning, Service Ops

Expected outage: 15-30 minutes

Please use the below sheet to detail any actions that are required in advance of the work:

https://docs.google.com/spreadsheets/d/1pLPpzGBmdExXxQ_0_eGXpO0VlUU5oPKZy-_KViMSwuM

Details

Other Assignee
MatthewVernon

Related Objects
Search...

Event Timeline

• ABran-WMF created this task.May 27 2024, 1:22 PM
• ABran-WMF updated Other Assignee, added: MatthewVernon.
• ABran-WMF mentioned this in T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2.
• ABran-WMF added projects: Data-Persistence, DBA.May 27 2024, 3:03 PM
• ABran-WMF moved this task from Triage to Ready on the DBA board.May 28 2024, 7:48 AM
• ABran-WMF added a project: SRE-swift-storage.May 28 2024, 10:24 AM
MatthewVernon updated the task description. (Show Details)May 28 2024, 10:34 AM

swift-wise, just need to check the cluster's happy afterwards.

moss-be1003 is part of the apus Ceph cluster, which should be in production by end of this quarter (i.e. before this work is due to happen), and will need a bit of care. Should just be a case of putting it into maintenance mode beforehand, but it's 1/3 of the cluster capacity.

cmooney claimed this task.Jun 5 2024, 7:16 PM
cmooney triaged this task as Medium priority.
cmooney updated the task description. (Show Details)
• jcrespo subscribed.EditedJun 12 2024, 9:39 AM

db1205 is the secondary media backups metadata db server, usually just a standby to db1204. Unless it is the active server because the primary is unavailable, it just has to be checked that replication restarts correctly after maintenance.

• ABran-WMF updated the task description. (Show Details)Tue, Jun 25, 9:23 AM
• Marostegui closed subtask T368374: Move one host temporarily to m2 as Resolved.Wed, Jun 26, 4:21 AM
• Marostegui updated the task description. (Show Details)Mon, Jul 1, 5:01 AM
• Marostegui updated the task description. (Show Details)Fri, Jul 5, 5:04 AM
ayounsi moved this task from Backlog to This quarter on the netops board.Mon, Jul 8, 7:24 AM
cmooney updated the task description. (Show Details)Mon, Jul 8, 4:07 PM
ops-monitoring-bot added a comment.Tue, Jul 9, 2:53 PM

Icinga downtime and Alertmanager silence (ID=6a298ae5-e736-4051-8220-9ec4f352950a) set by cmooney@cumin1002 for 0:40:00 on 1 host(s) and their services with reason: prep JunOS upgrade lsw1-e3-eqiad

lsw1-e3-eqiad.mgmt
ops-monitoring-bot added a comment.Tue, Jul 9, 3:00 PM

Icinga downtime and Alertmanager silence (ID=39fcbcd0-8c16-4208-ac06-f4b442e55a54) set by cmooney@cumin1002 for 0:30:00 on 4 host(s) and their services with reason: JunOS upgrade lsw1-e3-eqiad

lsw1-e3-eqiad,lsw1-e3-eqiad IPv6,ssw1-e1-eqiad.mgmt,ssw1-f1-eqiad.mgmt
ops-monitoring-bot added a comment.Tue, Jul 9, 3:03 PM

Icinga downtime and Alertmanager silence (ID=2a5cb43e-793c-4103-9499-369354315479) set by cmooney@cumin1002 for 0:40:00 on 27 host(s) and their services with reason: JunOS upgrade lsw1-e3-eqiad

an-presto1010.eqiad.wmnet,an-worker1154.eqiad.wmnet,backup1009.eqiad.wmnet,cephosd1003.eqiad.wmnet,db[1192,1198-1199,1204].eqiad.wmnet,druid1010.eqiad.wmnet,dse-k8s-worker1006.eqiad.wmnet,elastic[1093-1095].eqiad.wmnet,kafka-jumbo1012.eqiad.wmnet,kafka-stretch1001.eqiad.wmnet,kubernetes[1047-1051,1061].eqiad.wmnet,ml-serve1006.eqiad.wmnet,ms-be1074.eqiad.wmnet,mw[1491-1493].eqiad.wmnet,wdqs1015.eqiad.wmnet
Stashbot added a comment.Tue, Jul 9, 3:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-09T15:04:20Z] <topranks> rebooting lsw1-e3-eqiad to install updated JunOS version T365998

cmooney added a comment.Tue, Jul 9, 3:23 PM
This comment was removed by cmooney.
Clement_Goubert subscribed.Thu, Jul 18, 2:24 PM
Stashbot added a comment.Thu, Jul 18, 2:47 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-18T14:47:54Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'T365998 - depooling db1195 - s1 db1202 - s7 db1203 - s8', diff saved to https://phabricator.wikimedia.org/P66816 and previous config saved to /var/cache/conftool/dbconfig/20240718-144754-arnaudb.json

• ABran-WMF added a comment.Thu, Jul 18, 2:49 PM

data-persistence hosts handled, ready whenever you are @cmooney

Stashbot added a comment.Fri, Jul 19, 12:23 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-19T12:23:20Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'T365998 - depooling db1195 - s1 db1202 - s7 db1203 - s8', diff saved to https://phabricator.wikimedia.org/P66843 and previous config saved to /var/cache/conftool/dbconfig/20240719-122320-arnaudb.json

• ABran-WMF added a subscriber: • Marostegui.EditedFri, Jul 19, 12:24 PM

@Marostegui and I will be absent on tuesday, hosts have been depooled and are ready.

cmooney added a comment.Fri, Jul 19, 5:30 PM
In T365998#9998292, @ABran-WMF wrote:

@Marostegui and I will be absent on tuesday, hosts have been depooled and are ready.

Thanks @ABran-WMF, enjoy your weekend and/or break!

bking subscribed.Mon, Jul 22, 10:38 PM

elastic110[0-2] are banned and ready , as is wdqs1016.

Stashbot added a comment.Tue, Jul 23, 10:39 AM

Mentioned in SAL (#wikimedia-operations) [2024-07-23T10:39:11Z] <claime> Cordoning kubernetes1025.eqiad.wmnet kubernetes1026.eqiad.wmnet kubernetes1052.eqiad.wmnet kubernetes1053.eqiad.wmnet kubernetes1054.eqiad.wmnet kubernetes1055.eqiad.wmnet kubernetes1056.eqiad.wmnet mw1496.eqiad.wmnet for T365998

Stashbot added a comment.Tue, Jul 23, 1:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-23T13:05:17Z] <claime> Cordoning dse-k8s-worker1008.eqiad.wmnet for T365998

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits