Cycad network upgrade and outage

Service NOT available during period
There will be no network service in Cycad, She-Oak and Tuckeroo while an upgrade is performed. Please avoid the buildings where possible. For more information, please contact the Project Manager, Keryn Masters on 0458 585 405.
Service affected: Building Networks

Start: 2 Apr 2023, 10:00 AM
End: 2 Apr 2023, 9:00 PM
Login to request notification

VM Provisioning Unavailable

Service NOT available during period
Vendor security changes are causing some VM provisioning processes to fail. ITS Engineers are investigating.
Service affected: VPNext

Start: 31 Mar 2023, 7:28 AM
End: Unknown
Login to request notification

HAZARD AWARENESS - DC1 UPS Failure - On Mains Power Only

No impact on service during period
DC1 has lost UPS protection due to significant failure on all battery strings. There is NO impact at this stage to services, however the DataCentre is now on mains power only.

UPS Service Tech is on the way to site.
Service affected: Data Centre Network

Start: 10 Mar 2023, 9:30 AM
End: Unknown
Login to request notification

Call transfers failing

Reduced service delivery during period
Some phone calls are losing audio when being transferred. The customer will end up on a dead line.

Agents are advised to try warm transfer for the time being.
Update: 2 Feb 2023, 11:30 AM
Phone carrier has released a patch to try resolve the issue. Update: The issue continues to persist after the patch. Carrier investigating.
Service affected: Amazon Connect

Start: 23 Jan 2023, 1:00 PM
End: Unknown
Login to request notification

UQ Book It Booking Request Notification Issue

Reduced service delivery during period
Bookers are not receiving booking request notifications from the UQ Book It system. Other email notifications are still working such as accepted, rejected and cancelled.
Service affected: UQ Bookit

Start: 18 Jan 2023, 10:26 AM
End: Unknown
Login to request notification

Access to UQAPS Citrix only accessible via UQ Network or VPN

Reduced service delivery during period
Access to UQAPS / Citrix has been changed to be only accessible from the UQ Network / VPN for the time being.

At this stage there is no ETA when this will be re-enabled.
Service affected: UQAPS Citrix

Start: 13 Jan 2023, 3:42 PM
End: Unknown
Login to request notification

Prentice DC2 Rack B3 Power redundancy failure

No impact on service during period
A power rail has tripped RCD protection in Prentice DC2 Rack B3. All Equipment in this rack is running without power redundancy. The issue is being investigated by ITS engineers.
Update: 30 Sep 2022, 11:45 AM
All services are online from remaining power supply. ITS DC Engineers are investigating
Service affected: Data Centre

Start: 30 Sep 2022, 11:02 AM
End: Unknown
Login to request notification

Gerhamn Building 60 - Network redundancy reduced

No impact on service during period
Due to an un-resolved fault within the Network Access Layer, all access switches in building 60 are connected to a single Agg, therefore reducing network redundancy.
This will be in place until the fault can be found and rectified
Service affected: Building Networks

Start: 16 Sep 2022, 2:12 PM
End: Unknown
Login to request notification

WebPrism Slow Performance

Reduced service delivery during period
Intermittent issues will be experienced
Service affected: PrISM Identity Management

Start: 15 Aug 2022, 10:30 PM
End: Unknown
Login to request notification

Building 89f Boat Shed - No network services - No Power

Service NOT available during period
Due to flooding network services are currently not available
Service affected: Building Networks

Start: 1 Mar 2022, 11:34 AM
End: Unknown
Login to request notification

Apply ESXi 6.7 patches to PRN Nextgen Nutanix cluster

Resolved
Apply latest ESXi 6.7 patches to PRN Nextgen vSphere Datacenter - Prentice Nutanix cluster.
Resolution: 2 Apr 2023, 5:00 PM
Automatic resolution.
Service affected: VMware NextGEN PRN DC

Start: 2 Apr 2023, 9:00 AM
End: 2 Apr 2023, 5:00 PM
Login to request notification

Address Critical Vulnerability in the ITS Confluence Service

Resolved
A critical service vulnerability (Confluence Security Advisory 2022-06-02) has been identified to exist in the ITS Confluence Service. We need to apply the mitigation that Atlassian has provided.

More information about this vulnerability can be found at: https://confluence.atlassian.com/doc/confluence-security-advisory-2022-06-02-1130377146.html

This maintenance work is required to address and resolve this critical vulnerability.
Resolution: 31 Mar 2023, 6:00 PM
Automatic resolution.
Service affected: ITS Confluence

Start: 31 Mar 2023, 5:00 PM
End: 31 Mar 2023, 6:00 PM
Login to request notification

Disabling virtual machine logging (For Nutanix CVM) - Prn Nutanix cluster.

Resolved
Nutanix clusters on VMware ESXi 6.7U3i may experience stability issues
Products Affected:
Nutanix clusters running VMware ESXi 6.7U3 - 7.0U3

Severity: Critical


Resolution: 31 Mar 2023, 7:30 PM
Automatic resolution.
Service affected: VMware NextGEN PRN DC

Start: 31 Mar 2023, 5:00 PM
End: 31 Mar 2023, 7:30 PM
Login to request notification

Emergency reboot of Host in PRN NextGen DC. Refer to Description for VM Hosts require shutdown

Resolved
Reboot of ntnx-78qzfl2-a.inf.dc.uq.edu.au is required to resolve an issue causing the host to become unresponsive in vCenter server.

VM Hosts will need to be shutdown for the reboot, and service owners have been advised.


VMs which will be impacted -

DNS Name
corpusmate.compute.dc.uq.edu.au
fom-imu01.compute.dc.uq.edu.au
pf-arc-stl16.compute.dc.uq.edu.au
pf-ir.compute.dc.uq.edu.au
uq-fleetrmm-mgt.compute.dc.uq.edu.au
uq-mspapp03.compute.dc.uq.edu.au
auscontrol.compute.dc.uq.edu.au
auslegacy.compute.dc.uq.edu.au
auswekan.compute.dc.uq.edu.au
its-ds-sql16.compute.dc.uq.edu.au
auraps01....compute.dc.uq.edu.au
aurwcp01.compute.dc.uq.edu.au
im-duo-01.compute.dc.uq.edu.au
salt.compute.dc.uq.edu.au
uq-adstl07.uq.edu.au
uq-azmgmt01.test.ad.uq.edu.au
uq-azmgmt1.compute.dc.uq.edu.au
clag.compute.dc.uq.edu.au
crayola.compute.dc.uq.edu.au
gravitational.compute.dc.uq.edu.au
insist.compute.dc.uq.edu.au
kermit.compute.dc.uq.edu.au
ping.compute.dc.uq.edu.au
psyduck.compute.dc.uq.edu.au
qm-server3.compute.dc.uq.edu.au
rmuir-testbox.compute.dc.uq.edu.au
roof.compute.dc.uq.edu.au
roomba.compute.dc.uq.edu.au
ohna-db.compute.dc.uq.edu.au
baldonnell.inf.dc.uq.edu.au
ntnx-78qzfl2-a-cvm
ca-bahamas.compute.dc.uq.edu.au
cses-prod.compute.dc.uq.edu.au
ctmmfthub01.compute.dc.uq.edu.au
its-testlab3.compute.dc.uq.edu.au
ctmhubpd01.compute.dc.uq.edu.au
ajc.maths.uq.edu.au
uq-pcut-test.compute.dc.uq.edu.au
frontloader.compute.dc.uq.edu.au
smtp1.compute.dc.uq.edu.au
ise-pan
its-un-ucesql.compute.dc.uq.edu.au
sauron.compute.dc.uq.edu.au

Read more
Resolution: 31 Mar 2023, 3:45 PM
Automatic resolution.
Service affected: VMware NextGEN PRN DC

Start: 31 Mar 2023, 3:00 PM
End: 31 Mar 2023, 3:45 PM
Login to request notification

Reboot its-ds-sql16 - For Emergency Change Request #12351

Resolved
IS will reboot Reboot ntnx-78qzfl2-a.inf.dc.uq.edu.au to resolve an issue causing the host to become unresponsive in vCenter server. - Emergency Change Request #12351

Affected VM VM its-ds-sql16.compute.dc.uq.edu.au will be shutdown before the scheduled reboot at 3 pm.
Resolution: 31 Mar 2023, 3:15 PM
Automatic resolution.
Service affected: SQL 2017 Clustered Database Hosting

Start: 31 Mar 2023, 2:55 PM
End: 31 Mar 2023, 3:15 PM
Login to request notification

SI-net - Online Search Function Unavailable

Resolved
Search functionality from the banner at the top of the page will be unavailable.

Lookup search from withing individual pages will be available.
Resolution: 31 Mar 2023, 3:26 PM
services restarted as expected. Search functionality has been restored
Service affected: SI-net

Start: 31 Mar 2023, 2:55 PM
End: 31 Mar 2023, 3:26 PM
Login to request notification

Awareness Only - PRN NextGen resilance in reduced state

Resolved
The PRN NextGen environment is in a reduced state of resilance on the Nutanix cluster due to VMware bug affecting one host.

Currently in the process of shutting down VMs on the affected host in order to reboot and restore resilancy.
Resolution: 31 Mar 2023, 3:57 PM
Impacted host has been reloaded
Service affected: VMware NextGEN PRN DC

Start: 31 Mar 2023, 8:00 AM
End: 31 Mar 2023, 3:57 PM
Login to request notification

Q collection data service issue

Resolved
The storage infrastructure that provides back end delivery of older, larger files for UQ RDM and QRISCloud Q collections has been disrupted.

Access to older, less frequently accessed data may be impacted by this issue.

Technical staff are working with the storage vendor to restore full functionality.
Resolution: 30 Mar 2023, 1:15 PM
Aarnet fixed an issue on the link and storage services have been restored.
Update: 30 Mar 2023, 12:40 PM
The upstream network provider (AARNET) had determined that there is an active network fault in their services. This has caused cascading effects upon UQ infrastructure. AARNET NoC are working to correct the issues on their infrastructure. To resolve the cascading effects, UQ RCC staff are now bringing services back into production and a resolution time will be forthcoming.
Service affected: Giovanni SpectrumScale Cluster

Start: 30 Mar 2023, 10:59 AM
End: 30 Mar 2023, 1:15 PM
Login to request notification

Test failover CCR DC firewall

Resolved
Test failover CCR DC firewall
Resolution: 29 Mar 2023, 11:00 PM
Automatic resolution.
Service affected: NGDC Serverfarm Firewall

Start: 29 Mar 2023, 9:00 PM
End: 29 Mar 2023, 11:00 PM
Login to request notification

Herston CCR and HIRF network upgrade

Resolved
Full network outage in Herston CCR building and HIRF. Nurse call, PWD,
Resolution: 26 Mar 2023, 9:00 PM
Service affected: Building Networks

Start: 26 Mar 2023, 9:00 AM
End: 26 Mar 2023, 9:00 PM
Login to request notification

INFO ONLY - Internet Services to switch over to our secondary link for urgent maintenance

Resolved
INFO ONLY - Urgent maintenance is required on the GW10 internet connection to address port errors on our primary Aarnet Internet link.

Internet services shall be moved over to our GW20 Aarnet link to allow this work to be proceed.

There will be no service impact during this change. Redundant internet services will not be available during this change period.
Resolution: 25 Mar 2023, 8:00 PM
Automatic resolution.
Service affected: Internet Services UQ

Start: 25 Mar 2023, 7:00 PM
End: 25 Mar 2023, 8:00 PM
Login to request notification

Gatton Campus Power Outage - Network services impacted

Resolved
Network services have been impacted at Gatton campus due to a power loss in across the campus.

Outage started 4:03PM, network devices impacted came back online around 4:22PM.

P&F Have been advised.
Resolution: 24 Mar 2023, 4:22 PM
Automatic resolution.
Service affected: Building Networks

Start: 24 Mar 2023, 4:03 PM
End: 24 Mar 2023, 4:22 PM
Login to request notification

Weekly Production Migration for SI-net

Resolved
ENTSS Migrator will be migrating new codes changes to Production for SI-net applications

https://confluence.its.uq.edu.au/confluence/display/ESS/2023-03-21+Weekly+Production+Changes

If the change is simple enough then email approval will be sufficient. The email distribution should include Tech Masters, Functional Masters, Change Owner ( Func and Tech) and Manager to assess the impact before granting the Pre/approval.

If the change does not require an outage then seek the line manager for approval.

Refer to https://confluence.its.uq.edu.au/confluence/x/p220Dg

The Team Change Manager willl contain the Migrator name to migrat...e the changes into Production.

This service view will be marked to complete as soon as the migration completes.

Read more
Resolution: 23 Mar 2023, 7:30 AM
Automatic resolution.
Service affected: SI-net

Start: 23 Mar 2023, 7:00 AM
End: 23 Mar 2023, 7:30 AM
Login to request notification

Weekly Production Migration for UniFi

Resolved
ENTSS Migrator will be migrating new codes changed to Production for both SI-net and UniFi applications. Refer to the latest weekly migration minutes for the list approved changes:

https://confluence.its.uq.edu.au/confluence/display/ESS/2023-03-21+Weekly+Production+Changes

In order to seek for approval to migrate changes to Production, it is required that the required member of the team that made the changes to be present at the weekly migration meeting to present their changes. The attendees ( consists of ENTSS staff and EAPS staff ) will review these before making the decision to approve or pre-approval ( conditional to completion of UAT testing ). Refer to the ab...ove hyperlink for the attendees who would reviewed and approved the changes to production.
As such this service view will not have any Peer Reviews nor Approval as that would have been completed as outlined above.

The Team Change Manager willl contain the Migrator name to migrate the changes into Production.

This service view will be marked to complete as soon as the migration completes.


Additional information:
SI-net CAB Pre-Approved Changes (#7317)
https://sharepoint.uq.edu.au/its/ITGovernance/Lists/PreApproved%20Change%20List/DispForm.aspx?ID=45&Source=https%3A%2F%2Fsharepoint%2Euq%2Eedu%2Eau%2Fits%2FITGovernance%2FLists%2FPreApproved%2520Change%2520List%2FAllItems%2Easpx%23InplviewHashc5ffa001%2D2eb0%2D4be2%2D987f%2Dc20e677aea89%3D&ContentTypeId=0x0100D236C02FEDCD4E48B359619FB9CFDF4E


Read more
Resolution: 23 Mar 2023, 7:30 AM
Automatic resolution.
Service affected: UniFi

Start: 23 Mar 2023, 7:00 AM
End: 23 Mar 2023, 7:30 AM
Login to request notification

Large number of failing authentications

Resolved
Active Directory is experiencing a temporary increase in intermittent failed authentications.

Engineers are currently investigating.
Resolution: 21 Mar 2023, 3:45 PM
Three distinct periods of increased failed authentications occurred. Issues can be attributed to mismatch of new krb tickets with stale cached krb tickets of which the timeout period has now elapsed.
Service affected: Active Directory

Start: 21 Mar 2023, 1:00 PM
End: 21 Mar 2023, 3:45 PM
Login to request notification

New samba sessions may have intermittent issues connecting to file shares on nas01

Resolved
Resolution: 21 Mar 2023, 1:59 PM
AD connections re-established
Service affected: nas01 NFS File Storage

Start: 21 Mar 2023, 12:30 PM
End: 21 Mar 2023, 1:59 PM
Login to request notification

On Premise Exchange Mail Services - Exchange Patching

Resolved
Apply Security Updates for Microsoft Exchange Server 2013 (March 2023).
Resolution: 20 Mar 2023, 5:00 PM
Automatic resolution.
Service affected: Exchange Mail and Calendaring

Start: 20 Mar 2023, 4:00 PM
End: 20 Mar 2023, 5:00 PM
Login to request notification

Library PaperCut update to 22.0.9

Resolved
The Library PaperCut service will be updates to 22.0.9 to patch against some known vulnerabilies.
Releasing print jobs from Library printers will be unavailable while the upgrade it performed.
EAIT students should still be able to print, but charging may be delayed.
Resolution: 19 Mar 2023, 9:00 PM
Automatic resolution.
Service affected: Library PaperCut

Start: 19 Mar 2023, 8:00 PM
End: 19 Mar 2023, 9:00 PM
Login to request notification

Banksia network upgrade and outage

Resolved
Installation of building aggregation routers. Ful network outage throughout the building. For more information please contact the Project Manager, Keryn Masters on 0458 585 405.
Resolution: 19 Mar 2023, 5:00 PM
Automatic resolution.
Service affected: Building Networks

Start: 19 Mar 2023, 9:00 AM
End: 19 Mar 2023, 5:00 PM
Login to request notification

Apply Oracle PeopleTools and PUM critical patch updates (CPU) on UniFi (January)

Resolved
To ensure our systems remain secure and compliant, UQ must apply PeopleTools and PUM critical patches on UniFi.

Oracle has released their quarterly security vulnerability patches: https://www.oracle.com/security-alerts/cpujan2023.html.
Resolution: 18 Mar 2023, 5:00 PM
Automatic resolution.
Service affected: UniFi

Start: 18 Mar 2023, 7:00 AM
End: 18 Mar 2023, 5:00 PM
Login to request notification

Disabling virtual machine logging (For Nutanix CVM) - CCR Nutanix cluster.

Resolved
HAZARD ONLY

Emergency change to disable CVM logging as per Nutanix workaround to avoid CVM reloads that can cause loss of services.

CVM reload due to this bug has occurred on 4 occasions so far, one causing a major outage, therefore this is urgent to have applied.

Resolution: 17 Mar 2023, 10:00 PM
Automatic resolution.
Service affected: VMware NextGEN CCR DC

Start: 17 Mar 2023, 8:00 PM
End: 17 Mar 2023, 10:00 PM
Login to request notification

Papercut vulnerability patching to 21.2.11

Resolved
ITS will be patching servers to fix the vulnerabilities specified by PaperCut.
Vulnerability: ZDI-CAN-18987 / PO-1216
Vulnerability: ZDI-CAN-19226 / PO-1219
Resolution: 17 Mar 2023, 6:45 PM
Automatic resolution.
Service affected: Papercut Printing Services

Start: 17 Mar 2023, 6:00 PM
End: 17 Mar 2023, 6:45 PM
Login to request notification

Oracle Client Patching - conman1, cloud-conman

Resolved
January 2023 RU Oracle Client Patching - conman1, cloud-conman

For any individual connections during this window, would need to access via : conman2.compute.dc.uq.edu.au
Resolution: 17 Mar 2023, 4:00 PM
Automatic resolution.
Service affected: Oracle Development DB Hosting

Start: 17 Mar 2023, 3:00 PM
End: 17 Mar 2023, 4:00 PM
Login to request notification

Connectivity Issues for some clients across various sites

Resolved
Some clients are experience network connection issues. This is random across various sites

ITS is investigating potential DHCP issues
Resolution: 16 Mar 2023, 11:18 AM
The problem was traced back to several DHCP servers in CCR not working properly. These have been restarted and working ok now.
Update: 16 Mar 2023, 11:20 AM
ITS are actively investigating this issue.
Service affected: Networks

Start: 16 Mar 2023, 10:47 AM
End: 16 Mar 2023, 11:18 AM
Login to request notification

VMs are not accessible in CCR DC

Resolved
VM Hosts offline due to vCentre being down.

VMs recovered, leaving open as some may still require reloading. Apps is working with IS to complete.
Resolution: 16 Mar 2023, 7:00 PM
Every VM in CCR has been briefly inspected and those that required resetting to resolve file system issues have been done.
Update: 16 Mar 2023, 5:49 PM
While VMWare services have been restored, there are some vm's which have locked file systems and require a reboot. Application owners have been advised to do this, or contact ITS if they are having issues and need assistance. ITS is also checking every VM for confirm it is not locked, and reseting where that is required. Will keep this status active for now.
Resolution: 16 Mar 2023, 1:30 AM
Automatic resolution.
Service affected: VMware NextGEN CCR DC

Start: 15 Mar 2023, 10:20 PM
End: 16 Mar 2023, 7:00 PM
Login to request notification

Test failover CCR DC firewall

Resolved
Failover test CCR FW to analyse root cause of Incident 269
Resolution: 15 Mar 2023, 11:00 PM
Automatic resolution.
Service affected: NGDC Serverfarm Firewall

Start: 15 Mar 2023, 9:00 PM
End: 15 Mar 2023, 11:00 PM
Login to request notification

Weekly Production Migration for SI-net

Resolved
ENTSS Migrator will be migrating new codes changes to Production for SI-net applications

https://confluence.its.uq.edu.au/confluence/display/ESS/2023-03-14+Weekly+Production+Changes

If the change is simple enough then email approval will be sufficient. The email distribution should include Tech Masters, Functional Masters, Change Owner ( Func and Tech) and Manager to assess the impact before granting the Pre/approval.

If the change does not require an outage then seek the line manager for approval.

Refer to https://confluence.its.uq.edu.au/confluence/x/p220Dg

The Team Change Manager willl contain the Migrator name to migrat...e the changes into Production.

This service view will be marked to complete as soon as the migration completes.

Read more
Resolution: 15 Mar 2023, 5:30 PM
Automatic resolution.
Service affected: SI-net

Start: 15 Mar 2023, 5:00 PM
End: 15 Mar 2023, 5:30 PM
Login to request notification

Weekly Production Migration for UniFi

Resolved
ENTSS Migrator will be migrating new codes changed to Production for both SI-net and UniFi applications. Refer to the latest weekly migration minutes for the list approved changes:
https://confluence.its.uq.edu.au/confluence/display/ESS/2023-03-14+Weekly+Production+Changes

In order to seek for approval to migrate changes to Production, it is required that the required member of the team that made the changes to be present at the weekly migration meeting to present their changes. The attendees ( consists of ENTSS staff and EAPS staff ) will review these before making the decision to approve or pre-approval ( conditional to completion of UAT testing ). Refer to the above hyp...erlink for the attendees who would reviewed and approved the changes to production.
As such this service view will not have any Peer Reviews nor Approval as that would have been completed as outlined above.

The Team Change Manager willl contain the Migrator name to migrate the changes into Production.

This service view will be marked to complete as soon as the migration completes.


Additional information:
SI-net CAB Pre-Approved Changes (#7317)
https://sharepoint.uq.edu.au/its/ITGovernance/Lists/PreApproved%20Change%20List/DispForm.aspx?ID=45&Source=https%3A%2F%2Fsharepoint%2Euq%2Eedu%2Eau%2Fits%2FITGovernance%2FLists%2FPreApproved%2520Change%2520List%2FAllItems%2Easpx%23InplviewHashc5ffa001%2D2eb0%2D4be2%2D987f%2Dc20e677aea89%3D&ContentTypeId=0x0100D236C02FEDCD4E48B359619FB9CFDF4E


Read more
Resolution: 15 Mar 2023, 5:30 PM
Automatic resolution.
Service affected: UniFi

Start: 15 Mar 2023, 5:00 PM
End: 15 Mar 2023, 5:30 PM
Login to request notification

On Premise Exchange Mail Services - Windows OS Patching

Resolved
As part of ongoing and proactive OS patching maintenance, some Exchange services will experience brief service interruptions (mail flow may be queued for a minute whilst an active cluster node performs a service failover to the passive node). There may be a slight mail delay (usually less than a minute). However users should not notice or be affected by the patching.
Resolution: 15 Mar 2023, 5:00 PM
Automatic resolution.
Service affected: Exchange Mail and Calendaring

Start: 15 Mar 2023, 4:00 PM
End: 15 Mar 2023, 5:00 PM
Login to request notification

Oracle Client Patching - conman2, cloud-conman2

Resolved
January 2023 RU Oracle Client Patching - conman2, cloud-conman2

For any individual connections during this window, would need to access via : conman1.compute.dc.uq.edu.au
Resolution: 15 Mar 2023, 4:00 PM
Automatic resolution.
Service affected: Oracle Development DB Hosting

Start: 15 Mar 2023, 3:00 PM
End: 15 Mar 2023, 4:00 PM
Login to request notification

RCC/QRISCloud service disruption

Resolved
Multiple services are currently having connectivity issues. This is known to be impacting access to RCC HPC resources and QRIScloud storage (RDM Q collections).

RCC staff are working to resolve the issue.
Resolution: 15 Mar 2023, 4:01 PM
All services restored.
Service affected: RCC/QCIF GPFS Storage

Start: 15 Mar 2023, 2:47 PM
End: 15 Mar 2023, 4:01 PM
Login to request notification

Mailflow Change for ITS Domain

Resolved
Changing outbound mailflow for ITS.uq.edu.au mail domain.

The time and date have been chosen to ensure that the staff required for the change are available.
There is significant mail flow to monitor and test the changes as they are made.
Any issues that may arise can be captured and corrected or the change backed out in a timely fashion.

This initial change has been restricted to the ITS domain to limit any impact.
Resolution: 15 Mar 2023, 3:00 PM
The Scheduled Change went according to plan and within time.
Update: 15 Mar 2023, 1:56 PM
The scheduled change went according to plan and within time.
Service affected: Mail Routing

Start: 15 Mar 2023, 12:00 PM
End: 15 Mar 2023, 3:00 PM
Login to request notification

Upgrade ESXi version from 6.7 u3 to 7.0 u3 for CCR Nextgen Nutanix cluster

Resolved
Upgrade ESXi version from 6.7 u3 to 7.0 u3 for CCR Nextgen Nutanix cluster - vSphere Datacenter - CCR.

Remaining ESXi hosts as part of the Nutanix cluster: NTNX-CCR02.

The original change 12067 was postponed part way through due to a host failure due to a faulty DIMM.
which as now been replaced

Due to the number of hosts and the time required for each, this change occurs over 2 days to complete the remaining hosts. Four will be upgraded on the first day (Tuesday 14th Feb)and the remaining four on the second (Wednesday 15th March). The hosts are listed in the change scope section for each day.
Resolution: 16 Mar 2023, 1:00 AM
Automatic resolution.
Service affected: VMware NextGEN CCR DC

Start: 14 Mar 2023, 5:00 PM
End: 16 Mar 2023, 1:00 AM
Login to request notification

IMB PABX outage due to electrical storm

Resolved
Faults reported for TDM extensions at IMB. Isolated to physical handsets (Anlog & Digital) on IMB PABX only. Retstarted PIMs to restore services. Appears to have been caused by the electrical storm 12/3/23
Resolution: 13 Mar 2023, 11:00 AM
Automatic resolution.
Service affected: PABX Peripheral Support

Start: 13 Mar 2023, 10:00 AM
End: 13 Mar 2023, 11:00 AM
Login to request notification

Wireless Service outage - Wireless router reload

Resolved
Wireless services will be affected for 30 mins due to a reload required for the wireless router Megatron, to implement configuration changes
Resolution: 12 Mar 2023, 6:30 AM
Automatic resolution.
Service affected: Wireless Network

Start: 12 Mar 2023, 6:00 AM
End: 12 Mar 2023, 6:30 AM
Login to request notification

Switchover Some Databases Back to CCR

Resolved
Switchover Databases from magpie-cluster (PRN) to plover-cluster (CCR) .

approx. 5 minute minor disruption per database during switchover from PRN to CCR
Resolution: 12 Mar 2023, 8:00 AM
Automatic resolution.
Service affected: Oracle Production DB Hosting

Start: 12 Mar 2023, 5:30 AM
End: 12 Mar 2023, 8:00 AM
Login to request notification

Workday 2023R1

Resolved
Workday will be unavailable while the first bi-annual release for 2023 is applied.
Resolution: 11 Mar 2023, 8:00 PM
update completed
Resolution: 11 Mar 2023, 8:00 PM
update completed
Resolution: 11 Mar 2023, 8:00 PM
update completed
Service affected: Workday - Human Capital Management System

Start: 11 Mar 2023, 5:00 PM
End: 11 Mar 2023, 8:00 PM
Login to request notification

Exadata Platinum patching - plover-cluster

Resolved
Patch the CCR Exadata with the latest quarterly Oracle Critical Patch Updates (JAN 2023 CPU).

Resolution: 11 Mar 2023, 8:00 PM
Automatic resolution.
Service affected: Oracle Production DB Hosting

Start: 11 Mar 2023, 8:00 AM
End: 11 Mar 2023, 8:00 PM
Login to request notification

Switchover Databases to PRN and apply latest oracle patch

Resolved
Switchover Databases from plover-cluster (CCR) to magpie-cluster (PRN).

approx. 5 minute minor disruption per database during switchover from CCR to PRN.

Resolution: 11 Mar 2023, 8:00 AM
Automatic resolution.
Service affected: Oracle Production DB Hosting

Start: 11 Mar 2023, 5:30 AM
End: 11 Mar 2023, 8:00 AM
Login to request notification

On Premise Exchange Mail Services - Exchange Patching

Resolved
Apply Security Updates for Microsoft Exchange Server 2013 (February 2023).
Resolution: 9 Mar 2023, 5:00 PM
Automatic resolution.
Service affected: Exchange Mail and Calendaring

Start: 9 Mar 2023, 4:00 PM
End: 9 Mar 2023, 5:00 PM
Login to request notification

User directory sync issues

Resolved
User sync issues have been found in the UQ Digital Workspace environment after scheduled maintenance.

A re-sync of all users is occurring to re-sync user permissions for both desktops and applications.

ITS will continue to monitor the service for further interruptions.

A service request with the vendor has been raised to investigate.
Resolution: 10 Mar 2023, 3:57 PM
VMware resolved the issue.
Service affected: UQ Digital Workspace

Start: 9 Mar 2023, 10:00 AM
End: 10 Mar 2023, 3:57 PM
Login to request notification

Flexnet/Portal Down

Resolved
Flexnet Webpage is unavailable. Vendor is investigating.
Resolution: 9 Mar 2023, 9:24 AM
An issue with the SQL Cluster fail over for the FNMSCompliance database has been corrected.
Service affected: FlexNet

Start: 9 Mar 2023, 9:24 AM
End: 9 Mar 2023, 9:24 AM
Login to request notification

Workspace One Access connector upgrade

Resolved
Workspace One Access / Identity Manager upgrade is occurring due to The 19.03.0.1VMware Identity Manager connector reaching the end of extended support on August 31, 2022.
From the 15th March Workspace ONE Access tenants with 19.03.0.1 and earlier Connectors no longer will be able to create, edit, or delete directories.
Workspace ONE Access SaaS customers must upgrade to a supported version of the Workspace ONE Access connector to ensure the continued functionality of all features. Workspace ONE Access customers are strongly encouraged to migrate to the latest Connector (version 22.09.1.0) as soon as possible.
Resolution: 8 Mar 2023, 11:00 PM
Automatic resolution.
Service affected: UQ Digital Workspace

Start: 8 Mar 2023, 8:30 PM
End: 8 Mar 2023, 11:00 PM
Login to request notification

Weekly Production Migration for SI-net

Resolved
ENTSS Migrator will be migrating new codes changes to Production for SI-net applications

https://confluence.its.uq.edu.au/confluence/display/ESS/2023-03-07+Weekly+Production+Changes

If the change is simple enough then email approval will be sufficient. The email distribution should include Tech Masters, Functional Masters, Change Owner ( Func and Tech) and Manager to assess the impact before granting the Pre/approval.

If the change does not require an outage then seek the line manager for approval.

Refer to https://confluence.its.uq.edu.au/confluence/x/p220Dg

The Team Change Manager willl contain the Migrator name to migrat...e the changes into Production.

This service view will be marked to complete as soon as the migration completes.

Read more
Resolution: 8 Mar 2023, 5:30 PM
Automatic resolution.
Service affected: SI-net

Start: 8 Mar 2023, 5:00 PM
End: 8 Mar 2023, 5:30 PM
Login to request notification

Nutanix Host Failure, VM Hosts reloaded

Resolved
The Nutanix host - ntnx-55vt8x2-a.inf.dc.uq.edu.au failed due to a memory fault. VMs reloaded onto other hosts.
Update: 9 Mar 2023, 7:48 AM
VM's affected - uq-admgmt02 is disconnected Information 03/08/2023, 4:58:40 PM uq-admgmt02 vim.event.VmDisconnectedEvent atlas-ghranite is disconnected Information 03/08/2023, 4:58:40 PM atlas-ghranite vim.event.VmDisconnectedEvent uq-m365-mig-2 is disconnected Information 03/08/2023, 4:58:40 PM uq-m365-mig-2 vim.event.VmDisconnectedEvent PF-DIR-HER02 is disconnected Information 03/08/2023, 4:58:40 PM PF-DIR-HER02 vim.event.VmDisconnectedEvent hamachi is disconnected Information 03/08/2023, 4:58:40 PM hamachi vim.event.VmDisconnectedEvent curlew is disconnected Information 03/08/2023, 4:58:40 PM curlew vim.event.VmDisconnectedEvent ca-ems1 is disconnected Information 03/08/2023, 4:58:40 PM ca-ems1 vim.event.VmDisconnectedEvent ldap-prod-b02 is disconnected Information 03/08/2023, 4:58:40 PM ldap-prod-b02 vim.event.VmDisconnectedEvent sa-pum03 is disconnected Information 03/08/2023, 4:58:40 PM sa-pum03 vim.event.VmDisconnectedEvent NTNX-55WR8X2-A-CVM is disconnected Information 03/08/2023, 4:58:40 PM NTNX-55WR8X2-A-CVM vim.event.VmDisconnectedEvent qaafi-hygain01 is disconnected Information 03/08/2023, 4:58:40 PM qaafi-hygain01 vim.event.VmDisconnectedEvent uqp-ps-prod is disconnected Information 03/08/2023, 4:58:40 PM uqp-ps-prod vim.event.VmDisconnectedEvent mis-transfer is disconnected Information 03/08/2023, 4:58:40 PM mis-transfer vim.event.VmDisconnectedEvent uqp-booknet is disconnected Information 03/08/2023, 4:58:40 PM uqp-booknet vim.event.VmDisconnectedEvent uq-license2 is disconnected Information 03/08/2023, 4:58:40 PM uq-license2 vim.event.VmDisconnectedEvent pf-historian21 is disconnected Information 03/08/2023, 4:58:40 PM pf-historian21 vim.event.VmDisconnectedEvent ise-2.3.0.298.SPA.x86_64-template is disconnected Information 03/08/2023, 4:58:40 PM ise-2.3.0.298.SPA.x86_64-template vim.event.VmDisconnectedEvent its-spctlnk-ftp is disconnected Information 03/08/2023, 4:58:40 PM its-spctlnk-ftp vim.event.VmDisconnectedEvent mandarin is disconnected Information 03/08/2023, 4:58:40 PM mandarin vim.event.VmDisconnectedEvent its-ss-jenkins is disconnected Information 03/08/2023, 4:58:40 PM its-ss-jenkins vim.event.VmDisconnectedEvent uq-adcsweb02 is disconnected Information 03/08/2023, 4:58:40 PM uq-adcsweb02 vim.event.VmDisconnectedEvent its-nc-mgmt03 is disconnected Information 03/08/2023, 4:58:40 PM its-nc-mgmt03 vim.event.VmDisconnectedEvent uq-m365-mig-16 is disconnected Information 03/08/2023, 4:58:40 PM uq-m365-mig-16 vim.event.VmDisconnectedEvent copperhead is disconnected Information 03/08/2023, 4:58:40 PM copperhead vim.event.VmDisconnectedEvent aurwhh02 is disconnected Information 03/08/2023, 4:58:40 PM aurwhh02 vim.event.VmDisconnectedEvent pomelo is disconnected Information 03/08/2023, 4:58:40 PM pomelo vim.event.VmDisconnectedEvent burleighheads is disconnected Information 03/08/2023, 4:58:40 PM burleighheads vim.event.VmDisconnectedEvent sa-pum05 is disconnected Information 03/08/2023, 4:58:40 PM sa-pum05 vim.event.VmDisconnectedEvent uq-chkbox01 is disconnected Information 03/08/2023, 4:58:40 PM uq-chkbox01 vim.event.VmDisconnectedEvent its-gas1 is disconnected Information 03/08/2023, 4:58:40 PM its-gas1 vim.event.VmDisconnectedEvent svs-acarcinom is disconnected Information 03/08/2023, 4:58:40 PM svs-acarcinom vim.event.VmDisconnectedEvent pl-8-bkp is disconnected Information 03/08/2023, 4:58:40 PM pl-8-bkp vim.event.VmDisconnectedEvent mintslice is disconnected Information 03/08/2023, 4:58:40 PM mintslice vim.event.VmDisconnectedEvent mis-vs-dev1 is disconnected Information 03/08/2023, 4:58:40 PM mis-vs-dev1 vim.event.VmDisconnectedEvent intelmqprod is disconnected Information 03/08/2023, 4:58:40 PM intelmqprod vim.event.VmDisconnectedEvent gamora is disconnected Information 03/08/2023, 4:58:40 PM gamora vim.event.VmDisconnectedEvent sasd02 is disconnected Information 03/08/2023, 4:58:40 PM sasd02 vim.event.VmDisconnectedEvent bloodhound is disconnected Information 03/08/2023, 4:58:40 PM bloodhound vim.event.VmDisconnectedEvent fswd02 is disconnected Information 03/08/2023, 4:58:40 PM fswd02 vim.event.VmDisconnectedEvent jupyter is disconnected Information 03/08/2023, 4:58:40 PM jupyter vim.event.VmDisconnectedEvent fsas02 is disconnected Information 03/08/2023, 4:58:40 PM fsas02 vim.event.VmDisconnectedEvent uq-adfsp01-t is disconnected Information 03/08/2023, 4:58:40 PM uq-adfsp01-t vim.event.VmDisconnectedEvent garter is disconnected Information 03/08/2023, 4:58:40 PM garter vim.event.VmDisconnectedEvent hsu-test-dp8 is disconnected Information 03/08/2023, 4:58:40 PM hsu-test-dp8 vim.event.VmDisconnectedEvent smi-digdb01 is disconnected Information 03/08/2023, 4:58:40 PM smi-digdb01 vim.event.VmDisconnectedEvent bdds-dev is disconnected Information 03/08/2023, 4:58:40 PM bdds-dev vim.event.VmDisconnectedEvent fn-auto is disconnected Information 03/08/2023, 4:58:40 PM fn-auto vim.event.VmDisconnectedEvent ca-london is disconnected Information 03/08/2023, 4:58:40 PM ca-london vim.event.VmDisconnectedEvent cubone is disconnected Information 03/08/2023, 4:58:40 PM cubone vim.event.VmDisconnectedEvent ushas-win is disconnected Information 03/08/2023, 4:58:40 PM ushas-win vim.event.VmDisconnectedEvent pf-parkguide2 is disconnected Information 03/08/2023, 4:58:40 PM pf-parkguide2 vim.event.VmDisconnectedEvent tea is disconnected Information 03/08/2023, 4:58:40 PM tea vim.event.VmDisconnectedEvent londinium is disconnected Information 03/08/2023, 4:58:40 PM londinium vim.event.VmDisconnectedEvent dragonfruit4 is disconnected Information 03/08/2023, 4:58:40 PM dragonfruit4 vim.event.VmDisconnectedEvent doi-tern1 is disconnected Information 03/08/2023, 4:58:40 PM doi-tern1 vim.event.VmDisconnectedEvent krbs-prod-b01 is disconnected uqp-dsfwebprod is disconnected Information 03/08/2023, 4:58:40 PM uqp-dsfwebprod vim.event.VmDisconnectedEvent Doing updates dapp-ng is disconnected Information 03/08/2023, 4:58:40 PM dapp-ng vim.event.VmDisconnectedEvent Dapp.compute ??? smi-digdb01 is disconnected Information 03/08/2023, 4:58:40 PM smi-digdb01 vim.event.VmDisconnectedEven
Resolution: 8 Mar 2023, 5:30 PM
Automatic resolution.
Service affected: VMware NextGEN CCR DC

Start: 8 Mar 2023, 4:48 PM
End: 8 Mar 2023, 5:30 PM
Login to request notification

Aarnet Service outage to GPN DC - RCC Tape Libaries impacted

Resolved
Fibre leadin for Aarnet to GPN was partially damaged. THis affected RCC FC tape libaries (old and new). All other active service failed over to Prentice side.

Servcie restored now after repatching Aarnet leadin from GPN to Building 80 to a diferent fibre pair.
Resolution: 8 Mar 2023, 11:47 AM
Automatic resolution.
Service affected: Networks

Start: 8 Mar 2023, 10:58 AM
End: 8 Mar 2023, 11:47 AM
Login to request notification

Upgrade ESXi version from 6.7 u3 to 7.0 u3 for CCR Nextgen Nutanix cluster

Resolved
Upgrade ESXi version from 6.7 u3 to 7.0 u3 for CCR Nextgen Nutanix cluster - vSphere Datacenter - CCR.

All ESXi hosts as part of the Nutanix cluster: NTNX-CCR02

Due to the number of hosts and the time required for each, this change occurs over 3 days. Three will be upgraded on the first day (Tuesday 7th Feb), four on the second (Wednesday 8th March) and four on the third day Thursday 9th March. The hosts are listed in the change scope section for each day.
Resolution: 9 Mar 2023, 5:00 PM
Automatic resolution.
Service affected: VMware NextGEN CCR DC

Start: 7 Mar 2023, 7:30 AM
End: 9 Mar 2023, 5:00 PM
Login to request notification

Expand vSAN Cluster - vSAN-Prentice01

Resolved
This CR will add 2 nodes to the current 3 node VMWare vSAN cluster servicing NGDC PRN VMware farm vSAN-Prentice01.
Resolution: 6 Mar 2023, 3:00 PM
Automatic resolution.
Service affected: VMware NextGEN PRN DC

Start: 6 Mar 2023, 2:00 PM
End: 6 Mar 2023, 3:00 PM
Login to request notification

Expand vSAN Cluster - vSAN-CCR01

Resolved
This CR will add 2 nodes to the current 3 node VMWare vSAN cluster servicing NGDC CCR VMware farm - vSAN-CCR.
Resolution: 6 Mar 2023, 2:00 PM
Automatic resolution.
Service affected: VMware NextGEN CCR DC

Start: 6 Mar 2023, 1:00 PM
End: 6 Mar 2023, 2:00 PM
Login to request notification

Apply vSphere Appliance critical security updates - vCenter Server 7.0.3.01300

Resolved
Apply vCenter Server Appliance 7.0 Update 3 v7.0.3.01300 update to the vCenter server (With embedded PSC) for the following sites:

VMware NextGEN PRN DC (Infrastructure)
VMware NextGEN CCR DC (Infrastructure)
VMware Polaris DC (Infrastructure)
VMware GPN DC (Infrastructure)
itsosfpvc.inf.dc.uq.edu.au (Infrastructure)

itsresvc.research.dc.uq.edu.au


Resolution: 6 Mar 2023, 10:00 AM
Automatic resolution.
Service affected: VMware NextGEN CCR DC

Start: 6 Mar 2023, 7:00 AM
End: 6 Mar 2023, 10:00 AM
Login to request notification

Network services impacted due to loss of VLANS

Resolved
Network services have been impacted for AV, EAIT, and commercials groups due to vlan changes affecting switches teh changes were not meant for.

Changes were rolled back
Resolution: 6 Mar 2023, 2:41 PM
Automatic resolution.
Service affected: Networks

Start: 5 Mar 2023, 3:00 PM
End: 6 Mar 2023, 2:41 PM
Login to request notification

Exadata Platinum patching - magpie-cluster

Resolved
Patch the Prentice Exadata with the latest quarterly Oracle Critical Patch Updates (JAN 2023 CPU).

Resolution: 5 Mar 2023, 5:00 PM
Automatic resolution.
Service affected: Oracle Production DB Hosting

Start: 5 Mar 2023, 8:00 AM
End: 5 Mar 2023, 5:00 PM
Login to request notification

Switchover Databases to CCR

Resolved
Switchover Databases from magpie-cluster (PRN) to plover-cluster (CCR).

approx. 5 minute minor disruption per database during switchover from PRN to CCR.
Resolution: 5 Mar 2023, 8:00 AM
Automatic resolution.
Service affected: Oracle Production DB Hosting

Start: 5 Mar 2023, 5:30 AM
End: 5 Mar 2023, 8:00 AM
Login to request notification

SharePoint Cumulative Updates

Resolved
The latest SharePoint Cumulative Updates will be installed in the on-prem SharePoint environment.
Resolution: 4 Mar 2023, 3:00 PM
Automatic resolution.
Service affected: SharePoint

Start: 4 Mar 2023, 7:00 AM
End: 4 Mar 2023, 3:00 PM
Login to request notification

SCCM 2211 Upgrade

Resolved
Scheduled SCCM upgrade from version 2207 to 2211.

SCCM services and Software Centre will be unavailable during this time.
Resolution: 3 Mar 2023, 9:00 PM
Automatic resolution.
Service affected: System Center Configuration Manager

Start: 3 Mar 2023, 6:00 PM
End: 3 Mar 2023, 9:00 PM
Login to request notification

Deployment of Location API

Resolved
In Unitask, Singapore is not being returned in the country search results. We are deploying temporarily fix until UQMaps fixes the issue.

Location API( which interfaces with UQMaps) will be deployed in production. This will have minimal impact to the users.
Resolution: 3 Mar 2023, 5:15 PM
Automatic resolution.
Service affected: Unitask

Start: 3 Mar 2023, 5:00 PM
End: 3 Mar 2023, 5:15 PM
Login to request notification

UQ Longpocket Site Power outage

Resolved
The UQ Longpocket Site is experiencing an unscheduled Power outage. P&F investigating.
Resolution: 3 Mar 2023, 2:24 PM
Site power restored
Service affected: Networks

Start: 3 Mar 2023, 1:37 PM
End: 3 Mar 2023, 2:24 PM
Login to request notification

Apply ESXi patches for Prn Nextgen vSAN cluster

Resolved
Apply ESXi 7.0 u3 patches for Prn Nextgen vSAN cluster - vSphere Datacenter - Prentice.

Resolution: 3 Mar 2023, 4:00 PM
Automatic resolution.
Service affected: VMware NextGEN PRN DC

Start: 3 Mar 2023, 12:00 PM
End: 3 Mar 2023, 4:00 PM
Login to request notification

Apply ESXi patches for CCR Nextgen vSAN cluster

Resolved
Apply ESXi 7.0 u3 patches for CCR Nextgen vSAN cluster - vSphere Datacenter - CCR.
Resolution: 3 Mar 2023, 12:00 PM
Automatic resolution.
Service affected: VMware NextGEN CCR DC

Start: 3 Mar 2023, 8:00 AM
End: 3 Mar 2023, 12:00 PM
Login to request notification

Checkbox Website Migration

Resolved
The Checkbox survey websites will be migrated to a new server.
Resolution: 3 Mar 2023, 7:45 AM
Automatic resolution.
Service affected: Checkbox

Start: 3 Mar 2023, 7:30 AM
End: 3 Mar 2023, 7:45 AM
Login to request notification

On Premise Exchange Mail Services - Windows OS Patching

Resolved
As part of ongoing and proactive OS patching maintenance, some Exchange services will experience brief service interruptions (mail flow may be queued for a minute whilst an active cluster node performs a service failover to the passive node). There may be a slight mail delay (usually less than a minute). However users should not notice or be affected by the patching.
Resolution: 2 Mar 2023, 5:00 PM
Automatic resolution.
Service affected: Exchange Mail and Calendaring

Start: 2 Mar 2023, 4:00 PM
End: 2 Mar 2023, 5:00 PM
Login to request notification

Increase default SharePoint Online site / Teams quota to 100GB

Resolved
The M365 SharePoint Online site storage quota will be increased from 50GB to 100GB.

This updated configuration will provide any new SharePoint Online site, any new Microsoft Team, any new Microsoft Team Private or Shared channel, and any new Yammer community with 100GB of storage quota.

Existing SharePoint Online sites, Microsoft Teams and Yammer communities will have their existing storage quotas increased at a later date.
Resolution: 2 Mar 2023, 9:01 AM
Automatic resolution.
Service affected: Office 365 - SharePoint

Start: 2 Mar 2023, 9:00 AM
End: 2 Mar 2023, 9:01 AM
Login to request notification

Data Quality for bulk mailing segments

Resolved
There are approximately 15,000 additional contacts coming through in the Program Enrolled segment for students. Additional students may receive broadcast emails if using segments to create audiences for bulk mailings using Service CRM.
This is being investigated by ITS
Resolution: 6 Mar 2023, 10:06 AM
Student data backed-up. Affected student tables purged. Data feed disabled. Relevant student data imported over weekend and numbers checked against expected segments from SI-net.
Resolution: 6 Mar 2023, 10:06 AM
All relevant student data imported over weekend and numbers checked against expected segments from SI-net.
Service affected: Oracle Service Cloud a.k.a Service CRM

Start: 1 Mar 2023, 9:43 AM
End: 6 Mar 2023, 10:06 AM
Login to request notification

Planned disruption for OHS course content on Learn.UQ (Blackboard)

Resolved
Access to the Blackboard/Learn.UQ suite of courses which sit under the heading:NOT FOR STAFF OR HDR - UQ Workplace Inductions and Training OHSB0000N_0000_00000 will be offline during Tuesday night while ITS is updating the OHS course content. Access to other Learn.UQ (Blackboard) courses will be normal during this time.

Examples of what will be affected are: Annual Fire Safety Awareness (OHSB09), Undergraduate Student Laboratory Safety Induction (UGRD01) and all other OHS training modules.
Resolution: 28 Feb 2023, 11:00 PM
Automatic resolution.
Service affected: Blackboard Learn

Start: 28 Feb 2023, 9:00 PM
End: 28 Feb 2023, 11:00 PM
Login to request notification

Emergency upgrade for Wireless router - Megatron

Resolved
Wireless services will be unavailable for 1 hr during the change window while the wireless routers are being upgraded. This is an emergency change to address capacity issues affecting services with existing routers due to high wireless client numbers being experienced this semester.

Resolution: 28 Feb 2023, 11:59 PM
Automatic resolution.
Service affected: Wireless Network

Start: 28 Feb 2023, 9:00 PM
End: 28 Feb 2023, 11:59 PM
Login to request notification

Connectivity Issues if not already connected

Resolved
Wireless clients may experience internet issues after connecting to WiFi. This is a suspected capacity issue seen in wireless routers due to number of clients now connecting. Existing,working clients are not impacted.

ITS is configuring higher capapcity router to be installed this morning to take the high loads.
Resolution: 28 Feb 2023, 11:33 AM
Automatic resolution.
Service affected: Wireless Network

Start: 28 Feb 2023, 10:15 AM
End: 28 Feb 2023, 11:33 AM
Login to request notification

Emergency reload of wireless routers

Resolved
As per Cisco TAC recommendations, the wireless controllers - Megatron - shall be reloaded to clear suspect memory leak affecting services during peak periods. Wireless service will not be available across all sites for up to 20 minutes between 9:15PM and 11:15PM during the change period.
Resolution: 27 Feb 2023, 11:15 PM
Automatic resolution.
Service affected: Wireless Network

Start: 27 Feb 2023, 9:00 PM
End: 27 Feb 2023, 11:15 PM
Login to request notification

No Internet access for some wireless clients

Resolved
Wireless services are currently being impacted with new devices connecting unable to access the internet.

This is currently under investigation by ITS
Resolution: 27 Feb 2023, 7:46 PM
All wireless clients now have internet access. Wireless router memory is now at 12k free space
Service affected: Wireless Network

Start: 27 Feb 2023, 12:00 PM
End: 27 Feb 2023, 7:46 PM
Login to request notification

MyAurion Having Issues

Resolved
The MyAurion service is having issues and will be restarted.
Resolution: 27 Feb 2023, 9:15 AM
Automatic resolution.
Service affected: MyAurion

Start: 27 Feb 2023, 8:45 AM
End: 27 Feb 2023, 9:15 AM
Login to request notification

Aggregation Upgrade project - Herston router line card installation

Resolved
New line cards will be installed in Herston routers. No impact planned. For further information, please call the Project Manager, Keryn Masters 0458 585 405.
Resolution: 26 Feb 2023, 3:00 PM
Automatic resolution.
Service affected: Building Networks

Start: 26 Feb 2023, 10:30 AM
End: 26 Feb 2023, 3:00 PM
Login to request notification

MFA rollout - SI-net

Resolved
This change is to enable Multifactor Authentication for all mySI-net users.
Resolution: 26 Feb 2023, 6:00 PM
Automatic resolution.
Service affected: SI-net

Start: 25 Feb 2023, 6:00 AM
End: 26 Feb 2023, 6:00 PM
Login to request notification

Network outage for (7) Parnell

Resolved
ITS are investigating the issue
Resolution: 22 Feb 2023, 8:45 AM
Power to the communication room was not restored until 8:30am from a failed RCD test that uncovered a fault in the power system.
Service affected: Building Networks

Start: 22 Feb 2023, 7:00 AM
End: 22 Feb 2023, 8:45 AM
Login to request notification

SDA Project - Prentice level 3 pilot

Resolved
Full network outage on level 3 of Prentice while switches are reloaded for SDA pilot. Please avoid the area. For further information, please contact the project manager, Keryn Masters on 0458 585 405.
Resolution: 12 Feb 2023, 5:00 PM
Automatic resolution.
Service affected: Building Networks

Start: 11 Feb 2023, 9:00 AM
End: 12 Feb 2023, 5:00 PM
Login to request notification

United Care Hospital Eduroam Wireless Scheduled Outage

Resolved
Uniting Care QLD will be performing a generator test that will impact the wireless network
Resolution: 10 Feb 2023, 9:45 PM
Automatic resolution.
Service affected: Wireless Network

Start: 10 Feb 2023, 9:00 PM
End: 10 Feb 2023, 9:45 PM
Login to request notification

Weekly Production Migration for SI-net

Resolved
ENTSS Migrator will be migrating new codes changes to Production for SI-net applications


https://confluence.its.uq.edu.au/confluence/display/ESS/2023-02-07+Weekly+Production+Changes

If the change is simple enough then email approval will be sufficient. The email distribution should include Tech Masters, Functional Masters, Change Owner ( Func and Tech) and Manager to assess the impact before granting the Pre/approval.

If the change does not require an outage then seek the line manager for approval.

Refer to https://confluence.its.uq.edu.au/confluence/x/p220Dg

The Team Change Manager willl contain the Migrator name to... migrate the changes into Production.

This service view will be marked to complete as soon as the migration completes.

Read more
Resolution: 8 Feb 2023, 5:30 PM
Automatic resolution.
Service affected: SI-net

Start: 8 Feb 2023, 5:00 PM
End: 8 Feb 2023, 5:30 PM
Login to request notification

Weekly Production Migration for UniFi

Resolved
ENTSS Migrator will be migrating new codes changed to Production for both SI-net and UniFi applications. Refer to the latest weekly migration minutes for the list approved changes:
https://confluence.its.uq.edu.au/confluence/display/ESS/2023-02-07+Weekly+Production+Changes

In order to seek for approval to migrate changes to Production, it is required that the required member of the team that made the changes to be present at the weekly migration meeting to present their changes. The attendees ( consists of ENTSS staff and EAPS staff ) will review these before making the decision to approve or pre-approval ( conditional to completion of UAT testing ). Refer to the above hyp...erlink for the attendees who would reviewed and approved the changes to production.
As such this service view will not have any Peer Reviews nor Approval as that would have been completed as outlined above.

The Team Change Manager willl contain the Migrator name to migrate the changes into Production.

This service view will be marked to complete as soon as the migration completes.


Additional information:
SI-net CAB Pre-Approved Changes (#7317)
https://sharepoint.uq.edu.au/its/ITGovernance/Lists/PreApproved%20Change%20List/DispForm.aspx?ID=45&Source=https%3A%2F%2Fsharepoint%2Euq%2Eedu%2Eau%2Fits%2FITGovernance%2FLists%2FPreApproved%2520Change%2520List%2FAllItems%2Easpx%23InplviewHashc5ffa001%2D2eb0%2D4be2%2D987f%2Dc20e677aea89%3D&ContentTypeId=0x0100D236C02FEDCD4E48B359619FB9CFDF4E


Read more
Resolution: 8 Feb 2023, 5:30 PM
Automatic resolution.
Service affected: UniFi

Start: 8 Feb 2023, 5:00 PM
End: 8 Feb 2023, 5:30 PM
Login to request notification

Flexnet Offline

Resolved
Issue with the Flexnet VM host has caused a service outage
Resolution: 6 Feb 2023, 9:39 AM
Host restored
Service affected: FlexNet

Start: 6 Feb 2023, 9:15 AM
End: 6 Feb 2023, 9:39 AM
Login to request notification

its-isilon02 (nas02) - OS Upgrade + Patches + drive and node firmware

Resolved
Upgrade of cluster OS (OneFS) from 9.2.1.11 to 9.4.0.x (latest roll up patch currently 9.4.0.11)
Resolution: 5 Feb 2023, 5:00 PM
Change completed.
Service affected: nas02 Group & Home File Shares

Start: 5 Feb 2023, 7:00 AM
End: 5 Feb 2023, 5:00 PM
Login to request notification

HPCU Internet Traffic - potential service impact

Resolved
HPCU Internet traffic may be seeing some impact due to a faulty interface between GW20 and GW10.

ITS is onsite and investigating the fault now.
Resolution: 5 Feb 2023, 8:35 AM
HPCU Traffic affected as been failed across to another link while the faulty interface between GW10 and GW20 is fixed.
Service affected: Internet Services UQ

Start: 5 Feb 2023, 6:40 AM
End: 5 Feb 2023, 8:35 AM
Login to request notification

St Lucia Campus - Network outage to selected areas in some buildings

Resolved
UPDATE - All restored apart from -

B47, Level - Power off to various rooms, archibus logged
B12 Duhig North Level 4


These building will be impacted with loss of network services in part or all due to failed network access switches -
39, 60, 23, 01, 47, 45, 12, 76, 79, 64, 74, 23, 09, 50

Works are underway to fix/replace the failed switches.

Resolution: 14 Feb 2023, 12:18 PM
Resolved
Update: 2 Feb 2023, 11:50 AM
Remaining buildings/comms room affected - B47, Level - Power off to various rooms, archibus logged B12 Duhig North Level 4
Service affected: Networks

Start: 2 Feb 2023, 6:00 AM
End: 14 Feb 2023, 12:18 PM
Login to request notification

Weekly Production Migration for SI-net

Resolved
ENTSS Migrator will be migrating new codes changes to Production for SI-net applications

Details of the issue and the resolution is recorded in phire CS001838, JIRA, and confluence.
https://confluence.its.uq.edu.au/confluence/display/ESS/2023-01-31+Weekly+Production+Changes

If the change is simple enough then email approval will be sufficient. The email distribution should include Tech Masters, Functional Masters, Change Owner ( Func and Tech) and Manager to assess the impact before granting the Pre/approval.

If the change does not require an outage then seek the line manager for approval.

Refer to https://confluence.its.uq.edu.au/...confluence/x/p220Dg

The Team Change Manager willl contain the Migrator name to migrate the changes into Production.

This service view will be marked to complete as soon as the migration completes.

Read more
Resolution: 2 Feb 2023, 5:30 AM
Automatic resolution.
Service affected: SI-net

Start: 2 Feb 2023, 5:00 AM
End: 2 Feb 2023, 5:30 AM
Login to request notification

Weekly Production Migration for SI-net

Resolved
ENTSS Migrator will be migrating new codes changes to Production for SI-net applications

Details of the issue and the resolution is recorded in phire CS001838, JIRA, and confluence.
https://confluence.its.uq.edu.au/confluence/display/ESS/2023-01-31+Weekly+Production+Changes
https://confluence.its.uq.edu.au/confluence/pages/viewpage.action?pageId=246707312

If the change is simple enough then email approval will be sufficient. The email distribution should include Tech Masters, Functional Masters, Change Owner ( Func and Tech) and Manager to assess the impact before granting the Pre/approval.

If the change does not require an outage then se...ek the line manager for approval.

Refer to https://confluence.its.uq.edu.au/confluence/x/p220Dg

The Team Change Manager willl contain the Migrator name to migrate the changes into Production.

This service view will be marked to complete as soon as the migration completes.

Read more
Resolution: 1 Feb 2023, 6:00 PM
Automatic resolution.
Service affected: SI-net

Start: 1 Feb 2023, 5:00 PM
End: 1 Feb 2023, 6:00 PM
Login to request notification

Firewalling HPCU Traffic

Resolved

The University of Queensland (UQ) hosts a diverse cohort with complex information technology (IT) infrastructure. With the Higher Education sector being a prime target for cyber-criminal activity it is essential that the cybersecurity mechanisms in place at UQ are efficient and relevant to addressing these cyber threats. It is important for the cyber security and data integrity of UQ that the university appropriately manages the cyber risks associated with its networks and assets. This proposal targets improvements across several core infrastructure capabilities to further support the Universitys IT Security goals.

For this CR we plan to move HPCU networks behind a firewa...ll

Read more
Resolution: 29 Jan 2023, 3:00 PM
Change completed.
Service affected: RCC High Speed Data Centre Network Spine

Start: 29 Jan 2023, 9:00 AM
End: 29 Jan 2023, 3:00 PM
Login to request notification

its-isilon01 (nas01) - OS Upgrade + Patches + drive and node firmware

Resolved
Upgrade of cluster OS (OneFS) from 9.2.1.11 to 9.4.0.x (latest roll up patch currently 9.4.0.11)
Resolution: 29 Jan 2023, 5:00 PM
Completed Successfully
Service affected: nas01 NFS File Storage

Start: 29 Jan 2023, 7:00 AM
End: 29 Jan 2023, 5:00 PM
Login to request notification

UQ Authenticate failing for M365 logins

Resolved
UQ Authenticate is currently unable to authenticate users in to M365 (including Office Mail).

Existing M365 sessions are unaffected.

ITS staff are working to resolve the problem.
Resolution: 27 Jan 2023, 7:25 AM
This issue has been resolved.
Resolution: 27 Jan 2023, 7:00 AM
Service affected: UQ Authenticate

Start: 27 Jan 2023, 4:00 AM
End: 27 Jan 2023, 7:25 AM
Login to request notification

Users may intermittently be unable to access multiple Microsoft 365 services

Resolved
Microsoft have identified a potential networking issue and are reviewing service telemetry to determine the next troubleshooting steps.

Microsoft have received reports that the following services are impacted:

-Microsoft Teams
-Exchange Online
-Outlook
-SharePoint Online
-OneDrive

Next update by: Wednesday, January 25, 2023, 7:00 PM
Resolution: 25 Jan 2023, 9:00 PM
Microsoft has continued to take additional steps to expedite the recovery process, and their telemetry indicates that the majority of the affected M365 services have now returned to normal.
Update: 25 Jan 2023, 8:26 PM
Microsoft are continuing to monitor the recovery across the service and customers are also reporting mitigation. Microsoft are also connecting the service to additional infrastructure to expedite the recovery process.
Update: 25 Jan 2023, 6:49 PM
Impact is occurring to the following services but is not limited to them: -Microsoft Teams -Exchange Online -Outlook -SharePoint Online -OneDrive for Business -Microsoft Graph -PowerBi Current status: Microsoft have isolated the problem to a networking configuration issue, and are analyzing the best mitigation strategy to address it without causing additional impact. Microsoft will provide more information once they have additional information. Scope of impact: Any user serviced by the affected infrastructure may be unable to access multiple Microsoft 365 services.
Service affected: Office 365 - Tenant Administration

Start: 25 Jan 2023, 5:26 PM
End: 25 Jan 2023, 9:00 PM
Login to request notification

Weekly system maintenance

Resolved
System maintenance
Resolution: 24 Jan 2023, 4:55 PM
Automatic resolution.
Service affected: UniFi

Start: 24 Jan 2023, 4:50 PM
End: 24 Jan 2023, 4:55 PM
Login to request notification

Weekly system maintenance

Resolved
System maintenance
Resolution: 24 Jan 2023, 4:45 PM
Automatic resolution.
Service affected: SI-net

Start: 24 Jan 2023, 4:40 PM
End: 24 Jan 2023, 4:45 PM
Login to request notification

Update IMS configuration to use AES256 for Active Directory Kerberos encryption

Resolved
Update the IMS trust object and the keytab used by IMS hosts, so that the IMS is forced to use AES256 for its Kerberos ticket encryption.

Additionally, the IMS account password will be reset.
Resolution: 24 Jan 2023, 9:00 AM
Automatic resolution.
Service affected: Active Directory

Start: 24 Jan 2023, 8:00 AM
End: 24 Jan 2023, 9:00 AM
Login to request notification

Domain-wide policy (GPO) to define support of secure kerberos encryption types

Resolved
This change will deploy a domain-wide policy which will set Windows Server and Desktop Operating Systems to use secure kerberos encryption types when requesting kerberos tickets from the domain controllers.

Secure encryption types that will be allowed by the policy includes AES 256.

Insecure encryption types will not be allowed by the policy applied include RC4.

This GPO will affect all Windows domain-joined computers (including Servers and Workstations) that support secure encryption types. Secure encryption types are not supported in Windows 2000 Server, Windows XP, or Windows Server 2003.

When the windows clients process the policy, they... will update their own Active Directory msDS-SupportedEncryptionTypes attribute to indicate that they only support stronger kerberos encryption.

Read more
Resolution: 24 Jan 2023, 4:00 PM
Automatic resolution.
Service affected: Active Directory

Start: 24 Jan 2023, 8:00 AM
End: 24 Jan 2023, 4:00 PM
Login to request notification

Windows devices experiencing connection problems to UQ SSID

Resolved
Some devices, potentailly only windows based, are unable to connect to the UQ SSID.

As a workaround EDUROAM does work for those devices and can be used.

This is under investigation by ITS.

Apple/Android based do not appear to be affected.
Resolution: 23 Jan 2023, 10:00 AM
UQ Cert had expired earlier than noted in our cert calendar. Has been renewed and working again
Service affected: Wireless Network

Start: 23 Jan 2023, 8:00 AM
End: 23 Jan 2023, 10:00 AM
Login to request notification

Aggregation Upgrade project - Gatton router line card installation

Resolved
Additional capacity is being added to the Gatton routers to support upcoming aggregation upgrades. There will be no outage. If any questions, please contact the Project Manager, Keryn Masters 0458 585 405.
Resolution: 22 Jan 2023, 3:30 PM
Automatic resolution.
Service affected: Building Networks

Start: 22 Jan 2023, 10:30 AM
End: 22 Jan 2023, 3:30 PM
Login to request notification

Replace F5 load balancers: CCR

Resolved
ITS engineers will be replacing the F5 Load balancers in CCR DC
Resolution: 22 Jan 2023, 5:00 PM
Automatic resolution.
Service affected: NGDC Serverfarm Load Balancer

Start: 22 Jan 2023, 7:00 AM
End: 22 Jan 2023, 5:00 PM
Login to request notification

Replace F5 load balancers: PRN

Resolved
ITS engineers will be replacing the F5 Load balancers in PRN DC
Resolution: 21 Jan 2023, 6:00 PM
Automatic resolution.
Service affected: NGDC Serverfarm Load Balancer

Start: 21 Jan 2023, 8:00 AM
End: 21 Jan 2023, 6:00 PM
Login to request notification

Aggregation Upgrade Project - Network outage Hibiscus and Kurrajong buildings

Resolved
ITS are building a new communications cabinet in the Hibiscus building. The communications rack is being replaced. The network outage will impact the connected Kurrajong building also. Please avoid the area as alarms will not be active. For further information, please call the Project Manager, Keryn Masters on 0458 585 405.
Resolution: 21 Jan 2023, 4:00 PM
Automatic resolution.
Service affected: Building Networks

Start: 21 Jan 2023, 6:00 AM
End: 21 Jan 2023, 4:00 PM
Login to request notification

No access to Workday while maintenance is performed

Resolved
No access to Workday while maintenance is performed
Resolution: 21 Jan 2023, 2:00 AM
Automatic resolution.
Service affected: Workday - Human Capital Management System

Start: 20 Jan 2023, 5:00 PM
End: 21 Jan 2023, 2:00 AM
Login to request notification

Switchover Aurion Database to PRN

Resolved
Switchover Aurion Database to PRN
Resolution: 19 Jan 2023, 7:00 PM
Due to scheduled patching changes, this change will be cancelled
Update: 13 Jan 2023, 3:27 PM
Due to scheduled patching changes, this change will be cancelled
Service affected: Oracle Production DB Hosting

Start: 19 Jan 2023, 6:30 PM
End: 19 Jan 2023, 7:00 PM
Login to request notification

Gatton Campus network link in non-redundant state

Resolved
Hazardous awareness only - Due to an Aarnet fibre cut, the Gatton campus is now actively on the secondary link.

Redundancy is expected to to back in place around midnight.
Resolution: 20 Jan 2023, 6:00 AM
Aarnet has repaired the fibre
Service affected: Networks

Start: 19 Jan 2023, 6:06 PM
End: 20 Jan 2023, 6:00 AM
Login to request notification

On Premise Exchange Mail Services - Exchange Patching

Resolved
Apply Security Updates for Microsoft Exchange Server 2013 (January 2023).
Resolution: 19 Jan 2023, 5:00 PM
Automatic resolution.
Service affected: Exchange Mail and Calendaring

Start: 19 Jan 2023, 4:00 PM
End: 19 Jan 2023, 5:00 PM
Login to request notification

Network Services to Polaris are in Non-Redundant state

Resolved
Hazard Awareness Only - Due to damage to Aarnet fibre, Network services to Polaris are now running in a non-redundant state.

No users based services will be affected.
Resolution: 20 Jan 2023, 6:00 AM
Aarnet has repaired the fibre
Service affected: Polaris Ethernet Fabric

Start: 19 Jan 2023, 1:47 PM
End: 20 Jan 2023, 6:00 AM
Login to request notification