Operations Portal

This service is provided by CCIN2P3, and funded by the European Union’s Horizon 2020 research and innovation programme

Latest news

• se.igfae.usc.es (DOME access endpoint at USC-LCG2) will be decommissioned

November 24, 2022 15:20 | Marcos Seco Miguelez
More details

se.igfae.usc.es (DOME access endpoint at USC-LCG2) will be decommissioned

Our DOME access endpoint, se.igfae.usc.es, will be decommissioned in the following weeks.

Given the amount of space offered by our SE it is unlikely that LHCb VO will be affected by this operation. Should this not be the case, please let us know before January 15th 2023 so we can extend the foreseen downtime interval. A GGUS ticket, https://ggus.eu/index.php?mode=ticket_info&ticket_id=159611, to follow the decommission procedure has been created and has more information about the timeline.

No new SE will be available at our site since the current one have seldom been used.

November 24, 2022 15:20
by Marcos Seco Miguelez

• DESY VOMS server certificate migration completed

November 21, 2022 14:48 | Thomas Hartmann
More details

DESY VOMS server certificate migration completed

Dear Users of VOs using the DESY VOMS server,

as previously announced, we have completed the update of the host certificate of the VOMS server at
grid.desy.de
where your VO is hosted. As the host certificate details have changed, an update to the lsc and vomeses files is necessary to obtain new proxies without errors. If you encounter errors while obtaining a new grid proxy, please contact your resource admin to update the locally installed voms details.

Updated server credentials can be found as rpm or tarball under
https://gitlab.desy.de/grid/voms-cert-transition
The rpm package `desy-voms-all` with the new host certificate details can also be found in the WLCG repositories.

Also updated credential files have been propagated through the /cvmfs/grid.cern.ch repository for users of CVMFS based environments.

For OSG users, an updated vo roles package is available in the osg-testing repository, which is expected to move to the osg repository during the day.

If you have any questions, please do not hesitate to contact us.

Cheers,
Thomas for the DESY Grid team

November 21, 2022 14:48
by Thomas Hartmann

• DESY VOMS server downtime/update on Monday, Nov.21, requiring client updates

November 15, 2022 11:14 | Thomas Hartmann
More details

DESY VOMS server downtime/update on Monday, Nov.21, requiring client updates

Dear Users of the VOMS instance hosted at DESY,

coming week on Monday, November 21, the host certificate of the VOMS server
grid-voms.desy.de
will be updated for changing to a new CA.

With the instance's new host certificate, clients will need to update their vomses and lsc files to trust the changed DN.

Without updated VOMS descriptions, clients will fail to obtain new proxies.

If necessary, please update your clients during the downtime accordingly.

Updated rpm, lsc and vomses files as well as a more detailed description are available under
https://gitlab.desy.de/grid/voms-cert-transition

For any questions or suggestions, please contact us under
grid@desy.de

See also the downtime announcement under
https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=33173

Regards,
Thomas for the DESY Grid team

November 15, 2022 11:14
by Thomas Hartmann

• VOMS server downtime/update on Monday, Nov.21, requiring client updates

November 14, 2022 17:28 | Thomas Hartmann
More details

VOMS server downtime/update on Monday, Nov.21, requiring client updates

Dear Users of the VOMS instances hosted at DESY,

coming week on Monday, November 21, the host certificate for the VOMS server
grid-voms.desy.de
will be updated for changing to a new CA.

With the instance's new host certificate, clients will need to update their vomses and lsc files to trust the changed DN.

Without updated VOMS descriptions, clients will fail to obtain new proxies.

If necessary, please update your clients during the downtime accordingly.

Updated rpm, lsc and vomses files as well as a more detailed description are available under
https://gitlab.desy.de/grid/voms-cert-transition

For any questions or suggestions, please contact us under
grid@desy.de

See also the downtime announcement under
https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=33173

Regards,
Thomas for the DESY Grid team

November 14, 2022 17:28
by Thomas Hartmann

• GOCDB is read-only over the weekend

October 28, 2022 13:22 | Greg Corbett
More details

GOCDB is read-only over the weekend

Dear all,

The RAL-wide network issues that impacted GOCDB last week have been reoccurring this week, with minimal impact on GOCDB. However, this morning a full reboot of the underlying network was undertaken to resolve issues with other services.

The success of this intervention and general health of the network is still being determined - as such, the GOCDB team has taken the precaution of switching the DNS entry for goc.egi.eu to point at our failover infrastructure to ensure read-only access is maintained over the weekend.

If we receive the "all clear" before close of business today, we will disengage the failover and send a follow up broadcast.

We apologies for any inconvenience this may cause,
Kind Regards,
The GOCDB team

October 28, 2022 13:22
by Greg Corbett

More news

• Decommissioning of DPM / SRM at UKI-SCOTGRID-GLASGOW

October 19, 2022 11:27 | Gordon Stewart
More details

Decommissioning of DPM / SRM at UKI-SCOTGRID-GLASGOW

Dear all,

We would like to announce that UKI-SCOTGRID-GLASGOW has begun the decommissioning procedure for its DPM and SRM services. A tracking ticket can be found here:

https://ggus.eu/index.php?mode=ticket_info&ticket_id=159276

Our plan is to switch off this service from Wednesday 16th November. We have discussed this with LHCb directly, as the major remaining user of this service, but we recommend that other VOs check that all critical data has been transferred elsewhere.

Regards,

Gordon Stewart
ScotGrid Technical Co-ordinator

October 19, 2022 11:27
by Gordon Stewart

• [Update] RAL-LCG2 including GOCDB in unscheduled downtime

October 18, 2022 18:05 | Alastair Dewhurst
More details

[Update] RAL-LCG2 including GOCDB in unscheduled downtime

Dear all,

As of 16:00 UTC, the network failure that caused the loss of the databases services at RAL has been resolved, although we are currently running in a non-resilient manner. We will be running at risk over night.

The GOCDB is back in full production (i.e. both writes and reads).

The RAL Tier-1 Disk storage (Echo) and Tape storage (Antares) are back in production. The batch farm will be brought back online tomorrow morning. Further downtimes will be noted in the GOCDB (rather than more announcements).

Thank you for your patience

Alastair

RAL Tier-1 Manager

October 18, 2022 18:05
by Alastair Dewhurst

• [Update] RAL-LCG2 including GOCDB in unscheduled downtime

October 18, 2022 13:06 | Alastair Dewhurst
More details

[Update] RAL-LCG2 including GOCDB in unscheduled downtime

Dear all,

The site downtime at RAL caused by the core network failure is still ongoing. We are working on recovering the databases currently.

As mentioned in the previous update, the GOCDB is working (at risk) in read-only mode: https://goc.egi.eu/

Echo (Disk storage) is functioning but in a degraded state.

Antares (Tape storage) is currently down however we can bring the service back quickly once the network has stabilized.

We are expecting the batch farm to remain down until tomorrow morning.

We will provide an update by 14:00 UTC and hope to be in a position to announce the return of some production services before the end of the day.

Thanks,

Alastair

RAL Tier-1 Manager

October 18, 2022 13:06
by Alastair Dewhurst

• Ongoing GOCDB unavailability

October 18, 2022 11:02 | Greg Corbett
More details

Ongoing GOCDB unavailability

Dear all,

The network issues at RAL are still ongoing. Overnight, partial network connectivity was restored but this is not sufficient to restore the primary (read/write) instance of GOCDB.

https://goc.egi.eu/ is currently being served by the read-only failover, however this remains at risk while network recovery work is ongoing to restore services at RAL.

The failover remains directly accessible: https://gocdb.hartree.stfc.ac.uk.

A further update will be given by 12:00 UTC.

Thanks,
The GOCDB Team

October 18, 2022 11:02
by Greg Corbett

• RAL-LCG2 including GOCDB in unscheduled downtime

October 17, 2022 20:07 | Alastair Dewhurst
More details

RAL-LCG2 including GOCDB in unscheduled downtime

Dear All

At shortly after 15:00 local time (14:00 UTC) today (17/10/22) there was a major failure of the RAL core network. All services are currently down including the GOCDB which would normally be used to declare outages.

After 6 hours of work we have only been able to partially restore network connectivity and we are thus declaring a whole site outage until at least midday (11:00 UTC) tomorrow (18/10/22).

Alastair

RAL Tier-1 Manager

October 17, 2022 20:07
by Alastair Dewhurst

• Ongoing GOCDB unavailability

October 17, 2022 18:07 | Adrian Coveney
More details

Ongoing GOCDB unavailability

Due to network issues at RAL, GOCDB remains unavailable. Staff are on site investigating.

We will provide a further update once more information is available, but that may not be until tomorrow. Connectivity to GOCDB may be restored in the meantime.

October 17, 2022 18:07
by Adrian Coveney

• GOCDB 5.10.0 Update

October 12, 2022 12:54 | Greg Corbett
More details

GOCDB 5.10.0 Update

Dear all,

The GOCDB team will soon be releasing version 5.10.0 into production, which brings necessary changes relating to improving user privacy. This new version is already running on our preproduction system [1] (data updated 3rd Oct 2022) and there is a GGUS ticket [2] to track any issues with it.

Once version 5.10.0 is deployed into production, which is scheduled for week commencing 31st Oct 2022, users who have not visited the portal [3] in 18 months will have their GOCDB accounts automatically deleted. Going forward, warning emails will be sent monthly to users with 12 months or more of inactivity.

If you are concerned about your account being deleted, we recommend you visit the GOCDB portal [3] whilst authenticated to ensure your account is considered active.

Thanks,
The GOCDB team.

[1] https://gocdb-preprod.egi.eu/
[2] https://ggus.eu/index.php?mode=ticket_info&ticket_id=159180
[3] https://goc.egi.eu/portal/

October 12, 2022 12:54
by Greg Corbett

• EGI monthly broadcast - October 2022

October 3, 2022 12:36 | Alessandro Paolini
More details

EGI monthly broadcast - October 2022

=============== Content: ==================

1) Release of UMD 4.17.0 and 4.17.1

2) New configuration for monitoring webdav endpoints

3) New configuration for monitoring XrootD (and EOS) endpoints

============================================

1) Release of UMD 4.17.0 and 4.17.1

UMD 4.17.0 released: https://repository.egi.eu/UMD/4.17.0.html
- ArC 6.1.5.1
- Myproxy 6.2.14
- gridFTP 13.26.1
- xroot 5.4.2
- condor 9.0.16,
- htcondor-CE 5.1.3
- dCache 6.2.44
- gfal2 clients 2.20.5
- davix 0.8.2,
- dmlite 1.15.2
- cvmfs 2.9.4
- egi-cvmfs 4.4.0
- new nagios probes for ondedata and IM

UMD 4.17.1 released: https://repository.egi.eu/UMD/4.17.1.html
- dCache 7.2.15: new major release
- Argus 1.7.5: bug fix release that prevents argus pepd crash if non standard characters are found in the DN Certification Authorities released by the EGI Trust Anchor team.
- glite-infoprovider-ldap 1.5.0: bug fix release that suppress the software and job information to be added to the topbdii01 ldap preventing huge memory consuption.

2) New configuration for monitoring webdav endpoints

It was agreed a change on the GOCDB fields to use for monitoring the webdav endpoints.
For some time the webdav metric, which is in the critical profile, will take into account either the old or the new settings in order to guarantee a smooth transition to the new settings.

Please follow the indications as reported in https://docs.egi.eu/internal/configuration-database/adding-service-endpoint/#webdav

The endpoint url information used for monitoring purposes should be set in the extension properties section. Create the following:
- Name: ARGO_WEBDAV_OPS_URL
- Value: webdav URL containing also the VO ops folder

3) New configuration for monitoring XrootD (and EOS) endpoints

We would like to remind you to enable the ops VO on your EOS and Xrootd endpoints in order to allow the execution of the monitoring probe (see https://github.com/EGI-Federation/nagios-plugins-xrootd ) introduced some months ago.

On GOCDB the EOS endpoints are registered as XrootD service type hence these indications apply for any endpoint associated to XrootD service type.

It was agreed a change on the GOCDB fields to use for monitoring the XrootD endpoints, so please follow the guide in https://docs.egi.eu/internal/configuration-database/adding-service-endpoint/#eos-and-xrootd-service-endpoints

The endpoint url information used for monitoring purposes should be set in the extension properties section. Create the following:
- Name: ARGO_XROOTD_OPS_URL
- Value: XRootD base SURL to test (the path where ops VO has write access)

October 3, 2022 12:36
by Alessandro Paolini

• Service degradation on Tuesday, September 20th

September 13, 2022 15:59 | Gino MARCHETTI
More details

Service degradation on Tuesday, September 20th

Dear all,

Due to a network maintenance by the hosting institute, the service may be degraded on Tuesday, September 20th, from 9:00 to 14:00.

Best regards,
Dirac support.

September 13, 2022 15:59
by Gino MARCHETTI

Home

Broadcast

Downtimes

VO Management

VO List

Dashboard

Metrics

Sla

v6.5 - 2022-10-05