Operations Portal is an EGI service provided by CCIN2P3,
co-funded by EGI.eu and EGI-Engage


Release Version : 3.3 - December 29th

For any contact, use this section: contact us


The Operations Portal is a central portal for the EGI operations management that offers a different capabilities, such as the broadcast tool, VO management facilities and various of dashboards (Security, VO and Operations) to facilitate infrastructure oversight.

Latest news

The data from the  APEL summariser that was fixed yesterday has now propagated through the data pipeline and the Accounting Portal views and the Sync and Pub tests are all working again. 

Any outstanding nagios warnings or errors should go green the next time they update. If they don't they probably represent a real issue with the site and should be investigated.
A problem with the APEL accounting service has been observed and diagnosed. 

The result of this is that there has been no new data published to the Accounting Portal since last week and the Sync and Pub tests are producing warnings. 

No data have been lost, they have all been received safely by APEL but the summariser has been failing for a few days. The summariser output is used by the sync and pub tests.

The summariser is now running but will take many hours to run. It is expected that the issue will be resolved by tomorrow morning. 

A further broadcast will be issued on resolution. 
A hypervisor machine hosting the RAL  gridpp mail server has crashed. Subsequently mails to *@gridpp.rl.ac.uk may be delayed and tickets may not be updated in a timely manner. Some other production servers may also be affected.
Dear all,

EUGridPMA have announced a new set of CA rpms. Based on this IGTF release a new set of CA RPMs have been packaged for EGI. 

Please upgrade until 2016.02.01 at your earliest convenience. When this timeout is over, SAM will throw critical errors on CA tests if old CAs are still detected.

Please check https://wiki.egi.eu/wiki/EGI_IGTF_Release for more details

EGI UMD software provisioning Team

European Grid Infrastructure EGI Trust Anchor release 1.71          2016.01.25

------------------------------------------------------------------------------
   For release DOCUMENTATION available on this EGI Trust Anchor release see   
               https://wiki.egi.eu/wiki/EGI_IGTF_Release                      
------------------------------------------------------------------------------

This is the EGI Trust Anchor release, based on the updated IGTF Accredited CA
distribution version 1.71-1 with Classic, SLCS and MICS profiles, encoded in
meta-package "ca-policy-egi-core-1.71-1" (new installs) and "lcg-CA-1.71-1"
(for sites upgrading from EGEE/JSPG releases).

Your may install BOTH "egi-core" AND "lcg" meta-packages, according to your 
policies. Note that your organisation or NGI may have a specific policy and
may have added or removed CAs compared to the EGI core policy.

The following notices are republished from the IGTF, inasfar as pertinent to
this release. Details are found in the newsletter https://www.eugridpma.org/

Changes from 1.70 to 1.71
-------------------------
(25 January 2016)

* Added accredited classic KENET ICA and associated Root (KE)
* Added roll-over subordinate for the SDG CA G2 (CN)
* Removed expiring SDG CA (CN)
* Updated CyGrid Root CA with extended validity period (CY)
* Updated BG-ACAD-CA with extended validity period (BG)

The CA modifications encoded in both "requires" and "obsoletes" clauses (RPM)
and Conflicts/Replaced clauses (Debian)  have been incorporated in the above-
mentioned meta-packages.  This release is best enjoyed with  fetch-crl v3  or 
better, available from GNU/Linux OS add-on repositories Fedora, EPEL, Debian,
and from the IGTF at https://www.igtf.net/fetch-crl

Version information: ca-policy-egi-core = 1.71-1
Hello and happy new year to all,

this is to inform you that the GR-11-UPATRAS Resource Centre is going to be decommissioned. The downtime of all services will begin in 2 weeks from today and will last for 6 weeks. All the services (CREAM-CE, APEL, SITE_BDII, SRM, gLite-APEL, emi-ARGUS) will be decommissioned on 11th of March 2016.

Best regards,

Vasilis
Dear users,

The problem with the trigger was not completely solved and we have fix it yesterday around 17 h utc.

I've removed all multiple alarms and alarms > 24h .
Some of the assigned alarms are not necessary updated (some recoveries have been missed) .I will check it manually this morning and I will try to update it .

Sorry for the inconvenience .

NB : The effect is multiple alarms for the same host and test  and some alarms (especially related to tickets) are not updated properly .

Don't hesitate to close alarms if you are in this case .

Sorry for the inconvenience

Operations Portal Team

>>> More news <<<

Dear users,

We have encountered some problems with the database and the associated triggers. The effect is multiple alarms for the same host and test .
We have removed these multiple alarms but you might still see some of them . It affects also the recovery of some alarms .

Don't hesitate to close alarms if you are in this case .

Sorry for the inconvenience

Operations Portal Team
Dear site managers,

we would like to inform you that the VOMS server voms1.egee.cesnet.cz will be replaced by the new VOMS server voms1.grid.cesnet.cz. You get this email because your site supports at least one of the VOs hosted on this server.

The new server will be available from January 6th 13:00 UTC, the old VOMS server will be switched off the next day, January 7th at 13:00 UTC. 
In the meantime you can also use another VOMS server voms2.grid.cesnet.cz, which hosts the same VOs as voms1.* servers.

The VOMSES string and the LCS configuration for the voms2.grid.cesnet.cz server can be found at https://voms2.grid.cesnet.cz:8443/voms/$VO/configuration/configuration.action (replace the $VO string with the name of your VO). 

The VOMSES string and the LCS configuration for the voms1.grid.cesnet.cz server will be available at https://voms1.grid.cesnet.cz:8443/voms/$VO/configuration/configuration.action

Please change the configuration of your sites to reflect these changes.

Thank you for your help.

Jan Svec
CESNET
Dear VO managers,

We here at EGI.eu have started to work on an EGI Impact Report. This report, which will be designed as a glossy publication, aims to strengthen EGI’s position as a world-leading e-Infrastructure in preparation for the proposals of the upcoming work programme.

One of the impacts that EGI needs to demonstrate is scientific output. We need to show that thousands of scientists depend on our e-Infrastructure for their daily work. To do this, we propose to create a comprehensive list of scientific publications made possible thanks to EGI services.

No one is closer to the real users than the VO managers and, for this reason, we need your help!

What can you do?
Please send us (sara.coelho@egi.eu / sergio.andreozzi@egi.eu) a list of the scientific publications published with the support of your VO. If you have this list of publications online, please send us the link. If not, an excel or word file will also work for us.
We are looking for publications from, and including, the year 2014.
Ideally, the list should include:
a) DOI
b) Title
c) Access policy (open source, behind paywall, under embargo)
d) Date of publication

In order to improve the collection of this information for the future, we would like to know the exact words of the acknowledgement statement that you recommend to your VO members and that you specify in your Acceptable Use Policy. If we know this statement, we can mine databases and find your publications without having to ask for your time.

If you do not have an acknowledgement statement yet, we recommend you to define one and advertise among your VO members. Consider that we are finalizing a new EGI AUP that empowers you to do so (see article 2: https://documents.egi.eu/document/2623).  Please, consider using the EGI acknowledgement statement: http://go.egi.eu/acknowledge

Please, update the VO ID Card (http://operations-portal.egi.eu/vo/help) with:
a)    the acknowledgement statement that you recommend to your VO members
b)    the types of relationships that we can infer from this statement (e.g., “using:EGI” to say that EGI supports the work, “using VO:X” to say that the work is related to the VO X, “using NGI:FR” to say that the work is supported with resources from France)
c)    the URL to the list of publications generated by members of your VO (if available)

What will EGI.eu do with this information?
a) The publications will be used to demonstrate EGI’s impact in two ways: quantitative impact (number of publications, publications impact factor) & qualitative impact (case studies)
b) Number of publications is also a KPI of the EGI-Engage project
c) The publications will be used as a source of ideas for Case Studies (which are a powerful impact marketing tool). The EGI Case Studies publication (http://go.egi.eu/cst) attracted a lot of attention.
d) The case studies will be used for outreach work and to publicise the platforms and tools available to potential new users.

Example of platform-led case study:
New biomarkers for multiple sclerosis http://www.egi.eu/case-studies/medical/ms_biomarkers.html
And the companion story, about VIP http://www.egi.eu/news-and-media/newsletters/Inspired_Issue_19/vip.html

When?
We will start working on the data received in the first quarter of 2016. Please send us the publications of your NGI and update the VO ID Card by 22 January 2016.

Many thanks in advance for your help! 
Sara Coelho & Sergio Andreozzi
sara.coelho@egi.eu / sergio.andreozzi@egi.eu
Dear Users,

The Operations Portal  3.3 is now online.
The main features  are  : 

1) Security dashboard

- Notifications about security issues
https://operations-portal.egi.eu/csiDashboard/notificationsIssues

- capture WN instead of host : the Worker node is displayed (when available) instead of the host name into the list of issues

- metrics : we have put in place reports for the Nagios issues
https://operations-portal.egi.eu/csiDashboard/report

- sites under certification process : we have extended the list of sites to display potential issues on candidate sites

2) VO ID cards

- New section in the VO ID Card for Portal/Web Service Robot
https://operations-portal.egi.eu/vo/rbCert
        
- add filter input in the vo Update  page      

3) Dashboard

-Non-production alarms in ROD dashboard : if an alarm is raised on a service which is not production in GOCDB it won't be visible in the Operator page

- The template of the tickets has been improved .

Here is the complete list for all features / fixes  :
http://operations-portal.egi.eu/home/tasksList/release_id/19

Don't hesitate to contact us for comments, feedback, bugs at cic-information@in2p3.fr

Thanks by advance


Cheers,
Decommissioning SL5 services

Decommissioning of SL5 on the EGI infrastructure will start from 2016. Here the detailed schedule (based on https://wiki.egi.eu/wiki/PROC16#Decommissioning_deadline):

- January 1st is the decommissioning start date: sites have been warned about the decommissioning update at last December OMB https://indico.egi.eu/indico/conferenceDisplay.py?confId=2383 and Operations Meeting https://wiki.egi.eu/wiki/Agenda-14-12-2015#Decommission_of_SL5

- during January new probe will be deployed and starts returning WARNING;

- February 1st: probe starts returning CRITICAL; NGIs/ROCs open tickets (see steps 2-5 of the procedure) to upgrade their services to supported software or retire them, status will be tracked on
https://wiki.egi.eu/wiki/SL5_retirement;

- decommissioning deadline is March 31st;

- SL5 services must be decommissioned by end of April 2016 (decommissioning deadline + 1 month); by this date unsupported software MUST BE retired from the production infrastructure (they must be decommissioned or in downtime).

Best regards,
Tadeusz Szymocha
On behalf of EGI Operations Support
1. Decommissioning SL5 services

Decommissioning of SL5 on the EGI infrastructure will start from 2016. Here the detailed schedule (based on https://wiki.egi.eu/wiki/PROC16#Decommissioning_deadline):

- January 1st is the decommissioning start date: sites have been warned about the decommissioning update at last December OMB https://indico.egi.eu/indico/conferenceDisplay.py?confId=2383 and Operations Meeting https://wiki.egi.eu/wiki/Agenda-14-12-2015#Decommission_of_SL5

- during January new probe will be deployed and starts returning WARNING;

- February 1st: probe starts returning CRITICAL; NGIs/ROCs open tickets (see steps 2-5 of the procedure) to upgrade their services to supported software or retire them, status will be tracked on
https://wiki.egi.eu/wiki/SL5_retirement;

- decommissioning deadline is March 31st;

- SL5 services must be decommissioned by end of April 2016 (decommissioning deadline + 1 month); by this date unsupported software MUST BE retired from the production infrastructure (they must be decommissioned or in downtime).

2. dCache bug for pools with version >=2.12 configured with Berkeley DB

An advisory has been circulated within WLCG about an issue affecting dCache pools configured to use Berkeley DB as metadata backend, which can lead to data loss.

The problem affects dCache >2.12. The problem has been fixed by developers, please find details on the release notes of each release or from the Wiki:
https://twiki.cern.ch/twiki/pub/LCG/WLCGBaselineVersions/dcache-bug.txt

The dCache versions in UMD (2.10.41) and EMI (2.10.39) are NOT affected.

3. UMD 3.14.1 released

This UMD 3.14.1 release contains the following products for SL6: DPM 1.8.10, FTS3 3.3.1, MyProxy 6.1.15, GridFTP 9.1.0, XROOTD 4.2.3, globus-default-security 6.2.0, ARC 15.03.4. Please find all the details under http://repository.egi.eu/2015/12/18/release-umd-3-14-1/

Best regards,
Tadeusz Szymocha
On behalf of EGI Operations Support
Dear site managers,

In previous broadcast there were some typos. plese use this broadcast.

Starting from the 3rd November 2015 the INFN CA is using a new root certificate.
Unfortunately this change, in particular the fact that there is a *change of the CA DN*, creates issues to the VOs managed through VOMS servers that acquired new server certificates released recently by the INFN CA.
At this moment we have a new server certificate for the voms-02.pd.infn.it, that is supporting the following VOs:
• ams02.cern.ch
• compassit
• comput-er.it
• enmr.eu
• euchina
• euindia
• eumed
• glast.org
• ipv6.hepix.org
• pacs.infn.it
• superbvo.org

Therefore the configuration of grid services (SEs, CEs, UIs, WNs, ...) must be updated to ensure the correct function with the VO proxy certificates.
We would like to kindly ask you to update the LSC files (in /etc/grid-security/vomsdir/"vo_name"/ to match the configuration described here:
https://voms-02.pd.infn.it:8443/voms/"vo_name"/configuration/configuration.action
Please replace "vo_name" with the actual VO name from the ones listed above

You get this email because your site supports at least one VO hosted on this server or you are a VO-manager of one of the interested VOs (just for information). 

Bellow there are some details that can help you:
The steps to be followed in order to update the .lsc file are the following:
a. For services whose configuration is done using *YAIM*:
- Update the /"path_to"/"your_site_info.def" or /"path_to"/vo.d/"vo_name_file" to contain the new CA_DN, for the respective VOs.
For example, for the VO argo, there should be present the following lines:
SW_DIR=$VO_SW_DIR/enmr.eu
DEFAULT_SE=$SE_HOST
STORAGE_DIR=$CLASSIC_STORAGE_DIR/enmr.eu
VOMS_SERVERS="'vomss://voms2.cnaf.infn.it:8443/voms/enmr.eu?/enmr.eu' 'vomss://voms-02.pd.infn.it:8443/voms/enmr.eu?/enmr.eu'"
VOMSES="'enmr.eu voms2.cnaf.infn.it 15014 /C=IT/O=INFN/OU=Host/L=CNAF/CN=voms2.cnaf.infn.it enmr.eu' 'enmr.eu voms-02.pd.infn.it 15014 /C=IT/O=INFN/OU=Host/L=Padova/CN=voms-02.pd.infn.it enmr.eu'"
VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN Certification Authority' '/C=IT/O=INFN/CN=INFN CA'"

- Reconfigure the node by using only the config_vomsdir function:
# /opt/glite/yaim/bin/yaim -d 6 -r -s /"path_to"/"your_site_info.def" -f config_vomsdir 

- Check that the resulted .lsc file is correct
# cat /etc/grid-security/vomsdir/enmr.eu/voms-02.pd.infn.it.lsc
/C=IT/O=INFN/OU=Host/L=Padova/CN=voms-02.pd.infn.it
/C=IT/O=INFN/CN=INFN Certification Authority

b. For services whose configuration is *NOT done through YAIM*, please update your configuration tools, if any, to correctly set the content of the .lsc file for the respective VOs and the VOMS server indicated, like in the example:
# cat /etc/grid-security/vomsdir/enmr.eu/voms-02.pd.infn.it.lsc
/C=IT/O=INFN/OU=Host/L=Padova/CN=voms-02.pd.infn.it
/C=IT/O=INFN/CN=INFN Certification Authority

All the Best,
Sergio Traldi
for the NGI_IT
Dear site managers,

In previous broadcast there were some typo. Use this broadcast.

Starting from the 3rd November 2015 the INFN CA is using a new root certificate.
Unfortunately this change, in particular the fact that there is a *change of the CA DN*, creates issues to the VOs managed through VOMS servers that acquired new server certificates released recently by the INFN CA.
At this moment we have a new server certificate for the voms-01.pd.infn.it, that is supporting the following VOs:
• argo
• cdf
• compchem
• enea
• gridit
• inaf
• infngrid
• pamela
• planck
• theophys
• virgo

Therefore the configuration of grid services (SEs, CEs, UIs, WNs, ...) must be updated to ensure the correct function with the VO proxy certificates.
We would like to kindly ask you to update the LSC files (in /etc/grid-security/vomsdir/"vo_name"/ to match the configuration described here:
https://voms-01.pd.infn.it:8443/voms/"vo_name"/configuration/configuration.action
Please replace "vo_name" with the actual VO name from the ones listed above

You get this email because your site supports at least one VO hosted on this server or you are a VO-manager of one of the interested VOs (just for information). 

Bellow there are some details that can help you:
The steps to be followed in order to update the .lsc file are the following:
a. For services whose configuration is done using *YAIM*:
- Update the /"path_to"/"your_site_info.def" or /"path_to"/vo.d/"vo_name_file" to contain the new CA_DN, for the respective VOs.
For example, for the VO argo, there should be present the following lines:
SW_DIR=$VO_SW_DIR/argo
DEFAULT_SE=$SE_HOST
STORAGE_DIR=$CLASSIC_STORAGE_DIR/argo
VOMS_SERVERS="'vomss://voms.cnaf.infn.it:8443/voms/argo?/argo' 'vomss://voms-01.pd.infn.it:8443/voms/argo?/argo'"
VOMSES="'argo voms.cnaf.infn.it 15012 /C=IT/O=INFN/OU=Host/L=CNAF/CN=voms.cnaf.infn.it argo' 'argo voms-01.pd.infn.it 15012 /C=IT/O=INFN/OU=Host/L=Padova/CN=voms-01.pd.infn.it argo'"
VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN Certification Authority' '/C=IT/O=INFN/CN=INFN CA'"

- Reconfigure the node by using only the config_vomsdir function:
# /opt/glite/yaim/bin/yaim -d 6 -r -s /"path_to"/"your_site_info.def" -f config_vomsdir 

- Check that the resulted .lsc file is correct
# cat /etc/grid-security/vomsdir/argo/voms-01.pd.infn.it.lsc
/C=IT/O=INFN/OU=Host/L=Padova/CN=voms-01.pd.infn.it
/C=IT/O=INFN/CN=INFN Certification Authority

b. For services whose configuration is *NOT done through YAIM*, please update your configuration tools, if any, to correctly set the content of the .lsc file for the respective VOs and the VOMS server indicated, like in the example:
# cat /etc/grid-security/vomsdir/argo/voms-01.pd.infn.it.lsc
/C=IT/O=INFN/OU=Host/L=Padova/CN=voms-01.pd.infn.it
/C=IT/O=INFN/CN=INFN Certification Authority

All the Best,
Sergio Traldi
for the NGI_IT
Dear site managers,

Starting from the 3rd November 2015 the INFN CA is using a new root certificate.
Unfortunately this change, in particular the fact that there is a *change of the CA DN*, creates issues to the VOs managed through VOMS servers that acquired new server certificates released recently by the INFN CA.
At this moment we have a new server certificate for the voms-02.pd.infn.it, that is supporting the following VOs:
• ams02.cern.ch
• compassit
• comput-er.it
• enmr.eu
• euchina
• euindia
• eumed
• glast.org
• ipv6.hepix.org
• pacs.infn.it
• superbvo.org

Therefore the configuration of grid services (SEs, CEs, UIs, WNs, ...) must be updated to ensure the correct function with the VO proxy certificates.
We would like to kindly ask you to update the LSC files (in /etc/grid-security/vomsdir/"vo_name"/ to match the configuration described here:
https://voms-02.pd.infn.it:8443/voms/"vo_name"/configuration/configuration.action
Please replace "vo_name" with the actual VO name from the ones listed above

You get this email because your site supports at least one VO hosted on this server or you are a VO-manager of one of the interested VOs (just for information). 

Bellow there are some details that can help you:
The steps to be followed in order to update the .lsc file are the following:
a. For services whose configuration is done using *YAIM*:
- Update the /"path_to"/"your_site_info.def" or /"path_to"/vo.d/"vo_name_file" to contain the new CA_DN, for the respective VOs.
For example, for the VO argo, there should be present the following lines:
SW_DIR=$VO_SW_DIR/enmr.eu
DEFAULT_SE=$SE_HOST
STORAGE_DIR=$CLASSIC_STORAGE_DIR/enmr.eu
VOMS_SERVERS="'vomss://voms2.cnaf.infn.it:8443/voms/enmr.eu?/enmr.eu' 'vomss://voms-02.pd.infn.it:8443/voms/enmr.eu?/enmr.eu'"
VOMSES="'enmr.eu voms2.cnaf.infn.it 15012 /C=IT/O=INFN/OU=Host/L=CNAF/CN=voms2.cnaf.infn.it enmr.eu' 'enmr.eu voms-02.pd.infn.it 15012 /C=IT/O=INFN/OU=Host/L=Padova/CN=voms-02.pd.infn.it enmr.eu'"
VOMS_CA_DN="'/C=IT/O=INFN/CN=INFN Certification Authority' '/C=IT/O=INFN/CN=INFN CA'"

- Reconfigure the node by using only the config_vomsdir function:
# /opt/glite/yaim/bin/yaim -d 6 -r -s /"path_to"/"your_site_info.def" -f config_vomsdir 

- Check that the resulted .lsc file is correct
# cat /etc/grid-security/vomsdir/enmr.eu/voms-02.pd.infn.it.lsc
/C=IT/O=INFN/OU=Host/L=CNAF/CN=voms-02.pd.infn.it
/C=IT/O=INFN/CN=INFN Certification Authority

b. For services whose configuration is *NOT done through YAIM*, please update your configuration tools, if any, to correctly set the content of the .lsc file for the respective VOs and the VOMS server indicated, like in the example:
# cat /etc/grid-security/vomsdir/enmr.eu/voms-02.pd.infn.it.lsc
/C=IT/O=INFN/OU=Host/L=CNAF/CN=voms-02.pd.infn.it
/C=IT/O=INFN/CN=INFN Certification Authority

All the Best,
Sergio Traldi
for the NGI_IT