• Created by Unknown User (mlinden@csc.fi), last modified on May 06, 2015

Approved by the pilot steering group 27 March 2015. 

Summary

In the pilot, Moonshot login was integrated to CSC’s Taito supercomputer (SSH secure shell) and IDA data archive (iRODS command line). A handful of real end users (researchers) piloted Moonshot login to the Taito supercomputer from one Finnish research university.

The pilot still hampered from some technical limitations, such as

  • The need to install software to client devices complicates adoption

  • The missing Windows support hinders a large-scale adoption

  • Server side SSH secure shell functionality is missing from the OpenSSH upstream packages

  • There is no well understood way to do the user-to-account mapping in the service provider side

The technology in general was seen as beneficial from the service provider perspective. However, moonshot SSH secure shell does not provide significant benefits for existing users who are normally using public key authentication. Instead, the main benefit is the lower threshold to start to use CSC services; an end user does not need to learn an extra username and password to access CSC services. However, to gain these benefits, support to Moonshot needs to be properly integrated to the general CSC identity management and user provisioning processes.

Introduction

Moonshot is an emerging technology for federated access for non-web services. The technology combines elements from network access control (RADIUS), web application access control (SAML) and operating system security services (GSS API). The project has been led by Janet(UK) and funded by the GN3plus project. For more information on Moonshot, see http://www.project-moonshot.org/.

To get some hands-on experience on the technology and its applicability, CSC – the IT Center for Science Ltd has participated in the Moonshot pilot funded by the GN3plus project. In the pilot, CSC integrated Moonshot login to two of its services (SSH secure shell access to Taito supercomputer and iRODS command line access to IDA storage service) and one Finnish universities assumed the role of a Moonshot Identity Provider. To get some real user feedback, a handful of researchers using a moonshot CSC service were used as pilot users.

This document is the final report of the pilot, describing the pilot motivations, environment and findings. 

Description of CSC and its related services

CSC - IT Center for Science Ltd is a non-profit company providing IT support and resources for academia, research institutes and companies: modeling, computing and information services. CSC provides Finland's widest selection of scientific software and databases and Finland's most powerful supercomputing environment that researchers can use via the Funet network. CSC is owned by the Finnish Ministry of Education, Science and Culture.

Taito is a high performance supercluster providing resources for serial, medium parallel and cloud use. Taito consists of 576 HP ProLiant SL230s servers each equipped with two Intel Xeon 2.6 GHz E5-2670 CPUs and of 407 Apollo 6000 XL230a G9 server blades, each with two twelve core Intel Haswell E5-2690v3 CPUs. The current installation has a total theoretical capacity of 600 TFlop/s.

The Ministry of Education and Culture funds Finnish higher education institution researchers and research data management services for certain research projects funded by the Academy of Finland. One of the services is IDA, a secure, user-friendly storage service for data and metadata. During the pilot the IDA storage service was based on open-source iRODS technology (version 3.3.1). Recently, the iRODS technology has been upgraded to version 4.

Haka is the SAML based identity federation for the Finnish research and higher education. Researchers, teachers, students and other users in 44 home organisations can access 191 services with a single username and password. Haka covers all universities and universities of applied science that belong to the jurisdiction of the Ministry of Education and Culture. Haka is connected to the Nordic Kalmar2 confederation and the global eduGAIN interfederation service.

Funet, the Finnish university and research network, is the data communication network for the Finnish research and education. Funet has approximately 350 000 users. One of the associated services is wireless network roaming. CSC coordinates the eduroam service in Finland.

The steps that led to the project

Moonshot related preparations at CSC started in 1/2012 with an internal workshop where Rhys Smith of Janet was an invited guest speaker. The workshop had representatives from various CSC teams, including computing and data services, Haka and eduroam federations and the company-level information security management. As a result of the workshop, a CSC internal special interest group (SIG) was formed to further prepare Moonshot related issues at CSC.

Before September 2012, the SIG hammered out CSC’s Moonshot vision: CSC's customers from Finnish and foreign universities and research institutions are able to access CSC services securely using their home organisation authentication credentials and single sign-on. The SIG also studied potential Moonshot use cases at CSC, two of which were selected for further examination: SSH Secure Shell access to the computing environment and iRODS command line access to the IDA storage service. Individual staff members interested in working for Moonshot were identified.

A separate discussion was started if Moonshot related issues should be seen as an extension of the eduroam service (operated by the Funet service team) or the Haka federation (operated by the Haka service team). Although the underlying RADIUS technology and servers were used in eduroam, it was decided that Moonshot has more in common with the Haka web single sign-on service. Haka shares many of the policy related issues with Moonshot, such as reliability of authentication and data protection. Furthermore, in the customer organisations, network administrators managing wireless networks do not necessarily manage access to applications.

In 9/2012, Haka federation steering group decided to include a pilot on Moonshot to Haka federation Plan of Action for 2013 (and later also for 2014). Furthermore, CSC applied funding for Moonshot from the upcoming GN3plus project and received 14 PM for the two year period of the project (4/2013-3/2015).

To organise the Moonshot pilot in Finland and locally at CSC, CSC’s Moonshot pilot project was chartered 3/2013. The project was due to end in 12/2013 and deliver a pilot where real users from two Finnish universities can use Moonshot technology to access two CSC services; SSH secure shell to Taito computing server and iRODS command line to access IDA storage service. The pilot was later extended by 15 months due to delays with the pilot universities and immaturity of the technology. This document is the final report of the pilot.

Motivations for the pilot

Following motivations were identified for Moonshot and the pilot

  • The ease of use of the CSC services. Having no need to learn a new username and password was seen as a key step towards lowering the threshold for researchers to become CSC customers.
  • Increasing information security. There have been incidents where an intruder has managed to exploit a vulnerability to install a rootkit that collects users’ usernames and passwords for later misuse. Using Moonshot technology, the users’ password is never exposed to the server in cleartext, reducing the impact of a security incident.
  • Easier international collaboration. In the long run, Moonshot technology is assumed to ease the international collaboration for the researchers in Finland.

Pilot environment

Like most supercomputers, Taito is based on end users using their personal accounts to submit their jobs to the computing server’s batch job system. It is possible to use a web browser and an ordinary federated login (via Haka federation or Kalmar confederation) to Taito’s web front end (called Scientist’s user interface, sui.csc.fi), but many researchers prefer to use SSH secure shell for a command line session. For the SSH login, it is possible (and, among some research groups, popular) to use public key authentication.

For Taito login, a user needs to have a personal account at CSC, with an associated username and password. A separate and parallel Identity Management (IdM) project is streamlining the process for applying for and managing the accounts at CSC. The IdM project has delivered an LDAP directory which Taito computing server uses to authenticate the users. Among other things, the IdM system is able to map users’ federated identity (eduPersonPrincipalName, ePPN attribute) to his/her CSC-account (CSC-uid).

IDA storage service is based on the popular open source iRODS software. Like Taito, IDA users need a personal account for using the IDA service. Since the beginning, federated web single sign-on login to IDA has been available via Haka federation, but there is also a possibility to access IDA on a command line, using the i-commands of iRODS software.

CSC is an IT service organization that does not carry out research by itself. CSC’s services are used by researchers in universities and research institutions. There were two user organisations participating in the pilot; University of Helsinki and Tampere University of Technology. As the largest university in Finland, University of Helsinki is a major user of the IDA service. With focus on physics and biochemistry, Tampere University of Technology is a large user of CSC’s computing services. Helsinki University and Tampere University of Technology volunteered as home organisations for the IDA and Taito pilot.

Technical environment

The figure below presents the big picture of the pilot setup.

 

University of Helsinki and Tampere University of Technology had both deployed a Moonshot IdP which had a shared secret with CSC’s RADIUS proxy to guarantee message integrity. For internal test purposes, also CSC had deployed a Moonshot IdP for its own staff (not shown in the figure). Based on the domain part of the username, the RADIUS proxy routed the RADIUS requests to the proper IdP. No trust router or any other routing system was in place in the pilot.

After a successful authentication the IdPs delivered a RADIUS accept message, carrying an unsigned SAML assertion with the user’s ePPN attribute. The IdPs were configured to always deliver the assertion to the RADIUS proxy managed by CSC. In the beginning, all RADIUS messages were exchanged on UDP. In 12/2014, the communication between the Moonshot SPs and the RADIUS proxy was upgraded to use Radsec.

At University of Helsinki, the IdP consisted of a pilot RADIUS server (FreeRADIUS) that authenticated the user against the university's production RADIUS server and constructed a SAML assertion using the FreeRADIUS string manipulation functions. At Tampere University of Technology, the IdP consisted of a Radiator server to which the server admin had deployed a simple plugin that fetches the user attributes from the local LDAP and assembles a SAML assertion to be sent to the requestor.

The RADIUS proxy was a FREERADIUS installation dedicated for the pilot. Most importantly, the RADIUS proxy received users’ ePPN attribute from the SAML assertion and performed an LDAP query to the LDAP front end of CSC’s internal IdM system. The RADIUS proxy then placed the user’s CSC-uid to the RADIUS Chargeable User Identity and User Name attribute and passed the message to the service (Moonshot SP) to which the user was logging in. If the ePPN value did not map to any CSC account, the proxy assigned the CSC-uid value "nobody" and the Moonshot SPs denied the access based on it.

The two Moonshot SPs then started an ordinary session for the user. Taito server performed an ordinary LDAP query to learn the user’s home directory, login shell and other necessary parameters to initiate a command line session. IDA service had the necessary parameters in its local database.

For the pilot, Taito had a dedicated moonshot login node that was running the OpenSSH codebase provided by the Moonshot project. End users were using an OpenSSH client on a Linux client device. The first cross-organisational moonshot SSH login to Taito was carried out on 11th March, 2014. In 8/2014 the Moonshot project delivered also a moonshot PuTTY SSH secure shell client, which added support to Windows client devices, but the pilot was unable to make it work against Taito.

Before the pilot, there was no known previous integration of Moonshot to the iRODS system. Adaptation of the iRODS software (ver 3.3.1) for Moonshot was done in the Pathway project and the moonshot version of iRODS deployed to IDA service on 22nd of May, 2014. The i-command clients including Moonshot support were distributed in rpm- and deb-packages to pilot users. At the time of publishing the pilot report (6th May 2015), CSC has finished the work to integrate Moonshot to iRODS ver 4 which has a new architecture allowing dynamic authentication mechanism installation through plugins. However, a pending bug in the mech eap library makes the iRODS process crash when the mech eap library closes. When the Moonshot project has fixed the bug, CSC will ask iRODS developers to add support to Moonshot to the iRODS ver 4 upstream.

The pilot suffered a setback in 10/2014, when the Moonshot project unexpectedly dropped support to EAP-PEAP that was in use in the Finnish deployment. It took two months to locate and analyse the problem and to test and deploy the fix to the pilot environment. The operational pilot was restored in 12/2014, now using EAP-TTLS.

Pilot

The IT service units of University of Helsinki and Tampere University of Technology were asked to identify a handful of real end users (researchers) who use CSC’s Taito or IDA service.

For Tampere University of Technology, the requirement was to identify some Taito users whose client devices were running the Red Hat Linux administrated centrally by the IT service unit so that the users do not need to install and configure the necessary software by themselves. Two pilot users volunteered to the pilot, and they both carried out their first successful logins in 6/2014.

University of Helsinki was asked to identify some IDA users (researchers) who manage their own Linux client environments. Besides the IT staff involved in setting up and testing the pilot environment, there were no other end users using the moonshot IDA service before the end of the pilot.

Pilot findings

From the SSH secure shell end user perspectice

During the pilot, three pilot users had logged in to Taito 1-30 times each using Moonshot.

For two pilot users, the necessary client software was installed by the university IT service unit who also assisted in configuring the client properly. The third pilot user had installed the client software himself and complained about installation and configuration problems. It appears that the preferred approach is that the user does not need to install and configure the client environment him/herself.

The pilot users gave mixed feedback on if the Moonshot eases the use of CSC services. In general, the pilot users were happy if there are fewer usernames and passwords to remember, but they were normally using public key authentication for SSH secure shell, which is as simple as Moonshot but does not require any extra software in the client device. It appears that for existing SSH secure shell users, Moonshot login does not provide much extra. Instead, the benefit is for the new users who otherwise need to learn a CSC uid/password for their first login. After the first login, the users are going to use a public key for login, anyway.

Following usability issues were reported by the pilot users:

  • Moonshot login was 2-3 seconds slower than ordinary SSH secure shell login with username/password (this issue was later fixed by indexing CSC's LDAP directory by the eduPersonPrincipalName attribute)

  • When a Moonshot-enabled SSH client establishes a connection to an SSH server that does not support Moonshot, the client waits for a pretty long time until a timeout occurs and the client continues to the other authentication methods as a fallback. This extra waiting frustrates the user (to fix this issue, disable GSSAPI on command line by ssh -k or change authentication method order e.g. ssh -o PreferredAuthentications="password,keyboard-interactive,publickey,gssapi-with-mic")

  • The SSH client opens a Moonshot dialogue even if the user is logging in to an SSH server that does not support Moonshot. This is confusing and in any case an extra step that should be removed (to fix this issue, specify which hosts do or do not use GSSAPIAuthentication in the ssh_config file) 

  • Giving the wrong password to Moonshot-UI makes all moonshot logins fail, without a chance of re-entering the password (the Moonshot developers say they are working on this)
  • It would be good to have only a text shell without a UI (public key login works that way), for instance, for clients without X11 support

From the SSH system administrators perspective

Once the Moonshot SSH secure shell service was opened for end users, it worked quite reliably.

Even if integrating Moonshot to the running infrastructures was not terribly hard, it has some challenging aspects.

  • The documentation was lacking and there was a trial-and-error process to find the specific requirements for the scenarios.
  • The Shibboleth/Moonshot-EAP interplay documentation was vague. Figuring out how to map the users to their local accounts took a while, and in the end of the pilot the setup was still not optimal. Different scenarios for Moonshot deployment should be described for and some configuration recommendations made. (See related documentation in the Moonshot wiki)
  • Debugging errors was challenging. It was hard to get the relevant information from the log files.
  • SSH server packages needed to be patched manually. Deploying the patches against the latest server versions also required manual work, since it had received other patches after the Moonshot patch was created. Having to compile packages makes it hard to follow the security updates. Needless to say, this is not optimal, and even dangerous for components like SSH secure shell.
  • Load balancers seem to be hard to use with Moonshot. GSS seems to rely a lot on host names, and with load balancers you probably don't get the hostname you connect to. This problem is still unsolved.
  • The lack of SSH-agent like forwarding for Moonshot greatly limits the utility of Moonshot, and how to provide moonshot services. The fact that only the first connection out from a user's machine can be done with Moonshot (securely) severely limits possible use cases in moderately complex setups.

From the RADIUS administrators perspective

There were multiple ways to generate the assertion received from the organisational Moonshot IdP. At CSC, during the pilot, both the Freeradius-pysaml2 module and the RADIUS generated assertions were used. Basically the only difference between these two alternatives was that the Freeradius-pysaml2 module glued a SAML IdP and a Moonshot IdP together by requesting assertion from the SAML AA and the RADIUS generated way created assertion by itself without the assistance of any other components.

There were a couple of issues to tackle in our way to the successful Moonshot logins.

  • RADIUS uses UDP and, because the SAML assertions mounted on the RADIUS responses make them bigger than usual, they sometimes exceed the protocol layer's MTU size and make the RADIUS packet fragment. In the pilot, we faced a situation where the clients did not get the full RADIUS response but just the first part of it. After some studying we found that there was a stateless firewall between the RADIUS proxy and the Moonshot SP. In the stateless firewall fragments were denied by default and therefore only the first fragment was passed through. This lead to the situation where the client never received the complete response and kept on waiting until a timeout occurred. This was fixed by configuring the firewall to allow fragments for certain hosts. Use of Radsec should also prevent this from happening.
  • There was also some delay when we did eduPersonPrincipalName (ePPN) to username mapping. This was caused by our LDAP cluster where the ePPN attribute was not indexed. Therefore searching username by ePPN caused the ldapsearch to go through the whole directory. After indexing, the login delay was reduced noticeably. 

Relationship to CSC's Identity Management processes

A conclusion of the project was that Moonshot benefits more new users than existing users, because most existing users are using public key authentication and not local usernames and passwords. The new users, instead, must use a local CSC username and password at least for the first login, after which they can start to use public key authentication.

Therefore, the main benefit of Moonshot is lowering the threshold for the new users to start to use CSC services, or to ease the use of CSC services for the irregular users who do not use public key authentication. Extending the user base to reach  those researchers who don't normally use CSC services is an important goal, though.

However, Moonshot login needs to be seen as a piece in a larger picture of the identity management and user provisioning of CSC services. Moonshot is not the proper tool for creating user accounts for new CSC users and authorising the use of CSC services. Instead, a proper division of functionality needs to be developed together with CSC's internal identity management project.

The workflow could be, for instance, the following (see the figure above):

  1. A researcher of a Finnish university performs an ordinary Haka web single sign-on login to a CSC website dedicated for registering as a CSC user. This is doable because all Finnish universities and polytechnics in the jurisdiction of the Ministry of education belong to Haka.
  2. In the website, the users commit to the terms of use of the CSC services by clicking "I approve" button, which is logged for audit trail. This is doable, because Haka federation has a relatively high standard for the reliability of authentication.
  3. If needed, the user's registration can be exposed to an approval process by the CSC usermanager. However, this is not necessarily needed because the Ministry of education buys most CSC's services to all researchers in Finnish universities and polytechnics. The end user's affiliation as a researcher can be verified from the attributes (such as, the eduPersonAffiliation, ePA attribute) delivered by the Haka federation.
  4. Once approved, the user's account can be created to CSC's identity management system, and his/her eduPersonPrincipalName (ePPN) attribute associated to the account, allowing him/her to perform a subsequent login to a CSC maching using Moonshot.

Future steps

  • Continue the pilot, but don’t aggressively expand the pilot to new home organisations and services until the technology gets more mature (Windows support, trust router, documentation)

  • Continue to work with PuTTY to have also a Windows SSH secure shell client

  • Continue integrating Moonshot to iRODS ver 4, with the goal to have the integration published in the iRODS codebase

  • Get more hands on experience with Trust router together with Janet(UK) and other NRENS, and potentially with some research infrastructures relying on Moonshot

  • Collaborate with other GEANT Moonshot pilot partners to enable cross-national login to CSC services

  • Start to prepare a policy to introduce Moonshot non-web single sign-on as an extension of the Haka identity federation

Acknowledgements

The research leading to these results has received funding from the European Community’s Seventh Framework Programme (FP7 2007–2013) under Grant Agreement No. 605243 (GN3plus).

The Pathway research project is funded by Academy of Finland (Grant No. 255836).


  • No labels