Large Scale System Configuration Workshop

This event was an invited workshop for experienced practitioners to discuss all aspects of installing and maintaining system configurations on large numbers of nodes, including GRID farms, servers, and desktop workstations. A major aim was to bring together people from different backgrounds to share experiences, and inform the development of the new techniques which will be essential for the management of the very large numbers of nodes being envisaged for the next generation of GRID systems.

Talks

Short Presentations

Discussion Summary

(Prepared by Jessie Paterson, University of Edinburgh)

The discussion was focused around a series of issues that were proposed in advance of the workshop and also a few additional ones that came up during the talks. These have been roughly categorized and listed together with the names of the people who suggested the topic.

1. Declarative vs. Procedural Specifications - Will Partain, Lionel Cons, Patrick Goldsack

Question set: - If we have a declarative description of a configuration, to what extent can we "prove" that it isn't nonsense? How close can we come to a "(sub-) system prover" (cf. "theorem prover") or "system checker" (cf. "type checker")? Assuming the overall problem is Quite Hard, what bits of it are tractable (and useful)?

Is it true that we can't have a completely declarative high-level specification, and we do need to include some kind of code? Do we need completely arbitrary code, or can we restrict it in some way? What does this mean for validation (and comprehensibility?)?

Discussion: - In summary it was generally agreed that declarative specification was a "good idea" but most people felt that was sometimes necessary to "escape" to procedural code in practice, although this code should be "done well".

Two opposite views were voiced in terms of the LCFG model against the Arusha model. In LCFG the source contains no general "escapes", with any requirement for arbitrary procedural code "pushed" down the tree to the component level on client machines. Arusha takes the stance that the collaborative nature means it is necessary to allow contributors to use various escapes to keep the code simpler and understandable (too abstract they won't use). Continuous "hacking" was not advocated and at some point there was a need to stand back and change the language. This was backed up by the statement that code "evolves".

Two aspects of the problem were also identified; the completeness of any procedural language, and the scope of the data to which such code might have access. A reasonably complete language with restricted data access was a common feeling.

2. Aspects - Paul Anderson, Peter Dickman

Question set: - Different "aspects" of a node configuration in a large site are often managed by different people; for example the "web server manager", or the Solaris operating system manager. Often these aspects overlap. How can we provide a high-level way for these managers to control their own views of the configuration, and resolve conflicts between them?

How do we manage systems in which the administrative needs "overlap" e.g. Clusters A1, A2 owned by admin A, Clusters B1, B2 owned by admin B but a particular project uses A1 & B1 and requires dynamic management?

Discussion: - Across the participants there was no common or all encompassing solution to this problem, although it was recognized as an increasingly important issue.

Obviously the need and complexity increase with the size of the system. Solutions varied, with the both the CERN system, and Arusha relying on ordering of the configuration information, where in the case of two conflicting configurations the first (or last) one wins. The CERN system augmented this with the "FINAL" statement in the language, and support for formulas to resolve some issues automatically (e.g. disc space). LCFG also uses a similar system with arbitrary fragments of Perl code supported for combining conflicting values.

This question also relates to managing different access rights to the configuration files or database: no novel approaches to this problem were suggested.

3. Dynamic vs. Static Parameters - Lionel Cons, Paul Anderson

Question set: - It's nice to validate what is in the configuration database (e.g. that a package to install is indeed available from a repository or that a remotely mounted filesystem is indeed exported by the server). How to do it when the validation depends on some data, which is not in the configuration database but in other independent databases (e.g. Human Resources database)?

There are many advantages to having a central repository of declarative configuration information for a whole cluster. However, it is very useful for nodes to make some changes to their configuration autonomously, in response to the state of the local machine, or information obtained from other nodes. Can this be done in such a way as not to lose the benefits of the central, declarative configuration description?

Discussion: - In some ways this boiled down to the same arguments as the declarative versus procedural specification with the view that static would be "nice" but is hard. As systems grow (or became more disconnected) the need for some dynamic parameters becomes more important.

The LCFG context mechanism was given as an example of one solution - for laptops connected to various different networks. The SmartFrog approach was very flexible , providing "lazy evaluation" at runtime.

4. Versions - Lionel Cons, Alastair Scobie

Question set: - The configuration data will be used by some programs, which are not stored in the configuration database. How to make sure that the version of the data matches the version of the program, for instance when the structure of the data has changed?

How do we support multiple simultaneous versions of components with possibly different configuration schema - is this just a namespace problem?

Discussion: - This discussion quickly focussed on a need for sequencing (following topic) and showed that these two topics are highly inter-related.

Within SmartFrog there is no explicit support for versions, but there is support for a component "lifecycle", while the CERN DataGrid model plans to rely more on database control. With the SmartFrog system it was thought necessary not to stop a system, if it was related to another component that was still running and some sort of asynchronous approach was required. The DataGrid approach was more controlled, allowing component states to be forced under central control.

5. Sequencing - Paul Anderson, German Cancio

Question set: - A change to the configuration of a cluster may involve changes to individual nodes, which are interdependent. Should these changes be sequenced by some central agent, or should there be a peer-to-peer mechanism by which nodes can monitor the actual configuration state of other nodes?

Certain configuration changes on a node are more "intrusive" than others; for example, upgrading the kernel is more intrusive than changing the mail relay. If several configuration changes are required, do they need to be implemented strictly in order, or can a node safely delay "intrusive" changes? If so, when and how? How do we guarantee that the node remains consistent (secure?) during these transitions?

Discussion: - No one in the group had the ideal universal solution to either sequencing or intrusive operations. It was thought that with intrusive operations it was "safer" to gradually move the system over. However, it was recognized that this will take time and the intermediate states may be unstable. SmartFrog relied on supplying a sequencing toolkit (discovery protocols, partition managers etc) to choose whatever was appropriate at the time, while most others didn't really have any solution for the present time.

6. High-Level Specifications - Piotr Poznanski, Paul Anderson

Question set: - (High level) configuration description languages/abstractions and their transformation.

Is it possible/necessary/desirable to have a high-level configuration language that has a better way of representing configuration aspects relating to the relationships between machines?

Discussion: - Not explicitly discussed.

7. Scaling - Joseph Sventek, Paul Anderson, Peter Dickman

Question set: - Configuration in the face of extreme scale.

Cloning techniques tend to have fallen out of favour for large-scale workstation management due to their lack of inflexibility. However, most current large clusters seem to use cloning for configuring and installing their nodes. Is there some reason (perhaps performance?) why cloning is preferable (necessary?) in these cases?

How do we manage rapid re-configuration in order to support specific needs (e.g. the Cambridge Distributed System Tripos processor bank in the early 80's allocated a machine on demand and booted the requested OS, for a given user for that 'session'. What's the modern equivalent and can we do it?)

Discussion: - Most people recognized that the ability to scale depended on many factors - for example, network topology, and that exact performance was difficult to predict. However, many people felt that a highly distributed approach, with individual nodes autonomously converging towards a desired state, was more likely to scale than models requiring a tighter central control. LCFG is managing 300-400 nodes with a single configuration server and repository.

8. What is this subject? - Huw Evans, Paul Anderson

Question set: - What's the state of the art in this area in general so that those that attend can go away with some references for further study?

In the past, it is has seemed difficult to find people doing similar work to ourselves, but there are people at the workshop from different backgrounds, and there seem to be some common problems and philosophies emerging. (Where) are there people doing work in other areas, which might be relevant?

Distributed systems need to be able to deploy objects at run time. What's the commonality between system configuration as described in the above questions, and object deployment in the sense of distributed systems?

Discussion: - It was generally felt that at least there is an awareness of the subject but that in some way it was still early days. Over the last five years there has been an increase in papers submitted to LISA that show that when systems grow there is a need to consider configuration of large scale systems in an automated way.

It was felt that some people working on large systems (e.g. ISP's, e-commerce) were working in such hostile environments that their approach to configuration is probably to build in large redundancy where they can simply swap in and out nodes as required - no downtime, or slack allowed.

9. Methodology & Principles - Will Partain, Paul Anderson

Question set: - Are there "do's and don'ts" for a 'methodology' (i.e. the systematic interactions among the people managing a system) that follow from a particular (presumably "good") configuration spec/description?

A number of common themes seem to be emerging from people working independently, both in terms of recognized problems, and in terms of important properties. Is this an indication that we have some fundamental principles, or is this just too narrow a group. Can we identify these themes? For example:

• Declarative specifications are good, but some procedural content is inevitable.
• We can't "tightly" manage the configuration of large systems - we have to let the configurations migrate towards a designated specification.

Discussion: - Not explicitly discussed.

10. Culture - Peter Dickman, Paul Anderson

Question set: - How does the working culture and environment change the approach to configuration management?

Discussion: - Generally agreed that culture introduced varying approaches to the problem - peer-to-peer and hierarchical being at least two. It was also felt that policy and mechanisms should be kept separate and that although this was virtually impossible to achieve it should still be the goal. One interesting comment was that in some way the "root" of the problem was that everyone is trying to work with software systems (including operating systems) that were not were designed to be flexibly configured - making the problem hard so each will use what ever approach they feel most comfortable with.

Conclusions and other general points

• Databases were seen as useful for holding large quantities of "compiled" resources - or for holding per-machine configuration parameters which are independent of the rest of the system -but not as the primary "source" of the configuration data (specification languages are necessary for this).
• Most people thought that declarative specification languages were good but agreed that some kind of procedural "escape" were necessary in some cases. The ability to perform static validation of configurations was seen as an important property of declarative specifications.
• Prototype-based languages appeared independently in several systems, suggesting that they might be particularly suitable for this type of application. There are a variety of different approaches to prototype combination. On the whole the semantics of prototype combination in configuration systems seems to stem more from what turns out to be most needed, rather than - say - from the semantics of inheritance in object-oriented programming languages.
• Most people thought that explicit, static declarations of configuration parameters were desirable, but some dynamic configuration information may not be available until runtime, and is unavoidable. Several mechanisms including lazy binding of some variables, and "LCFG "contexts" were being used to support this. In addition, the demands of very large scale installations may require that some of this dynamic information is obtained via service discovery protocols, or similar mechanisms.
• The ability to compose host configurations from several independent aspects was seen as important.
• Different approaches to the configuration problem may be appropriate for sites with different priorities and cultures.

Attendees

• Paul Anderson <paul@dcs.ed.ac.uk>
• German Cancio <german.cancio.melia@cern.ch>
• Lionel Cons <lionel.cons@cern.ch>
• Peter Dickman <pd@dcs.gla.ac.uk>
• Alasdair Earl <aearl@ph.ed.ac.uk>
• Huw Evans <huw@dcs.gla.ac.uk>
• Michael George <mpg@hep.ph.liv.ac.uk>
• Patrick Goldsack <patrick_goldsack@hpl.hp.com>
• Matt Holgate <matt@dcs.gla.ac.uk>
• Alexander Holt <lex@fixedpoint.org>
• Adam Moskowitz <adamm@menlo.com>
• Will Partain <partain@dcs.gla.ac.uk>
• Jessie Paterson <Jessie.Paterson@ed.ac.uk>
• Piotr Poznanski <piotr.poznanski@cern.ch>
• Alastair Scobie <ajs@dcs.ed.ac.uk>
• Joe Sventek <joe_sventek@agilent.com>
• Peter Toft <peter_toft@hpl.hp.com>
• Simon Wilkinson <simon@sxw.org.uk>