Difference between revisions of "Conversion Process"

From GridPP Wiki
Jump to: navigation, search
 
(No difference)

Latest revision as of 13:10, 26 July 2012

Context

GridPP can be viewed as a set of operational sites related to regions, comprising:

Site Region
UKI-LT2-BRUNEL London Grid
UKI-LT2-IC-HEP London Grid
UKI-LT2-QMUL London Grid
UKI-LT2-RHUL London Grid
UKI-LT2-UCL-CENTRAL London Grid
UKI-LT2-UCL-HEP London Grid
UKI-NORTHGRID-LANCS-HEP North Grid
UKI-NORTHGRID-LIV-HEP North Grid
UKI-NORTHGRID-MAN-HEP North Grid
UKI-NORTHGRID-SHEF-HEP North Grid
UKI-SCOTGRID-DURHAM ScotGrid
UKI-SCOTGRID-ECDF ScotGrid
UKI-SCOTGRID-GLASGOW ScotGrid
UKI-SOUTHGRID-BHAM-HEP SouthGrid
UKI-SOUTHGRID-BRIS-HEP SouthGrid
UKI-SOUTHGRID-CAM-HEP SouthGrid
EFDA-JET SouthGrid
UKI-SOUTHGRID-OX-HEP SouthGrid
UKI-SOUTHGRID-RALPP SouthGrid
RAL-LCG2-Tier-1 Special

A wiki site, https://www.gridpp.ac.uk/wiki, is used to communicate. One of the purposes of the wiki is to collect information from the administrators of each site relating to the status of their site.

Description of Problem

For historical reasons, the pages containing the status information have been organised in a ad hoc manner according to the topic they involve. The names of the current set of status pages gives an insight into their purpose: Middleware_transition, Site_information, HEPSPEC06, Site_status_and_plans, Protected_Site_networking, Resiliency_and_Disaster_Planning, SL4_Survey.

Unfortunately, this ad hoc organisation turned out to be unsatisfactory because it did not match the organisation of GridPP itself leading to two problems:

  • administrators did not identity the pages they were required to update and
  • responsibility for each page was smeared across the whole team, instead of being the sole responsibility of an individual

However the ad hoc structure suited Project Managers who were able to get a view of the overall status of a particular topic without drilling down into the details of particular sites. Unfortunately,the ad hoc structure contained stale data, due to the reasons above.

Proposal

I propose to restructure the organisation of the site status pages. The new structure will contain pages that each relate to one individual site. Each site specific page will contain sections for each of the topics that the site administrator is supposed to report on. In this way, the two flaws identified above will be improved:

  • administrators need to identity only one page where they place all their information.
  • responsibility for each page is given to one individual at the specific site.

Once the data is in that structure, it will be possible to easily track up-to-dateness. Furthermore, the topic structure favoured by Project Managers can be implemented using a manager report tool, written in PHP, which combines all the individual pages and reformats them into a set of management topic summaries that give the required overview.

Prototype

I have prototyped the process by writing two scripts.

  • the reform.pl script which takes the current status sheets (in topic format) and renders them in site-specific format.
  • the topicise.pl script, which creates a management report in topic order, not site order.

Both scripts are presented at the end of this report. It may be necessary to reimplemented the topicise.pl in PHP to allow it to be run on the server.

In summary, the reform.pl script breaks up the topic oriented data and turns it into site oriented data, while the topicise.pl script breaks up the site oriented data and turns it into topic oriented data.

Results

The results of this process can be seen at:

https://www.gridpp.ac.uk/wiki/Separate_Site_Status_Pages

The last item, https://www.gridpp.ac.uk/wiki/Topic_View, shows the "reintegrated" management view.

Scripts

The reform.pl script

This script breaks up the topic oriented data and turns it into site oriented data.


#!/usr/bin/perl

use strict;

my %list;
my @meat;
my @all ;

# Get all the lines, omiting weird chars
while(<>) {
  my $line = $_;
  chomp($line);
  $line =~ s/\xe2\x80\x9c/\"/g;
  $line =~ s/\xe2\x80\x9d/\"/g;
  push(@all, $line);
}

# Get rid of all those regional headings
foreach my $l (@all) {
  if ($l =~ /==\s*LondonGrid\s*==/) {next;}
  if ($l =~ /==\s*NorthGrid\s*==/) {next;}
  if ($l =~ /==\s*SouthGrid\s*==/) {next;}
  if ($l =~ /==\s*ScotGrid\s*==/) {next;}
  if ($l =~ /==\s*Tier1\s*==/) {next;}
  push(@meat,$l);
}

# Find all the site names
foreach my $l (@meat) {
  if ($l =~ /.*=\s*(UKI-[A-Za-z0-9_\-]*)\s*=.*/) {
    $list{$1} = 1;
  }
  if ($l =~ /.*=\s*(EFDA-[A-Za-z0-9_\-]*)\s*=.*/) {
    $list{$1} = 1;
  }
  if ($l =~ /.*=\s*(RAL-LCG[A-Za-z0-9_\-]*)\s*=.*/) {
    $list{$1} = 1;
  }
}
my @sites;
foreach my $k(sort(keys(%list))) {
  push(@sites,$k);
}


# Get the topic text. Put it in a hash to get at later.
my @topics;
my $currentTopic;
my %topicText;
foreach my $l (@meat) {
  if ($l =~ /<!-- TOPIC https\:\/\/.*wiki\/([0-9\%A-Za-zi_\,]+)/){

    $currentTopic = $1; $currentTopic =~ s/\%[A-F0-9][A-F0-9]/_/g; $currentTopic =~ s/_+/_/g;

    if (!(defined($topicText{$currentTopic}))) {
      $topicText{$currentTopic} = [];
    }
  }
  else {
    push(@{$topicText{$currentTopic}},$l);
  }
}

# Now get the topic headings. We'll use that for general Topic Notes, at the end.
my %topicHeading;
foreach my $t (sort(keys(%topicText))) {
  $topicHeading{$t} = [];
  my @topicText = @{$topicText{$t}};
  foreach my $tl (@topicText) {
    if (($tl =~ /== UKI/)||($tl =~ /= EFDA/)||($tl =~ /= RAL-LCG/)) {
      last;
    }
    push(@{$topicHeading{$t}},$tl);
  }
}

# We have all the sites, and all the topics and topic headings.
# Now we'll print it all out, in the right order.

# So go ever each file, and open a wiki file for it
foreach my $s ( @sites) {
  open(SITE,">$s.wiki") or die("Could not open $s.wiki, $!");

  # Print the site header
  print (SITE "= $s =\n");

  # Go over the topic for this site
  foreach my $t (sort(keys(%topicText))) {
    print (SITE "== Topic: $t ==\n");

    # Get the text for this topic
    my @text   = @{$topicText{$t}};
    my $inSiteSection = 0;
    my $lineCount = 0;

    # Now extract for the full text the peice for this site
    foreach my $ttl (@text) {
      if (($ttl =~ /==\s*UKI.*\s*==/) || ($ttl =~ /==\s*RAL-LCG.*\s*==/) || ($ttl =~ /==\s*EFDA.*\s*==/) ){
        $inSiteSection = 0;
      }
      if ($ttl =~ /==\s*$s\s*==/) {
        $inSiteSection = 1;
      }

      if ($inSiteSection) {

        if ($ttl !~ /==\s*$s\s*==/) {
          print(SITE "$ttl");
          if (($ttl !~ /^\s*\=+.*\s*\=+\s*$/) && ($ttl !~ /\|/)) {
            print(SITE "<br />");
          }
          print(SITE "\n");
          $lineCount ++ unless ( ($ttl =~ /^\s*$/) || ($ttl =~ /===\s*.*\s*===/) || ($ttl =~ /No content/i) ); 
        }
      }
    }
    if ($lineCount == 0) {
      print(SITE "\n\n\n*** This section intentionally left blank \n\n\n");
    }
  }
  close(SITE);
}


open(INST,">TopicNotes.wiki") or die("Could not open instructions.wiki, $!");
print(INST "== Site Admin Notes ==\n\n");
print(INST "This page contains the notes on each of the topics that site admins are expect to\n");
print(INST "keep up to date.\n\n");

foreach my $t (sort(keys(%topicText))) {
print(INST "\n== Topic: $t ==\n\n");
  my @topicHeader = @{$topicHeading{$t}};
  foreach my $thl (@topicHeader) {
    $thl =~ s/page/topic/;
    $thl =~ s/Page/Topic/;
    if ($thl =~ /^\s*\=+.*\s*\=+\s*$/) {
      $thl =~ s/\=//g;
    }
    print(INST "$thl");
    if ($thl !~ /\=\s*$/) {
      print(INST "<br />");
    }
    print(INST "\n");
  }
}
print(INST "\n\n");
close(INST);

The topicise.pl script

This script breaks up the site oriented data and turns it into topic oriented data.


#!/usr/bin/perl

use strict;

# Get the site wiki files
opendir(DIR,'.') or die("Failed to open dir\n");
my @siteWikis  = sort(grep { /^UKI.*wiki/ || /^RAL.*wiki/ || /^EFDA.*wiki/} readdir(DIR));
closedir(DIR);

# Get the topic labels

my $exampleSite = $siteWikis[0];
my @topicLabels = ();
my %topicTexts;
open(EX,"$exampleSite") or die("Could not read example site, $!\n");
while(<EX>) {
  if (/==\s*Topic:\s*([A-Za-z0-9\-\_]+)\s*==/) {
    my $topicLabel = $1;
    push(@topicLabels,$topicLabel);
    $topicTexts{$topicLabel} = [];
    push(@{$topicTexts{$topicLabel}},"= $topicLabel =\n");
  }
}
close(EX);

# Now put the information in topic order, by going over the site data
# and allocting each topic section to the appropriate topic silo
my $currentSiteHeading;
my $currentTopic;
foreach my $f (@siteWikis) {
  open(SITEWIKI,"$f") or die("Could not open $f,$!\n");
  while(<SITEWIKI>) {
    my $l = $_;

    if ($l =~ /^\=\s*([A-Za-z0-9\-\_]+)\s*=$/) {
      $currentSiteHeading = $l;
      chomp($currentSiteHeading);
    }
    else {
      if ($l =~ /==\s*Topic:\s*([A-Za-z0-9\-\_]+)\s*==/) {
        $currentTopic = $1;
        push(@{$topicTexts{$currentTopic}},"=" . $currentSiteHeading . "=\n");
      }
      else {
        push(@{$topicTexts{$currentTopic}},$l);
      }
    }
  }
  close(SITEWIKI);
}

# Now print the whole caboodle out in topic order
open (TOPICVIEW,">topicView.wiki") or die("Could not open topicView.wiki, $!\n");
foreach my $tl  (@topicLabels) {
  my @topicText = @{$topicTexts{$tl}};


  foreach my $line (@topicText) {
    print(TOPICVIEW $line);
  }
}
close(TOPICVIEW);


END OF DOCUMENT