Difference between revisions of "LabNet News"

Latest revision as of 18:31, 19 April 2022

Please check https://www.labnet.mun.ca/ for the latest info on Labnet issues that may affect users, such as

- scheduled maintenance

- software upgrades

- equipment upgrades

- current system-wide issues

Difference between revisions of "LabNet News"

Latest revision as of 18:31, 19 April 2022

Navigation menu

Search

@@ Line 1: / Line 1: @@
-This page lists current and upcoming MUN LabNet projects that may affect users, such as
+Please check https://www.labnet.mun.ca/ for the latest info on Labnet issues that may affect users, such as
 - scheduled maintenance
@@ Line 8: / Line 8: @@
 - current system-wide issues
-Insert new reports below this line
---------------------
-'''November 4, 2015'''
-The migration of '''riemann''' and two Math and Stats LDAP servers ('''federov''' and '''fermi''') are scheduled to be migrated from XenServer to VirtualBox at 4am on Friday, November 6, 2015.
-'''October 30, 2015'''
-A number of accounts residing on carme were moved to either arche.pcglabs.mun.ca (students) or leda.pcglabs.mun.ca (faculty, grads and staff).
-Leda with 5 Tb of disk space is the new labnet faculty, grads and staff, home directory server. It is being backed up by bolt.cs.mun.ca.
-Arche, one of the labnet student home directory servers has 3 new Tb of disk space. The new partitions are being backed up by bumper.cs.mun.ca.
-We are awaiting new drives from MUN's ITS group so we can perform secondary backups for the latest 8Tb of user disk space placed in production.
-'''October 29, 2015'''
-The Math and Stats departmental Matlab license server, '''noether''', was upgraded, had an updated license file applied, and migrated to Ubuntu Server.
-'''October 27, 2015'''
-The Statistics compute node '''smith''' is offline for testing purposes.
-'''October 15, 2015'''
-We are in the process of moving system backups to two dedicated machines; clutch.math.mun.ca (/backup1/systems  and /backup2/systems) and crank.cs.mun.ca (backup1/systems).
-For labnet home directory servers the scheme is currently as follows until enough "depth" is reached on the new backup servers, at which point the older backups will be reconfigured:
-carme
-      -> bumper
-      -> MUN TSM
-lysithea
-         -> axle
-         -> spring/hobbes
-metis
-      -> shock (/backup2)
-      -> spring/hobbes/axle
-arche
-      -> shock (backup1)
-      -> spring
-'''October 9, 2015'''
-We have set up a new home directory server, leda.pcglabs.mun.ca  for labnet faculty, staff and grad students.
-We are also moving around which backup servers are being backed up to. This means that some backed up data may not be accessible from webtools for users to access, even though it might still be on an old backup server.
-'''October 2, 2015'''
-Upgraded phobos.cs.mun.ca to new hardware. We were experiencing NFS failures due to failing NIC's
-'''September 11, 2015'''
-Some CS student webpages (i.e. URLs like www.cs.mun.ca/~username) are temporarily unavailable due to hard problems on phobos.
-Phobos is being upgraded, but we have encountered a delay in the arrival of hardware. Please contact the systems group if you need an interim solution.
-'''September 8, 2015'''
-The linuxlj printer in EN2036 has been replaced with a new printer.
-'''August 31, 2015'''
-The linuxlj printer from EN2036 has been sent for servicing.
-'''August 24, 2015'''
-Started testing of the new OTRS ticket system.
-'''August 22, 2015'''
-Mounted a new LUN for /usr/local/pub on lysithea as we no longer had enough disk space on the old /usr/local/pub LUN.
-'''August 7, 2015'''
-The backup server, '''hood''', has failed hard drives in two mdraid arrays. The server will be shut down at approx. 1:30pm today for disk replacement.
-'''July 29, 2015'''
-The Math and Stats Windows 7 appserver, '''salvage''', is rebuilt.
-'''July 28, 2015'''
-Services of the Math and Stats Windows 7 appserver, '''sanikiluaq''', was affected by the /var directory filling up in a ''relatively'' short period of time. This issue is now resolved.
-'''July 14, 2015'''
-Two new virtual SAN servers have been set up and are currently being tested before they get handed over to ITS. '''Elara''' is an appserver. '''Otrstest''' is the server running the otrs ticket system.
-'''July 13, 2015'''
-Updated the VCL virtual machine creation utility to launch from webpage.
-'''July 8, 2015'''
-'''
-A 1.5Tb RAID array was installed, replacing the old 1Tb array for /users/cs/study on '''phobos''', which was was failing.
-'''July 5, 2015'''
-The Math and Stats Windows 7 appserver, '''salvage''', is out of service. An update will be posted when this is resolved.
-'''June 25, 2015'''
-The Math and Stats Windows 7 appserver, '''isthmus''', was rebuilt with new hard drives.
-'''May 27, 2015'''
-The Math and Stats authoritative DNS and redundant LDAP server, '''fermat''', was replaced with a new 64-bit installation.
-'''May 20, 2015'''
-The Math and Stats Windows 7 appserver, '''salvage''' was rebuilt with a new 64-bit installation.
-'''May 15, 2015'''
-The Math and Stats Win7 appserver, '''isthmus''' was rebuilt with a new 64-bit installation.
-'''May 13, 2015'''
-The Math and Stats Matlab license server, '''noether''', was replaced with an upgraded virtual host with the same name, MAC address, and IP address.
-'''May 11, 2015'''
-) Account archiving has begun on '''carme'''.
-'''May 8, 2015'''
-) Account archiving has been completed on '''metis'''. 4473 accounts unused in the last two years were archived, freeing up 563G of disk space.
-'''May 1, 2015'''
-) Account archiving has been completed on ''lysithea''.
-) Account archiving has begun on ''metis'' and will continue through next week.
-'''April 30, 2015'''
-Updated R packages in jon:/mnt/dbgen64test/usr/lib64/R/library and jon:/mnt/dbgen64dev/usr/lib64/R/library: '''survival''', '''Rlab'''.
-'''April 29, 2015'''
-) The compute nodes '''herschel''', '''nancy''', '''kepler''', and '''igor''' were upgraded to the latest 64-bit gentoo image.
-'''April 28, 2015'''
-) Accounts older than two years since last use are being archived from lysithea and metis this week.
-) The compute nodes '''hamilton''' and '''wallis''' were upgraded to the latest 64-bit gentoo image.
-'''April 27, 2015'''
-The virtual compute nodes '''box''', '''chow''', '''eden''', and '''norwood''' were consolidated. The new virtual compute node is '''box'''.
-'''April 24, 2015'''
-Lysithea and metis, student home directory servers, were upgraded to the 64 bit gentoo image.
-'''April 23, 2015'''
-Upgraded arche and mneme to 64 bit gentoo OS.
-'''April 22, 2015'''
-Finished upgrading vmcarme to 64 bit gentoo
-'''April 10, 2015'''
-The backup server '''hood''' has been upgraded.
-'''April 9, 2015'''
-) A new backup server for math department backups has been set up in the CS server room. '''Luggage''', the new server, replaces the old servers '''crank''' and '''gasket''' and will complement '''trunk''' the other math department backup server housed in CS.
-) The backup server '''clutch''' has been upgraded.
-'''April 8, 2015'''
-All printers in Math and Stats have been moved to the new print server, '''pomor'''. The old print server, '''avalon''', has been shut down.
-'''April 4, 2015'''
-) The home directory server, '''phobos''', had become unresponsive and required a reboot. This server hosts the directories for /users/cs/grad, /users/cs/study, and /users/cs/misc.
-'''April 2, 2015'''
-'''Pomor''', the replacement for the print server '''avalon''' in Math and Stats, has been built, configured and is being tested. Remaining printers in Math and Stats will be moved to the new server early next week.
-'''March 18, 2015'''
-NTPD has been upgraded on our primary ntp servers so that they would not be susceptible for use in a distributed denial-of-service (DDoS) attack. A third primary ntp server (ender.cs.mun.ca) has been brought on line, which brings the total to three primary ntp servers for use by the MUN labnet community.
-'''March 13, 2015'''
-) A new version of diskless Ubuntu (14x8) has been released on CS and PCGLABS servers. Older versions have been removed from CS servers, and will disappear from PCG Labs next week. Other labs throughout campus will be updated on request from their admins.
-'''March 11, 2015'''
-) The old 32-bit servers '''thebe''', '''ls1''', '''curo''' and '''cialis''' have been taken out of service.
-'''March 10, 2015'''
-) Yesterday we encountered NFS file corruption issues on the labnet/st1 student file system. These should all be repaired now, but please report any problems you may encounter.
-) The diskless Ubuntu distribution contains an application called zeitgeist which triggers NFS problems. We are building a new distribution without the problem application.
-) Bonavista was taken offline.
-'''March 5, 2015'''
-) Riemann was updated (distribution, kernel). Required functionality (e.g., ssh server, alpine) was tested. Anybody who uses riemann is asked to provide feedback on any NFS issues encountered.
-'''March 3, 2015'''
-) Installed on jon in server dev64 and server test64 a new lnmenuserver to fix issues with the latest win7logon not authenticating students. - PP
-'''March 2, 2015'''
-) Removed dependency on old configuration data that was causing several account creation webtools (Temporary Account Creation and Create External Account via Adv. Account Mngr. as well as Machine Account Creation) to not function properly. - AC
-'''February 26, 2015'''
-) We are gradually updating the few remaining 32-bit servers to 64-bit distributions.
-) Most of the servers supporting diskless ubuntu distributions started misbehaving today. The causes are not entirely clear, but we've got them all working properly again. If you find that they don't work for you at any point, please let us know.
-) Please let students know that if they encounter systems issues, they should let a sysadmin know about it.
-) The csgljet printer has been replaced with a brand new LaserJet 3015.
-'''February 17, 2015'''
-) We will be upgrading various servers in the near future, as circumstances allow.  A security vulnerability in the C library has necessitated this. Newer and more critical systems have already been upgraded. Older systems generally have a more challenging upgrade path for pragmatic reasons as well as technical ones.
-) The server '''europa''' is being removed from service. Europa primarily acted as a web server for applications which have been obsoleted and unsupported for several years.
-) The server '''nermal''' is being removed from service. Nermal was the primary print server for C.S. and has been replaced by megatron.
-'''February 10, 2015'''
-) All cs printers that were spooling through nermal have now been moved to megatron. If you are running a diskless client and you find that printing doesn't work on a particular printer (although it should), rebooting should fix the issue. If you prefer not to reboot, contact the systems group for an alternative solution.
-) Arlene, the master configuration server for LabNet, will be down briefly at 3pm while we move it physically to its new home.
-'''February 3, 2015'''
-) Fixed the problem with gatekeeper so now remote scripts should run.  This includes the rename script that failed for bradleyd.
-) Fixed the problems with the missing executables for exemptuser and printrefund webtools.
-) Added code to the win7login.py script to cause inactive or nonexistent accounts to become activated automatically.
-) Fixed erroneous queries within webtools that were causing issues with WIMPng loading and removing entities in System Configuration.
-'''February 2, 2015'''
-) Arlene's O/S and packages were upgraded over the weekend. There is currently a compatibility issue between the python libraries and WebTools authentication.  This prevents people from logging into WebTools, and so is a priority for this morning.
-)  New PAMpython.so was deployed to fix issues with WebTools authentication in the afternoon.
-)  Final reconfiguration of Arlene was completed.
-)  The menuserver daemon was modified to detect inactive accounts so that account reactivation can occur.
-'''January 31, February 1, 2015'''
-)  Upgraded arlene, the master configuration server, as a part of the ongoing upgrade process mandated by the vulnerablility in glibc.  ALthough not absolutely required arlene was upgraded to the 64 bit test image as it was very much in need of an upgrade.
-'''January 30, 2015'''
-)  Connected student housing servers to new cabling and tested client computer.  Then Arranged with C&C to move client in West Towers to the proper Labnet VLAN and then tested client.
-'''January 22, 2015'''
-) The colour printer is now back in the CS general office and working well.
-) Added alias entry for g++-4.9.2 as g++11 for use in a cs lab course
-'''January 20, 2015'''
-)  Built server disk on loader for a new server being set up in the Commons for creating new images.
-)  Continued on with the project to enable Linux on all engineering computers.  The compressed images had all synced over last night so  the actual ssd images were made on each of the engineering application servers except for lobo (not enough space).  The computers in EN3000 were then configured to run Linux and booted into Gentoo in order to make the ssh keys.  The computers were then tested to see that they worked in Ubuntu, Gentoo and Windows7.
-'''January 19, 2015'''
-)  Fixed problem in CS2718 Lab when one of the application servers lost the virtual disk containing the Ubuntu14x5 distribution.  The problem was initially remedied by pointing the clients to a different application server.  Later the virtual disk was restored.
-)  Started enabling Linux supprt for Engineering labs.  Created ssh keys and uploaded keys into our master database for computers in EN1038B.  Set up the rsyncing of the '''/images/linux''' images directory to all the Engineering application servers.
-'''January 18,2015'''
-)  Set up two new computers in EN1049 to provide additional seating for Dr. Byrne's CS2718 course.  This endevour was fraught with many delays due to a lack of basic components.  An attempt should be made to have a few spare commonly used components readily available.
-'''January 17, 2015'''
-)  After collecting changes to the Ubuntu14x5 image over the past week, a snapshot of this image was made and pushed up to pooky to be distributed to the various labnet application servers via the nightly software distribution cron job.
-)  The image was manually rsyned over to the departmental application servers, odie ane meatron and tests were made to ensure that the images worked in EN2036 and EN1049.
-)  It was noted that the computer, "chase" in EN2036 was inoperable due to a locked COW partition due to the COW partition having fill up.  Normal attempts to get rid of this COW partition were unsuccessful but a web page with the following URL:
-https://www.globallinuxsecurity.pro/recovering-an-overflowed-lvm-volume-configured-with-virtualsize/
-proved to be useful in remedying the problem.  Note this approach can be used whenever the '''lvs''' state of a volume look like '''-wi-Io----''' with a capital I as the fifth flag.  The capital i indicates that the logical volume is invalid.
-'''January 16, 2015'''
-) Issue with stewart.lnesd.mun.ca contacting the Labnet configuration data base on arlene was resolved.
-'''January 15, 2015'''
-) The C.S. genofflj printer has been temporarily replaced with an older printer.  A new printer will replace it in early February.
-) The colour printer in the C.S. general office is out for servicing.
-) Diskless on megatron.cs.mun.ca was upgraded to the test image so it would be in line with what is currently running in the other CS labs and in the Commons.
-'''January 13, 2015'''
-)  CS faculty please note that speakers have been added to the A/V podium in EN1049.  (This was done in Dec., but not widely announced.)
-)  New wireless access points have been installed on the first floor of the Engineering building. This should greatly improve
-service in the CS grad student lounge as well as EN1051 and EN1052.
-)  Network issues on login by the  CS secretarial staff this morning were found to due to  problems connecting with lysithea.pcgabs.mun.ca 's  /usr/local/pub share.
-The reliance on this share, which has been an issue on past occasions,  has now been  disabled for all office staff machines.
-The source of the problem was found to be the primary ldap server (scout.pcglabs.mun.ca)  for lysithea which had become unresponsive, The ldap service on scout.pcglabs.mun.ca was restarted. This resolved login issues being experienced across campus in addition to login and printing issues in the Commons.
-)  At the request of IA's and Dr. Byrne, a new writable version of our iSCSI Ubuntu has been set up so that new software can be added.  The writable image is ssh'able on csgrad01.  The andrew, aaron, marian and pprice have been added to
-to the sudoers config file and can do installs.  The computer will be available for updates until Friday at which point
-a snapshot of the new image will be taken.  In the mean time feal free to add any relevent software that comes to mind.
-'''January 12, 2015'''
-)  Fixed the problems in the automatic account generation code that have been plaguing the Commons.
-) Found a fix for the problem with Ubuntu14 login issues that failed due to "Failed session" error.
-NOTE:  If people are having this problem logging in to Ubuntu then have the students use another session manager other than the default "Ubuntu" (unity) session manager.  The other two options are "Compiz" and "Metacity".  Basically these two look the same but "Compiz" is better able to take advantage of graphics card accellerators on supportted models (ie non Nvidia).  The button, a circular Ubuntu logo, that allows the user to switch session managers is located just above and to the right of the text box used to enter the user name.
-'''January 10,11, 2015'''
-)  Set up '''sudoers''' managed file for Ubuntu computers on Campus.  This was done to allow members of CS2718 to
-be able to run certain commands requiring super user privilege in a safe and secure way.  This will be used as a test case
-that can be used for future instances where elevated privleges are required.
-)  Created a series of test cases for final the testing of the pam_labnet module and ran the code through the debugger to
-validate the logic of each test case.  Then ran the code through '''valgrind'' code checker to check for use of uninitialized
-valiables and memory leaks etc.  Installed the software on the development image for 32/64 bit severs and for development,
-diskless client computers.  On Sunday after checking that authentication is still working on development images, the test and production environments were built and installed.  Notes: there may need to be some work done on some memory leaks on client side (see log[1-6].vlg files).
-)  Switched the commons over to the new Ubuntu14x4 image and tested a number of clients.
-)  Added Ubuntu14x4 image to the IA's office PC's.
-'''January 9, 2015'''
-) Brought salsa.pcglabs.mun.ca the appserver for the Social work lab back up to date. It had been turned off since
-last August (2014) due to building renovations.
-Its new home is in CL3000. It was synced/upgraded to run the new server and diskless test images. The machine
-was booted onto the latest kernel that we have tested without any problems.  It also has been reconfigured
-to use grub2 a newer version of the grand unified boot loader. The grub2 boot loader is now in use on all CS department
-machines and is being rolled out to other Labnet servers as they get upgraded.
-)  Setup and tested client support in CS1019 for the new Ubuntu14x4 image.  Discussed with Steve Johnson about the
-possibility of using Ubuntu as an alternative to Gentoo in the labs.
-) Brought hope.engr.mun.ca back into sync in conjunction with the engineering department.
-) Found and adjusted a parameter on our master ldap server that prevented new slave ldap installations from completing to build.
-'''January 8, 2015'''
-) Tested the booting of the new Ubuntu iSCSI image in computer Labs EN1049 and EN-2036.  Also added
-Ubuntu to the list of images that 'helppc' could boot and tested it to make sure that it works.
-Also distributed the new Ubuntu image to the application servers that boot the CS-1019 computer
-lab in preparation of rolling out Ubuntu in this computer Lab.  Distributed the new image to the Commons as well.
-)  Noticed that there were intermittent problems with booting the iSCSI images in EN-2036.  When the operating
-system is selected from the boot menu the initrd loads but fails to open the iSCSI disk.  It seems that the
-customization process runs without the iSCSI partition being properly mounted leaving configuration files in
-the mount point which in turn prevents the system from removing the mount point and thereby causing the
-image creation to abort. The solution right now is to manually remove the mount point (rm /tmp/<computer name>).  If it
-reoccurs, a more permanent solution will be applied.
-)  Talked to classroom support about the possibility of running Labnet on all the multimedia classrooms
-in the engineering building.
-) Started work on getting printing support working from the new Ubuntu image.  This will probably involve
-creating a template file that will display a list of printers available to each client.  The printer names
-associated with each client  will be stored in the sys_config database.
-'''January 7, 2015'''
-) Rolled out and announced initial version of this information page.
-) We experienced samba problems with access to home directories on lysithea.pcglabs.mun.ca. Decided to restart the samba service after consulting with the commons, which seems to have resolved the problem.
-) The EN1066 Mac lab has been re imaged as it was used for Mac OS X training during December. It is now running the latest version of OS X; Yosemite. It has been set up to use ldap authentication with our Labnet ldap servers so that users do not need to remember and use additional passwords beyond their MUN login one. The latest version of Xcode has been installed at the request of the mobile application course instructors. All students enrolled in the course have been added to the _developer group on each system in order to allow them to run Xcode.
-) Booted all the computers in CP2003 into Ubuntu in preparation for Dr. Byrne's course.
-) Updated the Ubuntu image and added several new packages needed by Dr. Byrne's course as well as some packages recommended by students and staff. Pushed the new image up to the master server for distribution this evening's rsync.
-) Set up the the classroom computers in EN1051, EN1052 and EN1054 so that they could
-avail of the new Ubuntu iSCSI boot image.  To use simple reboot the computer and at the
-boot menu select the "Ubuntu Linux" menu item.  When the login screen comes up login
-using you Labnet credentials.
-) Modified the configuration files for apache to include include files that will allow students
-with Dr. Byrnes course to be able to access course material from Dr. Byrne's GIT repository.
-'''January 6, 2015'''
-) Set up the the classroom computers in EN1051, EN1052 and EN1054 so that they could avail of the new Ubuntu iSCSI boot image. To use, simply reboot the computer and at the boot menu select the "Ubuntu Linux" menu item. When the login screen comes up, login using your Labnet credentials.
-) Modified the configuration files for the apache server on stretch. The modifications direct apache to include files that will allow Dr. Byrne's students to be able to access course material from his GIT repository.
-) Sent off for a request for comments proposal to Hewlett Packard on a new printer standards initiative that will allow Labnet printing to be more functional with respect to printer job control and better cost recovery metrics.
-) CS genofflj printer has been temporarily removed for servicing.
-'''Christmas Projects List 2014'''
-During the Christmas break a number Labnet of projects were undertaken.
-Many of these projects could not be performed during the semester due to
-their intrusive nature.  The following are brief descriptions of the projects
-arranged in more or less chronological order:
-)
-Configured the computers in the  Computer Science labs and the CP2003
-labs to boot the new iSCSI versions of the popular Ubuntu 14.0 distribution.
-This involved updating the image and pushing the image up to our master
-server and allowing it to update the various application servers.  The virtual
-images were then created on the SSD disks of the application servers and the
-database configurations were updated appropriately.  This will be used to
-support one of Dr. Byrne's courses.
-)
-Installed the server images for "isthmus" for the Math department.  This
-was necessary because the libparted.so was upgraded and the buildserver
-application no longer builds the partition tables properly.  It was necessary to manually
-build the partitions and then proceed with the install.  This project is stalled
-due to problems loading the boot sector.
-)
-The size of our newer disk drives now exceeds the capabilities of the old "msdos"
-partition tables.  The "GUID Partition Tables" or GPT will now be used in future server
-builds to support the larger disks.  To boot from GPT partitioned disks the use of grub2
-is under development.  In addition UEFI boot support is under investigation.  As servers
-are upgraded our server builds will be utilizing GPT and will be migrating to grub2 for
-booting.  When better support for UEFI booting becomes more mature, support
-for UEFI will be incorporated.  Currently there are a couple of servers that have been
-moved over to GPT partitions and the CS department is now booting with grub2.
-)
-One of our core Labnet daemons, "master_configd", responsible for managing the
-remote distribution of configuration services such as master file templates,
-certificates, printer account numbers etc. was becoming severely
-impacted by the shear number of requests that it received.  To ameliorate
-this problem one of the modules was rewritten to perform work asynchronous
-and thereby reduce the strain on the daemon.  The resulting code runs an
-order of magnitude faster and service requests are no longer creating backlogs.
-)
-Another service that was becoming bogged down was the user/system logging
-database that provides statistics on when and where students log into Labnet and
-what services they are requesting as well as the reboot logging of computers.  An
-analysis of the data queries was performed to isolate the culprits.  As
-it turned out the indices were not properly set up for several common queries and
-this turned out to be the culprit.  The indexing was altered and now the queries
-work 2 orders of magnitude faster.
-)
-Performed network performance analysis on the Commons client/server
-communications.  This was requested as a result of changes to the network
-infrastructure.  Recommendations have been forwarded to the networking
-group within C&C as well as the Commons.  It is hoped that this will be used
-to redesign the network layout to remove some of the bottlenecks.
-)
-The popular Ubuntu Linux 14.0 distribution was enabled for use in the Commons.  This
-upgrade is similar to the upgrade to CP2003 and the Computer Science Department
-computer labs.  This technology allows a client computer to mount a virtual disk over
-IP networks using the iSCSI protocol.  The virtual disk is located on an application server
-that can remotely support an entire lab.
-)
-One of the final components of the "Account Archival Project" is the PAM Labnet module that
-reenables an account when a student returns to university.  After the user's data has been collected and
-written to DVD's the home directory is removed and the LDAP entry is put in hibernation.  When
-an archived Labnet user attempts to login, the login credential is restored and a new home directory
-is created.
-Wrote and debugged the PAM code that will be needed to reactivate user
-accounts after they have been archived.
-)
-To prepare for an up coming security, audit additions were made to the Labnet PAM module to allow for
-the seamless update of our remaining DES password hashes to the SHA1 hashes used by C&C.  This will
-provide greater protection of our password hashes in the event that our password hashes are exposed.
-The newer SHA1 hashes are much more resiliant to brute force cracking than their DES counterparts..