Difference between revisions of "LabNet News"

From CSWiki
Jump to navigation Jump to search
(liz update)
 
(21 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
This page lists current and upcoming MUN LabNet projects that may affect users, such as
+
Please check https://www.labnet.mun.ca/ for the latest info on Labnet issues that may affect users, such as
  
 
- scheduled maintenance
 
- scheduled maintenance
Line 8: Line 8:
  
 
- current system-wide issues
 
- current system-wide issues
−
 
−
Insert new reports below this line
 
−
--------------------
 
−
'''September 26, 2017'''
 
−
 
−
* The backup and sync server '''liz''' is now online. There was an issue migrating it to new hardware and it is currently using its previous hardware configuration.
 
−
 
−
'''September 22, 2017'''
 
−
 
−
* The compute node '''igor''' was migrated to the rootfs_dbgen64_test distribution. Relevant users were informed. The kernel was upgraded to kernel-4.9.5-gentoo-64-20170731.
 
−
 
−
* The compute node '''kepler''' was migrated to the rootfs_dbgen64_test distribution. Relevant users were informed. The kernel was upgraded to kernel-4.11.8-gentoo-64-20170630.
 
−
 
−
* The server '''vortex''' (Math & Stats but located in Computer Science) was migrated to the rootfs_dbgen64_test distribution. The server owner was previously informed. The kernel was upgraded to kernel-4.11.8-gentoo-64-20170630.
 
−
 
−
* Maple 18 was previously added to the ubuntu16x14 image on isthmus. The front half of the Math & Stats lab (HH3030) was configured to boot from isthmus on September 21. The back half of the Math & Stats lab (HH3056) is also now configured to boot from isthmus. (If you want to know why the file was added to the ubuntu16x14 image directly on one appserver instead of a new image, please contact Lawrence Greening in Math & Stats directly.)
 
−
 
−
'''September 21, 2017'''
 
−
 
−
* The backup and sync server '''liz''' was migrated to the rootfs_dbgen64_test distribution. The kernel was upgraded to kernel-4.9.5-gentoo-64-20170731.
 
−
 
−
* Maple 18 was added to the ubuntu16x14 image on isthmus. The front half of the Math & Stats lab (HH3030) was configured to boot from isthmus. The remainder of the lab PCs, classroom PCs and remaining office PCs will be upgraded later. (If anybody wants to know why the file was added to the ubuntu16x14 image directly on one appserver instead of the image being cloned and the cloned image updated, please contact Lawrence Greening in Math & Stats directly.)
 
−
 
−
'''September 21, 2017'''
 
−
 
−
* A number of server images did not propagate from '''liz''' overnight. This is being investigated.
 
−
 
−
'''September 20, 2017'''
 
−
 
−
* R 3.4.1 has been added to the usr_local_math_test distribution. It should propagate to compute nodes overnight.
 
−
 
−
'''September 19, 2017'''
 
−
 
−
* (September 18, 2017) The compute nodes '''hamilton''', '''herschel''', and '''nancy''' were migrated to the rootfs_dbgen64_test distribution. Relevant users were informed. The kernel was upgraded to kernel-4.11.8-gentoo-64-20170630.
 
−
* (September 18, 2017) Broken links were fixed and new symbolic links added to the dbgen32_diskless_test distribution for diskless clients booting from '''burin''' only to resolve the issue of applications dynamically linked to liblapack not being able to find the library files. This change will be added to the LabNet-wide dbgen32_diskless_test distribution.
 
−
* The compute node '''kepler''' is scheduled to be migrated to the rootfs_dbgen64_test distribution and its kernel upgraded to 4.11.8-gentoo-64-20170630 on Friday. Relevant users have been notified.
 
−
 
−
'''September 14, 2017'''
 
−
 
−
* The Math and Stats compute node '''bristol''' was migrated to the rootfs_dbgen64_test distribution. Relevant users were previously informed. The kernel was upgraded to kernel-4.9.5-gentoo-64-20170731.
 
−
 
−
* The m2130.sty file and other Math & Stats departmental LaTeX style files were added to the ubuntu16x14 image on sanikiluaq. The TeXLive package database (ls-R file) was updated to reflect this change. The Math & Stats lab, classroom PCs, and a few office PCs were configured to boot from sanikiluaq. (If anybody wants to know why the file was added to the ubuntu16x14 image directly on one appserver instead of the image being cloned and the cloned image updated, please contact Lawrence Greening in Math & Stats directly.)
 
−
 
−
'''September 13, 2017'''
 
−
 
−
* (September 12, 2017) The research backup server '''towmater''' was migrated to the rootfs_dbgen64_test distribution. The kernel was upgraded to kernel-4.9.5-gentoo-64-20170731.
 
−
 
−
'''September 12, 2017'''
 
−
 
−
* The Math and Stats appserver '''burin''' was migrated to the rootfs_dbgen64_test distribution today and the cluster rebooted. Relevant users were previously informed. The kernel was upgraded to kernel-4.9.5-gentoo-64-20170731.
 
−
 
−
'''August 31, 2017'''
 
−
 
−
* The Math and Stats servers '''fermat''' (LDAP, authoritative DNS), '''pomor''' (print), and '''kummer''' (samba), were migrated to the rootfs_dbgen64_test distribution. The kernel was upgraded to kernel-4.11.8-gentoo-64-20170630. The department was notified internally by email on August 11 and August 29 of upcoming server upgrades and on August 31 before specific upgrades.
 
−
 
−
'''August 28, 2017'''
 
−
 
−
* (August 24, 2017) The Math and Stats file servers '''pitassi''', '''wigner''', '''pascal''', '''moody''', '''peterson''' were migrated to the rootfs_dbgen64_test distribution. The host '''riemann''' was also migrated to the rootfs_dbgen64_test distribution. The kernel was upgraded to kernel-4.11.8-gentoo-64-20170630. The department was notified internally by email on August 11 of upcoming server upgrades and on August 24 before these specific upgrades.
 
−
 
−
'''June 15, 2017'''
 
−
 
−
* Due to a number of issues, mostly related to libraries, OpenGL and drivers, the Maple installations and configurations will all be reverted back, incl. symbolic links in /usr/local/bin. Changes should propagate across LabNet overnight. The update of Maple will be rescheduled after these issues are investigated. There is no estimated time frame as other issues/projects are higher priority at the moment.
 
−
 
−
'''June 14, 2017'''
 
−
 
−
* The rootfs_dbgen64_dev, rootfs_dbgen64_test, and rootfs_dbgen64_prod images have been updated so their /usr/local/bin/maple and xmaple symbolic links point to the new location. Changes should propagate across LabNet overnight.
 
−
 
−
* Maple 2017 has been added to the usr_local_math_dev, usr_local_math_test, and usr_local_math_prod images. It has been tested with the usr_local_math_test image (which is in production). Maple 16 and Maple 17 were removed to save space. Changes should propagate across LabNet overnight.
 
−
 
−
* (Update) The updated Maple license server has now been installed.
 
−
 
−
* Syncing of the md raid5 array to the replacement hard drive in '''trunk''' stalled. After extended observation, on June 13, the hard drive was removed from the array. All md and MBR metadata was removed and the drive was wiped. After wiping, a SMART extended offline test was performed and it completed without error. Further investigation is ongoing.
 
−
 
−
'''June 2, 2017'''
 
−
 
−
* Defective hard drive was replaced in '''trunk'''.
 
−
 
−
'''May 15, 2017'''
 
−
 
−
* R package '''numDeriv''' (for R 3.3.2) has been added to /usr/local/math/R-3.3.2 on the '''dbgen64test''' distribution, as well as to a number of Math & Stats compute nodes (upon request). It should propagate to the various sync servers and end nodes overnight.
 
−
 
−
'''February 20, 2017'''
 
−
 
−
* MATLAB R2016b has been added to /usr/local/math on the '''dbgen64test''' and '''dbgen64dev''' distributions. It should propagate to the departmental sync servers and end nodes overnight.
 
−
 
−
* The symbolic link in /usr/local/bin has not been switched over yet, pending notification of the various server owners.
 
−
 
−
'''February 20, 2017'''
 
−
 
−
All components of one of the mdraid arrays on the backup server '''trunk''' failed on Friday, February 17. The server was restarted and the array came back online with all components. The situation will be monitored.
 
−
 
−
'''October 25, 2017'''
 
−
 
−
(October 24, 2017) The Math and Stats print server '''pomor''' was synced to the latest dbgen64prod distribution and configured to remain in sync. Printing was verified working in LabNet Linux (Gentoo Linux, mathubuntu14x20x1) and Windows 7 Pro in the Math and Stats lab.
 
−
 
−
'''July 26, 2016'''
 
−
 
−
(July 20, 2016) A defective hard drive was replaced in '''trunk'''.
 
−
 
−
'''June 29, 2016'''
 
−
 
−
* (June 27, 2016) There was an air conditioning failure in the Math and Stats server room. Compute nodes were powered off and mitigation steps were taken to vent heat from the room. Facilities Management brought in a contractor who identified a tripped circuit breaker on the unit on the roof as the reason for the failure. The root cause of the tripped breaker was not identified. The situation will be monitored.
 
−
 
−
* (June 28, 2016) Because most of the compute nodes were not in use, nobody was accessing '''peterson''', the Math and Stats research file server. The server was restarted and is now using a later kernel (kernel-4.4.0-gentoo-r1-64-20160125).
 
−
 
−
* (June 28, 2016) Compute nodes were powered back on.
 
−
 
−
'''June 3, 2016'''
 
−
 
−
(POSTPONED) <strike>(Scheduled: June 6, 2016) The Math and Stats LabNet print server '''pomor''' is scheduled to be upgraded at approx. 4:00pm.</strike>
 
−
 
−
'''May 10, 2016'''
 
−
 
−
In coordination with the server's owner (and by proxy, current users), the Math and Stats research file server '''gauss''' has been updated to the dbgen64test gentoo distribution. Its kernel is set to kernel-4.4.0-gentoo-r1-64-20160125.
 
−
 
−
'''May 5, 2016'''
 
−
 
−
* Math and Stats hosts were reconfigured to address a /dev/pts issue that affected MATLAB and OpenMPI.
 
−
 
−
* MATLAB 2016a was installed in /usr/local/math/matlab in the dbgen64 distributions on May 4. It should have propagated overnight. Older versions were '''not''' removed. Symbolic links from /usr/local/bin were '''not''' updated. These remain unchanged at the request of faculty who are in the middle of projects. Until these are updated, MATLAB executables can be run by specifying the full path or by changing your path. The executables are located in the '''/usr/local/math/matlab/R2016a/bin''' directory.
 
−
 
−
'''May 4, 2016'''
 
−
 
−
* The Math and Stats appserver '''burin''' had its '''appserver''' function replaced with '''w7appserver'''.
 
−
 
−
* The Math and Stats compute node '''igor''' has been upgraded to the dbgen64test gentoo distribution on May 2, 2016. This was coordinated with the owner of this host.
 
−
 
−
'''April 28, 2016'''
 
−
 
−
The Math and Stats file server '''moody''' has been upgraded to the dbgen64test gentoo distribution. Its kernel is set to kernel-4.4.0-gentoo-r1-64-20160125. The Math and Stats Department has been updated by email.
 
−
 
−
'''April 27, 2016'''
 
−
 
−
The Math and Stats file server '''pitassi''' has been upgraded to the dbgen64test gentoo distribution. Its kernel is set to kernel-4.4.0-gentoo-r1-64-20160125. The Math and Stats Department has been updated by email.
 
−
 
−
'''April 26, 2016'''
 
−
 
−
* The Math and Stats staff file server '''pascal''' has been upgraded to the dbgen64test gentoo distribution. Its kernel is set to kernel-4.4.0-gentoo-r1-64-20160125. The Math and Stats Department has been updated by email.
 
−
* The Math and Stats Matlab license manager was upgraded on '''noether''' and a new license file applied to accommodate installing Matlab 2016a.
 
−
 
−
'''April 26, 2016''' (Notice of scheduled updates)
 
−
 
−
* (Scheduled: April 26, 2016) The Math and Stats staff file server '''pascal''' is scheduled to be upgraded at 6:00pm. An email notification was previously sent to staff who have their home directories on this server.
 
−
* (Scheduled: April 27, 2016) The Math and Stats post-doc/visitor file server '''pitassi''' is scheduled to be upgraded at 8:00pm. An email notification was previously sent to the Math and Stats Department.
 
−
* (Scheduled: April 28, 2016) The Math and Stats faculty and graduate student file server '''moody''' is scheduled to be upgraded at 8:00pm. An email notification was previously sent to the Math and Stats Department.
 
−
 
−
'''April 20, 2016'''
 
−
 
−
* Due to the postponement of final exams, the upgrade to the Math and Stats MATLAB license server '''noether''' has been postponed. It is now scheduled for 7:00pm, Tuesday, April 26, 2016. The Math and Stats Department has been updated by email.
 
−
 
−
* The Math and Stats research file server '''peterson''' was upgraded on April 19, 2016. Hardware upgrades include new motherboard, processor, RAM, hard drives, and the installation of a second nic dedicated to iSCSI. Software upgrades include the upgrade to the dbgen64test gentoo distribution. The kernel was updated to the same kernel build used on '''wigner''' (kernel-4.2.3-gentoo-64-20151009) which itself was updated to the same kernel build used on '''terra'''. The Math and Stats Department has been updated by email.
 
−
 
−
'''April 15, 2016'''
 
−
 
−
* The Math and Stats research backup server '''towmater''' has been upgraded. Hardware upgrades include new motherboard, processor, RAM, hard drives, and the installation of a second nic dedicated to iSCSI. Software upgrades include the upgrade to the dbgen64test gentoo distribution. Its kernel is set to kernel-4.4.0-gentoo-r1-64-20160125. The Math and Stats Department has been updated by email.
 
−
 
−
* The upgrade to the Math and Stats research file server '''peterson''' has been postponed. It is now rescheduled for 10:00am, Tuesday, April 19, 2016. The Math and Stats Department has been updated by email.
 
−
 
−
'''April 14, 2016'''
 
−
 
−
The Math and Stats compute node '''hamilton''' has been upgraded to the dbgen64test gentoo distribution. The department has been notified by email.
 
−
 
−
'''April 13, 2016'''
 
−
 
−
* (Upcoming: April 14, 2016) The Math and Stats compute node '''hamilton''' is scheduled to be upgraded to the dbgen64test gentoo distribution on Thursday, April 14, 2016. The department has been notified by email.
 
−
* (Upcoming: April 15, 2016) The Math and Stats research file server '''peterson''' and research backup server '''towmater''' are scheduled to be upgraded on Friday, April 15, 2016. Upgrades will consist of both hardware and software. The department has been notified by email.
 
−
* (Upcoming: April 21, 2016) The Math and Stats MATLAB license server '''noether''' is scheduled to be upgraded at approx. 7:00pm on Thursday, April 21 (the day after the end of exams). This upgrade is necessary in order to license installations of MATLAB 2016a against the departmental license server. During this time, Matlab will be unavailable in the Math and Stats lab, Clarke Place, and all offices in the Math and Stats department. The department was notified by email on Monday, April 11.
 
−
* (April 12, 2016) The Math and Stats compute nodes '''kepler''' and '''herschel''' were upgraded yesterday to the dbgen64test gentoo distribution. '''Kepler''' was upgraded from 24GB RAM to 128GB RAM. '''Herschel''' was upgraded from 24GB RAM to 48GB RAM. The department was notified beforehand by email on Monday, April 11, and before the upgrades were started.
 
−
 
−
'''March 29, 2016'''
 
−
 
−
The Math and Stats server '''riemann''' has been upgraded to the dbgen64test gentoo distribution. The Math and Stats department has been notified. The kernel was updated to the same build used on '''garfield''' (kernel-4.4.0-gentoo-r1-64-20160125).
 
−
 
−
'''March 28, 2016'''
 
−
 
−
The Math and Stats sync+backup server '''liz''' is now online. Drives have been replaced. The server was rebuilt to the dbgen64test gentoo distribution and is using a recent kernel. The dbgen64 server (dev, test, prod) distributions, dbgen32 diskless (dev, test, prod) distributions, Math and Stats diskless images (Ubuntu and Windows 7), and LabNet Ubuntu images are synced over and are in sync. An extra network interface was installed.
 
−
 
−
'''March 28, 2016'''
 
−
 
−
The Math and Stats home directory server '''wigner''' has been upgraded to the dbgen64test gentoo distribution. The applicable users have been notified. The kernel was updated to the same build used on '''terra''' (kernel-4.2.3-gentoo-64-20151009).
 
−
 
−
'''March 28, 2016'''
 
−
 
−
The Math and Stats server '''riemann''' is scheduled to be upgraded at 12:30pm on Tuesday, March 29, 2016. An email was sent to the department.
 
−
 
−
'''March 28, 2016'''
 
−
 
−
The Math and Stats home directory server '''wigner''' is scheduled to be upgraded at approx. 2:00pm today. The applicable users have been notified.
 
−
 
−
'''March 22, 2016'''
 
−
 
−
The Math and Stats sync+backup server '''liz''' is offline. Drives have been replaced. The server is to be rebuilt and synced.
 
−
 
−
'''March 19, 2016'''
 
−
 
−
The Math and Stats appserver '''sanikiluaq''' had a hard drive replaced and was upgraded to the current 64-bit gentoo test distribution.
 
−
 
−
'''February 29, 2016'''
 
−
 
−
Upgrade to dev and test gentoo server images to address an openssl vulnerability.
 
−
 
−
'''February 19, 2016'''
 
−
 
−
Ongoing upgrade of server images to address a glibc vulnerability.
 
−
 
−
'''February 18, 2016'''
 
−
 
−
Added a line to the ssh_config files as a workaround for a openssh vulnerability.
 
−
 
−
'''February 17, 2016'''
 
−
 
−
On Monday, Feb. 15, 2016, NFS-served home directory filesystems hosted on '''wigner''' were not accessible on diskless gentoo (dbgen32test) systems. Affected users were informed at the time. Although it had been working since January 21, 2016, it stopped working sometime between Friday, Feb. 12 and Monday, Feb. 15 due to a reconfiguration of unknown origin. Mount points of wigner's home directory filesystems were (re)created in the dbgen32 distributions and the issue was resolved.
 
−
 
−
'''January 11, 2016'''
 
−
 
−
A new Math and Stats home directory server, '''wigner''', was built. This home directory server will be used to help troubleshoot the ongoing NFS issues encountered in production.
 
−
 
−
'''November 19, 2015'''
 
−
 
−
R packages optimx (with its dependencies) and jpeg were added to the '''dbgen64prod''' distribution on '''jon''' and '''hood''' for distribution last night.
 
−
 
−
'''November 6, 2015'''
 
−
 
−
The migration of '''riemann''' and two Math and Stats LDAP servers ('''federov''' and '''fermi''') from XenServer to VirtualBox has been postponed until further notice.
 
−
 
−
'''November 4, 2015'''
 
−
 
−
The migration of '''riemann''' and two Math and Stats LDAP servers ('''federov''' and '''fermi''') from XenServer to VirtualBox is scheduled for 4am, Friday, November 6, 2015.
 
−
 
−
'''October 30, 2015'''
 
−
 
−
A number of accounts residing on carme were moved to either arche.pcglabs.mun.ca (students) or leda.pcglabs.mun.ca (faculty, grads and staff).
 
−
 
−
Leda with 5 Tb of disk space is the new labnet faculty, grads and staff, home directory server. It is being backed up by bolt.cs.mun.ca.
 
−
 
−
Arche, one of the labnet student home directory servers has 3 new Tb of disk space. The new partitions are being backed up by bumper.cs.mun.ca.
 
−
 
−
We are awaiting new drives from MUN's ITS group so we can perform secondary backups for the latest 8Tb of user disk space placed in production.
 
−
 
−
'''October 29, 2015'''
 
−
 
−
The Math and Stats departmental Matlab license server, '''noether''', was upgraded, had an updated license file applied, and migrated to Ubuntu Server.
 
−
 
−
'''October 27, 2015'''
 
−
 
−
The Statistics compute node '''smith''' is offline for testing purposes.
 
−
 
−
'''October 15, 2015'''
 
−
 
−
We are in the process of moving system backups to two dedicated machines; clutch.math.mun.ca (/backup1/systems  and /backup2/systems) and crank.cs.mun.ca (backup1/systems).
 
−
 
−
For labnet home directory servers the scheme is currently as follows until enough "depth" is reached on the new backup servers, at which point the older backups will be reconfigured:
 
−
 
−
carme
 
−
     
 
−
      -> bumper
 
−
      -> MUN TSM
 
−
 
−
lysithea
 
−
        -> axle
 
−
        -> spring/hobbes
 
−
 
−
metis
 
−
      -> shock (/backup2)
 
−
      -> spring/hobbes/axle
 
−
 
−
arche
 
−
      -> shock (backup1)
 
−
      -> spring
 
−
 
−
 
−
 
−
'''October 9, 2015'''
 
−
 
−
We have set up a new home directory server, leda.pcglabs.mun.ca  for labnet faculty, staff and grad students.
 
−
We are also moving around which backup servers are being backed up to. This means that some backed up data may not be accessible from webtools for users to access, even though it might still be on an old backup server.
 
−
 
−
'''October 2, 2015'''
 
−
 
−
Upgraded phobos.cs.mun.ca to new hardware. We were experiencing NFS failures due to failing NIC's
 
−
 
−
'''September 11, 2015'''
 
−
 
−
Some CS student webpages (i.e. URLs like www.cs.mun.ca/~username) are temporarily unavailable due to hard problems on phobos.
 
−
 
−
Phobos is being upgraded, but we have encountered a delay in the arrival of hardware. Please contact the systems group if you need an interim solution.
 
−
 
−
'''September 8, 2015'''
 
−
 
−
The linuxlj printer in EN2036 has been replaced with a new printer.
 
−
 
−
'''August 31, 2015'''
 
−
 
−
The linuxlj printer from EN2036 has been sent for servicing.
 
−
 
−
'''August 24, 2015'''
 
−
 
−
Started testing of the new OTRS ticket system.
 
−
 
−
'''August 22, 2015'''
 
−
 
−
Mounted a new LUN for /usr/local/pub on lysithea as we no longer had enough disk space on the old /usr/local/pub LUN.
 
−
 
−
'''August 7, 2015'''
 
−
 
−
The backup server, '''hood''', has failed hard drives in two mdraid arrays. The server will be shut down at approx. 1:30pm today for disk replacement.
 
−
 
−
'''July 29, 2015'''
 
−
 
−
The Math and Stats Windows 7 appserver, '''salvage''', is rebuilt.
 
−
 
−
'''July 28, 2015'''
 
−
 
−
Services of the Math and Stats Windows 7 appserver, '''sanikiluaq''', was affected by the /var directory filling up in a ''relatively'' short period of time. This issue is now resolved.
 
−
 
−
'''July 14, 2015'''
 
−
 
−
Two new virtual SAN servers have been set up and are currently being tested before they get handed over to ITS. '''Elara''' is an appserver. '''Otrstest''' is the server running the otrs ticket system.
 
−
 
−
'''July 13, 2015'''
 
−
 
−
Updated the VCL virtual machine creation utility to launch from webpage.
 
−
 
−
'''July 8, 2015'''
 
−
'''
 
−
A 1.5Tb RAID array was installed, replacing the old 1Tb array for /users/cs/study on '''phobos''', which was was failing.
 
−
 
−
'''July 5, 2015'''
 
−
 
−
The Math and Stats Windows 7 appserver, '''salvage''', is out of service. An update will be posted when this is resolved.
 
−
 
−
'''June 25, 2015'''
 
−
 
−
The Math and Stats Windows 7 appserver, '''isthmus''', was rebuilt with new hard drives.
 
−
 
−
'''May 27, 2015'''
 
−
 
−
The Math and Stats authoritative DNS and redundant LDAP server, '''fermat''', was replaced with a new 64-bit installation.
 
−
 
−
'''May 20, 2015'''
 
−
 
−
The Math and Stats Windows 7 appserver, '''salvage''' was rebuilt with a new 64-bit installation.
 
−
 
−
'''May 15, 2015'''
 
−
 
−
The Math and Stats Win7 appserver, '''isthmus''' was rebuilt with a new 64-bit installation.
 
−
 
−
'''May 13, 2015'''
 
−
 
−
The Math and Stats Matlab license server, '''noether''', was replaced with an upgraded virtual host with the same name, MAC address, and IP address.
 
−
 
−
'''May 11, 2015'''
 
−
 
−
1) Account archiving has begun on '''carme'''.
 
−
 
−
'''May 8, 2015'''
 
−
 
−
1) Account archiving has been completed on '''metis'''. 4473 accounts unused in the last two years were archived, freeing up 563G of disk space.
 
−
 
−
'''May 1, 2015'''
 
−
 
−
1) Account archiving has been completed on ''lysithea''.
 
−
 
−
2) Account archiving has begun on ''metis'' and will continue through next week.
 
−
 
−
'''April 30, 2015'''
 
−
 
−
Updated R packages in jon:/mnt/dbgen64test/usr/lib64/R/library and jon:/mnt/dbgen64dev/usr/lib64/R/library: '''survival''', '''Rlab'''.
 
−
 
−
'''April 29, 2015'''
 
−
 
−
1) The compute nodes '''herschel''', '''nancy''', '''kepler''', and '''igor''' were upgraded to the latest 64-bit gentoo image.
 
−
 
−
'''April 28, 2015'''
 
−
 
−
1) Accounts older than two years since last use are being archived from lysithea and metis this week.
 
−
 
−
2) The compute nodes '''hamilton''' and '''wallis''' were upgraded to the latest 64-bit gentoo image.
 
−
 
−
'''April 27, 2015'''
 
−
 
−
The virtual compute nodes '''box''', '''chow''', '''eden''', and '''norwood''' were consolidated. The new virtual compute node is '''box'''.
 
−
 
−
'''April 24, 2015'''
 
−
 
−
Lysithea and metis, student home directory servers, were upgraded to the 64 bit gentoo image.
 
−
 
−
'''April 23, 2015'''
 
−
 
−
Upgraded arche and mneme to 64 bit gentoo OS.
 
−
 
−
'''April 22, 2015'''
 
−
 
−
Finished upgrading vmcarme to 64 bit gentoo
 
−
 
−
'''April 10, 2015'''
 
−
 
−
The backup server '''hood''' has been upgraded.
 
−
 
−
'''April 9, 2015'''
 
−
 
−
1) A new backup server for math department backups has been set up in the CS server room. '''Luggage''', the new server, replaces the old servers '''crank''' and '''gasket''' and will complement '''trunk''' the other math department backup server housed in CS. 
 
−
 
−
2) The backup server '''clutch''' has been upgraded.
 
−
 
−
'''April 8, 2015'''
 
−
 
−
All printers in Math and Stats have been moved to the new print server, '''pomor'''. The old print server, '''avalon''', has been shut down.
 
−
 
−
'''April 4, 2015'''
 
−
 
−
1) The home directory server, '''phobos''', had become unresponsive and required a reboot. This server hosts the directories for /users/cs/grad, /users/cs/study, and /users/cs/misc.
 
−
 
−
'''April 2, 2015'''
 
−
 
−
'''Pomor''', the replacement for the print server '''avalon''' in Math and Stats, has been built, configured and is being tested. Remaining printers in Math and Stats will be moved to the new server early next week.
 
−
 
−
'''March 18, 2015'''
 
−
 
−
NTPD has been upgraded on our primary ntp servers so that they would not be susceptible for use in a distributed denial-of-service (DDoS) attack. A third primary ntp server (ender.cs.mun.ca) has been brought on line, which brings the total to three primary ntp servers for use by the MUN labnet community.
 
−
 
−
'''March 13, 2015'''
 
−
 
−
1) A new version of diskless Ubuntu (14x8) has been released on CS and PCGLABS servers. Older versions have been removed from CS servers, and will disappear from PCG Labs next week. Other labs throughout campus will be updated on request from their admins.
 
−
 
−
'''March 11, 2015'''
 
−
 
−
1) The old 32-bit servers '''thebe''', '''ls1''', '''curo''' and '''cialis''' have been taken out of service.
 
−
 
−
'''March 10, 2015'''
 
−
 
−
1) Yesterday we encountered NFS file corruption issues on the labnet/st1 student file system. These should all be repaired now, but please report any problems you may encounter.
 
−
 
−
2) The diskless Ubuntu distribution contains an application called zeitgeist which triggers NFS problems. We are building a new distribution without the problem application.
 
−
 
−
3) Bonavista was taken offline.
 
−
 
−
'''March 5, 2015'''
 
−
 
−
1) Riemann was updated (distribution, kernel). Required functionality (e.g., ssh server, alpine) was tested. Anybody who uses riemann is asked to provide feedback on any NFS issues encountered.
 
−
 
−
'''March 3, 2015'''
 
−
 
−
1) Installed on jon in server dev64 and server test64 a new lnmenuserver to fix issues with the latest win7logon not authenticating students. - PP
 
−
 
−
'''March 2, 2015'''
 
−
 
−
1) Removed dependency on old configuration data that was causing several account creation webtools (Temporary Account Creation and Create External Account via Adv. Account Mngr. as well as Machine Account Creation) to not function properly. - AC
 
−
 
−
'''February 26, 2015'''
 
−
 
−
1) We are gradually updating the few remaining 32-bit servers to 64-bit distributions.
 
−
 
−
2) Most of the servers supporting diskless ubuntu distributions started misbehaving today. The causes are not entirely clear, but we've got them all working properly again. If you find that they don't work for you at any point, please let us know.
 
−
 
−
3) Please let students know that if they encounter systems issues, they should let a sysadmin know about it.
 
−
 
−
4) The csgljet printer has been replaced with a brand new LaserJet 3015.
 
−
 
−
'''February 17, 2015'''
 
−
 
−
1) We will be upgrading various servers in the near future, as circumstances allow.  A security vulnerability in the C library has necessitated this. Newer and more critical systems have already been upgraded. Older systems generally have a more challenging upgrade path for pragmatic reasons as well as technical ones.
 
−
 
−
2) The server '''europa''' is being removed from service. Europa primarily acted as a web server for applications which have been obsoleted and unsupported for several years.
 
−
 
−
3) The server '''nermal''' is being removed from service. Nermal was the primary print server for C.S. and has been replaced by megatron.
 
−
 
−
'''February 10, 2015'''
 
−
 
−
1) All cs printers that were spooling through nermal have now been moved to megatron. If you are running a diskless client and you find that printing doesn't work on a particular printer (although it should), rebooting should fix the issue. If you prefer not to reboot, contact the systems group for an alternative solution.
 
−
 
−
2) Arlene, the master configuration server for LabNet, will be down briefly at 3pm while we move it physically to its new home.
 
−
 
−
'''February 3, 2015'''
 
−
 
−
1) Fixed the problem with gatekeeper so now remote scripts should run.  This includes the rename script that failed for bradleyd. 
 
−
 
−
2) Fixed the problems with the missing executables for exemptuser and printrefund webtools.
 
−
 
−
3) Added code to the win7login.py script to cause inactive or nonexistent accounts to become activated automatically.
 
−
 
−
4) Fixed erroneous queries within webtools that were causing issues with WIMPng loading and removing entities in System Configuration.
 
−
 
−
'''February 2, 2015'''
 
−
 
−
1) Arlene's O/S and packages were upgraded over the weekend. There is currently a compatibility issue between the python libraries and WebTools authentication.  This prevents people from logging into WebTools, and so is a priority for this morning.
 
−
 
−
2)  New PAMpython.so was deployed to fix issues with WebTools authentication in the afternoon.
 
−
 
−
3)  Final reconfiguration of Arlene was completed.
 
−
 
−
4)  The menuserver daemon was modified to detect inactive accounts so that account reactivation can occur.
 
−
 
−
'''January 31, February 1, 2015'''
 
−
 
−
1)  Upgraded arlene, the master configuration server, as a part of the ongoing upgrade process mandated by the vulnerablility in glibc.  ALthough not absolutely required arlene was upgraded to the 64 bit test image as it was very much in need of an upgrade.
 
−
 
−
'''January 30, 2015'''
 
−
 
−
1)  Connected student housing servers to new cabling and tested client computer.  Then Arranged with C&C to move client in West Towers to the proper Labnet VLAN and then tested client.
 
−
 
−
'''January 22, 2015'''
 
−
 
−
1) The colour printer is now back in the CS general office and working well.
 
−
 
−
2) Added alias entry for g++-4.9.2 as g++11 for use in a cs lab course
 
−
 
−
'''January 20, 2015'''
 
−
 
−
1)  Built server disk on loader for a new server being set up in the Commons for creating new images.
 
−
 
−
2)  Continued on with the project to enable Linux on all engineering computers.  The compressed images had all synced over last night so  the actual ssd images were made on each of the engineering application servers except for lobo (not enough space).  The computers in EN3000 were then configured to run Linux and booted into Gentoo in order to make the ssh keys.  The computers were then tested to see that they worked in Ubuntu, Gentoo and Windows7.
 
−
 
−
'''January 19, 2015'''
 
−
 
−
1)  Fixed problem in CS2718 Lab when one of the application servers lost the virtual disk containing the Ubuntu14x5 distribution.  The problem was initially remedied by pointing the clients to a different application server.  Later the virtual disk was restored.
 
−
 
−
2)  Started enabling Linux supprt for Engineering labs.  Created ssh keys and uploaded keys into our master database for computers in EN1038B.  Set up the rsyncing of the '''/images/linux''' images directory to all the Engineering application servers.
 
−
 
−
'''January 18,2015'''
 
−
 
−
1)  Set up two new computers in EN1049 to provide additional seating for Dr. Byrne's CS2718 course.  This endevour was fraught with many delays due to a lack of basic components.  An attempt should be made to have a few spare commonly used components readily available.
 
−
 
−
'''January 17, 2015'''
 
−
 
−
1)  After collecting changes to the Ubuntu14x5 image over the past week, a snapshot of this image was made and pushed up to pooky to be distributed to the various labnet application servers via the nightly software distribution cron job.
 
−
 
−
2)  The image was manually rsyned over to the departmental application servers, odie ane meatron and tests were made to ensure that the images worked in EN2036 and EN1049.
 
−
 
−
3)  It was noted that the computer, "chase" in EN2036 was inoperable due to a locked COW partition due to the COW partition having fill up.  Normal attempts to get rid of this COW partition were unsuccessful but a web page with the following URL:
 
−
 
−
https://www.globallinuxsecurity.pro/recovering-an-overflowed-lvm-volume-configured-with-virtualsize/
 
−
 
−
proved to be useful in remedying the problem.  Note this approach can be used whenever the '''lvs''' state of a volume look like '''-wi-Io----''' with a capital I as the fifth flag.  The capital i indicates that the logical volume is invalid.
 
−
 
−
'''January 16, 2015'''
 
−
 
−
1) Issue with stewart.lnesd.mun.ca contacting the Labnet configuration data base on arlene was resolved.
 
−
 
−
'''January 15, 2015'''
 
−
 
−
1) The C.S. genofflj printer has been temporarily replaced with an older printer.  A new printer will replace it in early February.
 
−
 
−
2) The colour printer in the C.S. general office is out for servicing.
 
−
 
−
3) Diskless on megatron.cs.mun.ca was upgraded to the test image so it would be in line with what is currently running in the other CS labs and in the Commons.
 
−
 
−
'''January 13, 2015'''
 
−
 
−
1)  CS faculty please note that speakers have been added to the A/V podium in EN1049.  (This was done in Dec., but not widely announced.)
 
−
 
−
2)  New wireless access points have been installed on the first floor of the Engineering building. This should greatly improve
 
−
service in the CS grad student lounge as well as EN1051 and EN1052.
 
−
 
−
3)  Network issues on login by the  CS secretarial staff this morning were found to due to  problems connecting with lysithea.pcgabs.mun.ca 's  /usr/local/pub share.
 
−
The reliance on this share, which has been an issue on past occasions,  has now been  disabled for all office staff machines.
 
−
The source of the problem was found to be the primary ldap server (scout.pcglabs.mun.ca)  for lysithea which had become unresponsive, The ldap service on scout.pcglabs.mun.ca was restarted. This resolved login issues being experienced across campus in addition to login and printing issues in the Commons. 
 
−
 
−
4)  At the request of IA's and Dr. Byrne, a new writable version of our iSCSI Ubuntu has been set up so that new software can be added.  The writable image is ssh'able on csgrad01.  The andrew, aaron, marian and pprice have been added to
 
−
to the sudoers config file and can do installs.  The computer will be available for updates until Friday at which point
 
−
a snapshot of the new image will be taken.  In the mean time feal free to add any relevent software that comes to mind.
 
−
 
−
'''January 12, 2015'''
 
−
 
−
1)  Fixed the problems in the automatic account generation code that have been plaguing the Commons.
 
−
 
−
2) Found a fix for the problem with Ubuntu14 login issues that failed due to "Failed session" error.
 
−
 
−
NOTE:  If people are having this problem logging in to Ubuntu then have the students use another session manager other than the default "Ubuntu" (unity) session manager.  The other two options are "Compiz" and "Metacity".  Basically these two look the same but "Compiz" is better able to take advantage of graphics card accellerators on supportted models (ie non Nvidia).  The button, a circular Ubuntu logo, that allows the user to switch session managers is located just above and to the right of the text box used to enter the user name.
 
−
 
−
'''January 10,11, 2015'''
 
−
 
−
1)  Set up '''sudoers''' managed file for Ubuntu computers on Campus.  This was done to allow members of CS2718 to
 
−
be able to run certain commands requiring super user privilege in a safe and secure way.  This will be used as a test case
 
−
that can be used for future instances where elevated privleges are required.
 
−
 
−
2)  Created a series of test cases for final the testing of the pam_labnet module and ran the code through the debugger to
 
−
validate the logic of each test case.  Then ran the code through '''valgrind'' code checker to check for use of uninitialized
 
−
valiables and memory leaks etc.  Installed the software on the development image for 32/64 bit severs and for development,
 
−
diskless client computers.  On Sunday after checking that authentication is still working on development images, the test and production environments were built and installed.  Notes: there may need to be some work done on some memory leaks on client side (see log[1-6].vlg files).
 
−
 
−
3)  Switched the commons over to the new Ubuntu14x4 image and tested a number of clients.
 
−
 
−
4)  Added Ubuntu14x4 image to the IA's office PC's.
 
−
 
−
'''January 9, 2015'''
 
−
 
−
1) Brought salsa.pcglabs.mun.ca the appserver for the Social work lab back up to date. It had been turned off since
 
−
last August (2014) due to building renovations.
 
−
Its new home is in CL3000. It was synced/upgraded to run the new server and diskless test images. The machine
 
−
was booted onto the latest kernel that we have tested without any problems.  It also has been reconfigured
 
−
to use grub2 a newer version of the grand unified boot loader. The grub2 boot loader is now in use on all CS department
 
−
machines and is being rolled out to other Labnet servers as they get upgraded.
 
−
 
−
2)  Setup and tested client support in CS1019 for the new Ubuntu14x4 image.  Discussed with Steve Johnson about the
 
−
possibility of using Ubuntu as an alternative to Gentoo in the labs.
 
−
 
−
3) Brought hope.engr.mun.ca back into sync in conjunction with the engineering department.
 
−
 
−
4) Found and adjusted a parameter on our master ldap server that prevented new slave ldap installations from completing to build.
 
−
 
−
'''January 8, 2015'''
 
−
 
−
1) Tested the booting of the new Ubuntu iSCSI image in computer Labs EN1049 and EN-2036.  Also added
 
−
Ubuntu to the list of images that 'helppc' could boot and tested it to make sure that it works.
 
−
Also distributed the new Ubuntu image to the application servers that boot the CS-1019 computer
 
−
lab in preparation of rolling out Ubuntu in this computer Lab.  Distributed the new image to the Commons as well.
 
−
 
−
2)  Noticed that there were intermittent problems with booting the iSCSI images in EN-2036.  When the operating
 
−
system is selected from the boot menu the initrd loads but fails to open the iSCSI disk.  It seems that the
 
−
customization process runs without the iSCSI partition being properly mounted leaving configuration files in
 
−
the mount point which in turn prevents the system from removing the mount point and thereby causing the
 
−
image creation to abort. The solution right now is to manually remove the mount point (rm /tmp/<computer name>).  If it
 
−
reoccurs, a more permanent solution will be applied.
 
−
 
−
3)  Talked to classroom support about the possibility of running Labnet on all the multimedia classrooms
 
−
in the engineering building.
 
−
 
−
4) Started work on getting printing support working from the new Ubuntu image.  This will probably involve
 
−
creating a template file that will display a list of printers available to each client.  The printer names
 
−
associated with each client  will be stored in the sys_config database.
 
−
 
−
'''January 7, 2015'''
 
−
 
−
1) Rolled out and announced initial version of this information page.
 
−
 
−
2) We experienced samba problems with access to home directories on lysithea.pcglabs.mun.ca. Decided to restart the samba service after consulting with the commons, which seems to have resolved the problem.
 
−
 
−
3) The EN1066 Mac lab has been re imaged as it was used for Mac OS X training during December. It is now running the latest version of OS X; Yosemite. It has been set up to use ldap authentication with our Labnet ldap servers so that users do not need to remember and use additional passwords beyond their MUN login one. The latest version of Xcode has been installed at the request of the mobile application course instructors. All students enrolled in the course have been added to the _developer group on each system in order to allow them to run Xcode.
 
−
 
−
4) Booted all the computers in CP2003 into Ubuntu in preparation for Dr. Byrne's course.
 
−
 
−
5) Updated the Ubuntu image and added several new packages needed by Dr. Byrne's course as well as some packages recommended by students and staff. Pushed the new image up to the master server for distribution this evening's rsync.
 
−
 
−
6) Set up the the classroom computers in EN1051, EN1052 and EN1054 so that they could
 
−
avail of the new Ubuntu iSCSI boot image.  To use simple reboot the computer and at the
 
−
boot menu select the "Ubuntu Linux" menu item.  When the login screen comes up login
 
−
using you Labnet credentials.
 
−
 
−
7) Modified the configuration files for apache to include include files that will allow students
 
−
with Dr. Byrnes course to be able to access course material from Dr. Byrne's GIT repository.
 
−
 
−
'''January 6, 2015'''
 
−
 
−
1) Set up the the classroom computers in EN1051, EN1052 and EN1054 so that they could avail of the new Ubuntu iSCSI boot image. To use, simply reboot the computer and at the boot menu select the "Ubuntu Linux" menu item. When the login screen comes up, login using your Labnet credentials.
 
−
 
−
2) Modified the configuration files for the apache server on stretch. The modifications direct apache to include files that will allow Dr. Byrne's students to be able to access course material from his GIT repository.
 
−
 
−
3) Sent off for a request for comments proposal to Hewlett Packard on a new printer standards initiative that will allow Labnet printing to be more functional with respect to printer job control and better cost recovery metrics.
 
−
 
−
4) CS genofflj printer has been temporarily removed for servicing.
 
−
 
−
'''Christmas Projects List 2014'''
 
−
 
−
During the Christmas break a number Labnet of projects were undertaken.
 
−
Many of these projects could not be performed during the semester due to
 
−
their intrusive nature.  The following are brief descriptions of the projects
 
−
arranged in more or less chronological order:
 
−
 
−
1)
 
−
Configured the computers in the  Computer Science labs and the CP2003
 
−
labs to boot the new iSCSI versions of the popular Ubuntu 14.0 distribution.
 
−
This involved updating the image and pushing the image up to our master
 
−
server and allowing it to update the various application servers.  The virtual
 
−
images were then created on the SSD disks of the application servers and the
 
−
database configurations were updated appropriately.  This will be used to
 
−
support one of Dr. Byrne's courses.
 
−
 
−
2)
 
−
Installed the server images for "isthmus" for the Math department.  This
 
−
was necessary because the libparted.so was upgraded and the buildserver
 
−
application no longer builds the partition tables properly.  It was necessary to manually
 
−
build the partitions and then proceed with the install.  This project is stalled
 
−
due to problems loading the boot sector.
 
−
 
−
3)
 
−
The size of our newer disk drives now exceeds the capabilities of the old "msdos"
 
−
partition tables.  The "GUID Partition Tables" or GPT will now be used in future server
 
−
builds to support the larger disks.  To boot from GPT partitioned disks the use of grub2
 
−
is under development.  In addition UEFI boot support is under investigation.  As servers
 
−
are upgraded our server builds will be utilizing GPT and will be migrating to grub2 for
 
−
booting.  When better support for UEFI booting becomes more mature, support
 
−
for UEFI will be incorporated.  Currently there are a couple of servers that have been
 
−
moved over to GPT partitions and the CS department is now booting with grub2.
 
−
 
−
4)
 
−
One of our core Labnet daemons, "master_configd", responsible for managing the
 
−
remote distribution of configuration services such as master file templates,
 
−
certificates, printer account numbers etc. was becoming severely
 
−
impacted by the shear number of requests that it received.  To ameliorate
 
−
this problem one of the modules was rewritten to perform work asynchronous
 
−
and thereby reduce the strain on the daemon.  The resulting code runs an
 
−
order of magnitude faster and service requests are no longer creating backlogs.
 
−
 
−
5)
 
−
Another service that was becoming bogged down was the user/system logging
 
−
database that provides statistics on when and where students log into Labnet and
 
−
what services they are requesting as well as the reboot logging of computers.  An
 
−
analysis of the data queries was performed to isolate the culprits.  As
 
−
it turned out the indices were not properly set up for several common queries and
 
−
this turned out to be the culprit.  The indexing was altered and now the queries
 
−
work 2 orders of magnitude faster.
 
−
 
−
6)
 
−
Performed network performance analysis on the Commons client/server
 
−
communications.  This was requested as a result of changes to the network
 
−
infrastructure.  Recommendations have been forwarded to the networking
 
−
group within C&C as well as the Commons.  It is hoped that this will be used
 
−
to redesign the network layout to remove some of the bottlenecks.
 
−
 
−
7)
 
−
The popular Ubuntu Linux 14.0 distribution was enabled for use in the Commons.  This
 
−
upgrade is similar to the upgrade to CP2003 and the Computer Science Department
 
−
computer labs.  This technology allows a client computer to mount a virtual disk over
 
−
IP networks using the iSCSI protocol.  The virtual disk is located on an application server
 
−
that can remotely support an entire lab.
 
−
 
−
8)
 
−
One of the final components of the "Account Archival Project" is the PAM Labnet module that
 
−
reenables an account when a student returns to university.  After the user's data has been collected and
 
−
written to DVD's the home directory is removed and the LDAP entry is put in hibernation.  When
 
−
an archived Labnet user attempts to login, the login credential is restored and a new home directory
 
−
is created. 
 
−
Wrote and debugged the PAM code that will be needed to reactivate user
 
−
accounts after they have been archived.
 
−
 
−
9)
 
−
To prepare for an up coming security, audit additions were made to the Labnet PAM module to allow for
 
−
the seamless update of our remaining DES password hashes to the SHA1 hashes used by C&C.  This will
 
−
provide greater protection of our password hashes in the event that our password hashes are exposed.
 
−
The newer SHA1 hashes are much more resiliant to brute force cracking than their DES counterparts..
 

Latest revision as of 18:31, 19 April 2022

Please check https://www.labnet.mun.ca/ for the latest info on Labnet issues that may affect users, such as

- scheduled maintenance

- software upgrades

- equipment upgrades

- current system-wide issues

Retrieved from "https://wiki.labnet.mun.ca/index.php?title=LabNet_News&oldid=8880"