Recover LVM Boot partition within a live CD (installing GRUB)

This allows you to recover the Master Boot record when a RAID 5 array fails.

1. Activate LVM partitions

   sudo mdadm --assemble --scan
   sudo vgchange -a y [name of volume group]

2. Mount all volumes to a single mount-point

   sudo mount /dev/mapper/[volume group]/[root volume] /mnt/
   sudo mount /dev/mapper/[volume group]/[usr volume] /mnt/usr/
   sudo mount /dev/mapper/[volume group]/[home volume] /mnt/home/
   ... etc

3. Mount and bind directories for grub to detect the underlying hardware

   sudo mount --bind /dev /mnt/dev
   sudo mount --bind /dev/pts /mnt/dev/pts
   sudo mount --bind /proc /mnt/proc
   sudo mount --bind /sys /mnt/sys

4. Jump into the new mounted system

   sudo chroot /mnt

5. Install GRUB

   grub-install /dev/sda
   grub-install --recheck /dev/sda
   update-grub

6. Exit the environment and restart

   exit
   sudo umount /mnt/dev/pts
   sudo umount /mnt/*
   sudo shutdown -r now

Recovering a RAID 5 array

Recently had a RAID 5 array fail on me. These are the steps I took to recover the data.

NOTE! The order of the disks in /dev/sdX follows the numbering of the ports on the motherboard. So if a disk is plugged into port 0, it will show up as /dev/sda. Keep this in mind as if you remove a failed disk, it might mess up the references in the array!.

1. Boot up from a Live CD
2. Install the software RAID management software

   sudo apt-get install mdadm

3. Make sure RAID and LVM are unmounted

   sudo vgchange -a n [name of your volume group]  sudo mdadm -S /dev/md0

4. Copy the failed disk to the new disk (in case the disk was the boot disk, you need to copy that flag across so that your system boots)

   sudo sfdisk -d /dev/sdx | sudo sfdisk /dev/sdy

5. Check that the disks are the same

   sudo fdisk -l

6. Mount the RAID array, and remove the failed disk

   sudo mdadm --assemble --scan  sudo mdadm --manage --remove /dev/sdx

7. Add the new disk

   sudo mdadm --manage --add /dev/sdy

8. Watch the progress as mdadm rebuilds your RAID array

   sudo cat /proc/mdstat

Securing Ubuntu 12.04

I have been extended my knowledge of Linux System Administration (in particular, securing Linux systems), and as such the first thing I did when I installed Ubuntu 12.04 LTS was to lock it down.

Installing Ubuntu is outside the scope of this blog post, so if you don't have Ubuntu yet I suggest you follow the official documentation.

Networking

We will first deal with locking down our machines network access. By restricting how our machine communicates with others we narrow down the attack vectors available.

1. The first thing we should do is turn on a firewall. Ubuntu comes with ufw pre-installed so we will just use that (I have covered this in a previous blog post).
   

   sudo ufw enable

   Check its status with:
   

   sudo ufw allow ssh

2. Enable any services you will need. For instance a web server will need the HTTP port of 80 open.
   

   sudo ufw allow ssh
   
   # You can specify a port directly
   sudo ufw allow 80
   # You can also specify whether it is TCP or UDP
   sudo ufw allow 80/tcp
   # Finally, you can specify whether it is incoming or outgoing
   sudo ufw allow in 80
   sudo reject out 1337

3. Sysctl allows you to configure the Linux kernel during runtime. We will edit the file /etc/sysctl to harden our network interface; open the file in your favorite editor and make the following changes:

   #IP spoofing/forging protection by turning on the reverse path filter
   net.ipv4.conf.all.rp_filter=1
   net.ipv4.conf.default.rp_filter=1
   
   # Protect against ICMP attacks
   net.ipv4.icmp_echo_ignore_broadcasts=1
   net.ipv4.icmp_ignore_bogus_error_responses=1
   
   # Turn off IPv4 features that are easy to abuse
   net.ipv4.conf.all.accept_source_route=0
   net.ipv6.conf.all.accept_source_route=0
   net.ipv4.conf.default.accept_source_route=0
   net.ipv6.conf.default.accept_source_route=0
   net.ipv4.conf.all.send_redirects=0
   net.ipv4.conf.default.send_redirects=0
   net.ipv4.conf.all.accept_redirects=0
   net.ipv6.conf.all.accept_redirects=0
   net.ipv4.conf.all.secure_redirects=0
   
   # Block SYN attacks
   net.ipv4.tcp_syncookies=1
   net.ipv4.tcp_max_syn_backlog=2048
   net.ipv4.tcp_synack_retries=2
   net.ipv4.tcp_syn_retries=2
   
   # Log Martians
   net.ipv4.conf.all.log_martians=1
   
   # Ignore directed ICMP pings
   net.ipv4.icmp_echo_ignore_all=1
   
   # Don't perform IP forwarding
   net.ipv4.ip_forward=0
   
   #####
   # IPv6
   #####
   
   # Number of router solicitations to send until assume no routers present
   net.ipv6.conf.default.router_solicitations=0
   
   # Do not accept router preferences
   net.ipv6.conf.default.accept_ra_rtr_pref=0
   
   # Do not accept prefix info from router
   net.ipv6.conf.default.accept_ra_pinfo=0
   
   # Do not accept Hop limit settings from router
   net.ipv6.conf.default.accept_ra_defrtr=0

   
   
4. Reload sysctl with your changes:
   

   sudo sysctl -p

5. Secure your TCP Wrapper by editing the /etc/hosts.deny file, ensuring the following line is the only one uncommented:
   

   ALL: ALL

6. Allow your TCP Wrapper services (like SSH) by editing the /etc/hosts.allow file. The basic syntax is:
   

   <service>: <host/network>

7. Prevent IP Spoofing via DNS by editing the file /etc/host.conf and adding the following lines:
   

   order bind,hosts
   nospoof on

8. If you have not already done so, update your system so that there are no security vulnerabilities:
   

   sudo apt-get update
   sudo apt-get upgrade

9. Install nmap, a tool for network discovery and security auditing:
   

   sudo apt-get install nmap

10. Perform a local nmap TCP scan of your machine and ensure that all ports that are open are supposed to be open.
    

    sudo nmap -v -sT localhost

    Perform a SYN scan, which is another way a hacker can probe your system:
    

    sudo nmap -v -sS localhost

    Perform a UDP scan to determine which UDP services are operational:
    

    sudo nmap -v -sU localhost

11. Perform the same NMap tests but on another host. If you followed the above instructions you may want to add -PN to the command so that nmap ignores the fact that your machine does not respond to pings. Note that this scan may take some time...

 

Filesystem

We will now protect our file-system.  

Note: you will get the best security by putting your system directories into their own partition. This will allow you to specify the mount options for each directory. This guide will only cover the default install.

1. Protect your shared memory by editing /etc/fstab as follows:
   

   tmpfs  /dev/shm  tmpfs  defaults,noexec,nosuid  0  0

2. Bind /var/tmp to /tmp so that we limited what applications can do with that system directory. Edit /etc/fstab as follows:
   

   /tmp  /var/tmp  none  rw,noexec,nosuid,nodev,bind  0  0

 

Startup Applications

We will now modify the start-up applications and services that turn on during boot.

1. Display the hidden start-up applications:
   

   sudo sed -i 's/NoDisplay=true/NoDisplay=false/g' /etc/xdg/autostart/*.desktop

2. Press the windows key on your keyboard, type in 'Startup Applications' and launch the program of the same name
3. Disable the following services (Note: These may change depending on your personal situation):
• Backup monitor
• Bluetooth manager
• Chat
• Desktop Sharing
• Gwibber
• Orca Screen Reader
• Personal File Sharing
• Ubuntu One

 

Disable Guest Login

Just edit /etc/lightdm/lightdm.conf and add the following line:

allow-guest=false

 

References:

• 'How to secure Ubuntu 12.04 LTS server' by thefanclub.co.za
• 'Securing a Linux VPS' from wolfpaw.co.uk
• NSA guide to securing RHEL 5
• NSA hardening tips for RHEL 5
• NSA guides to securing other operating systems
• 'Tuning the Linux kernel', a paper by Long Yi and James Connan
• 'Linux Kernel /etc/sysctl.conf Security Hardening' from www.cyberciti.biz
• 'Security auditing with NMAP' by www.techrepublic.com 
• Nmap reference for scanning techniques
• 'Securing an Ubuntu Server' by www.andrewault.net
• Ubuntu 12.04 LTS security by rationally paranoid
• Modify Ubuntu blog

Access an LVM module (Ubuntu 12.04)

A computer died on me and I needed to recover some of the data. I thought it would simply be a case of popping the disk into an external hard-drive case and ripping the data out. Unfortunately, I encountered a little snag involving LVM.

My Ubuntu 12.04 machine could not recognise the LVM partition natively. Basically this meant I had to configure the machine so that I could manipulate the disk; a quick Google search revealed this post from linux-sxs.org.

To make your life easier, here are the steps:

1. Install lvm2, which is the libraries and utilities required to manipulate LVM volumes.
   

   sudo apt-get install lvm2

2. Use the following command to scan for available LVM volume groups and find the one you are looking for (in this example we are going to assume our volume group is VolumeGroup01)
   

   sudo vgscan

3. Activate your volume
   

   sudo vgchange -ay VolumeGroup01

4. Search for the volume that contains the partition you are looking for (we are going to assume that the volume is called LogicalVolume01)
   

   sudo lvs

5. Create a mount point for your volume
   

   sudo mkdir /mnt/destination

6. Mount the volume
   

   sudo mount /dev/VolGroup00/LogVol00 /mnt/fcroot -o ro,user

 

References:

• 'Accessing a Fedora Logical Volume from Ubuntu' from linux-sxs.org
• Relevant question that was posted in linuxquestions.org
• LVM reference from tlpd.org
• Beginners guide to LVM from howtoforge.com

Setting up a CentOS 6.2 web server: Securing the file system

This is a follow on post from my guide to installing CentOS 6.2. We will go through some of the steps required to secure your server and get it ready for production use.

This section will outline how to lock down your partitions and file system. We will assume that you went for a file system structure similar to the one outlined in the above blog post.

Since the filesystem flags will be the most foriegn concept in this guide, I will give a quick outline about file system flags. However, I strongly suggest you follow the links provided in the references for more in-depth detail on any topic provided in this guide.

• nosuid disallows the filesystem from granting a user the temporary elevated privileges of a file's owner or group.
• noexec disallows the filesystem from running an executable.
• nodev disallows the filesystem from running files as block devices (i.e. treat the file as an I/O source or sink).

The process for securing your file system is as follows:

1. Secure your partitions by editing /etc/fstab as follows:
• Add nosuid, noexec, and nodev to partitions like  /dev/shm, /var/log, /tmp, and /var/log/audit. Basically any partition where you only expect to read and write files.
• Add nodev to all non-root file systems like /home and /var/www. You can add the noexec flag if you want, but note that cgi scripts stored in /var/www will break (as well as any scripts stored in the user's home directory).
• Add nosuid only to file systems like /var.
• DO NOT ADD ANY OF THESE FLAGS TO /!!!!!

 
2.  Add the following line to /etc/fstab to hardlink /var/tmp to /tmp
   
   /tmp /var/tmp none rw,noexec,nosuid,nodev,bind 0 0
    
3. Disable the autofs service if you do not need NFS (unless you have already un-installed the service):
   
   chkconfig autofs off 

References

• Howto Forge's guide: 'The Perfect Server - CentOS 6.2 x86_64 with Apache2'
• Howto Forge's guide: 'Installing Apache2 with PHP5 and MySQL support on CentOS 6.2'
• Red Hat Linux 5 Hardening Tips - National Security Agency
• Guide to the secure configuration of Red Hat Linux 5 - National Security Agency
• CentOS wiki on hardening the OS