curtis628
Posts: 7
Joined: Tue Aug 27, 2019 6:06 pm

Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Tue Aug 27, 2019 6:47 pm

Goal:
I'm trying to make a separate partition on my 64GB micro sd card, which I also plan on using to run Raspbian Buster Lite.

Background:
I'm looking to run a 3-node kubernetes (via k3s) cluster across 3 of my Pis. I've followed Alex's Will it Cluster? blog (taking notes from ljfranklins A "production-ish" Kubernetes cluster on Raspberry Pi notes as well)

I'd like to use GlusterFS as a dynamic persistence provider for my k8s cluster. In order to do so, my Pis will need a separate storage device formatted to use the XFS filesystem. I've successfully followed this GlusterFS on ARM blog, but it uses USB flash drives for this separate storage device.

I'd really like to avoid powering/buying USB flash drives for each GlusterFS Pi in my cluster. (My RPi 4 already runs too hot as it is in this cluster). Plus my 64GB micro SD card has plenty of space for my purposes.

I flashed my SD cards using Etcher + Raspian Buster Lite ... but I've been wildly unsuccessful at adding an additional non-boot partition. (Admittedly, my expertise when it comes to storage devices and partitioning is quite limited).

Here's some details I got from the Pi running Buster Lite.

Code: Select all

pi@raspberrypi:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0     179:0    0 59.5G  0 disk 
├─mmcblk0p1 179:1    0  256M  0 part /boot
└─mmcblk0p2 179:2    0 59.2G  0 part /

pi@raspberrypi:~ $ sudo fdisk /dev/mmcblk0
[...]
Command (m for help): p

Disk /dev/mmcblk0: 59.5 GiB, 63864569856 bytes, 124735488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa92c3c20

Device         Boot  Start       End   Sectors  Size Id Type
/dev/mmcblk0p1        8192    532480    524289  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      540672 124735487 124194816 59.2G 83 Linux

Command (m for help): q

pi@raspberrypi:~ $ cat /sys/block/mmcblk0/queue/optimal_io_size 
0
pi@raspberrypi:~ $ sudo cat /sys/block/mmcblk0/queue/optimal_io_size 
0
pi@raspberrypi:~ $ sudo cat /sys/block/mmcblk0/queue/minimum_io_size 
512
pi@raspberrypi:~ $ sudo cat /sys/block/mmcblk0/alignment_offset 
0
pi@raspberrypi:~ $ sudo cat /sys/block/mmcblk0/queue/physical_block_size 
512
I tried to resize the primary partition like this. I wasn't sure if resizing the primary partition of the running Pi was a good idea, but it's easy enough to re-flash the SD card with Buster and start all over. Here's my failed attempt using "parted"

Code: Select all

pi@raspberrypi:~ $ sudo parted
GNU Parted 3.2
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: SD SC64G (sd/mmc)
Disk /dev/mmcblk0: 63.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  273MB   268MB   primary  fat32        lba
 2      277MB   8000MB  7723MB  primary  ext4


(parted) mkpart                                                      
Partition type?  primary/extended? extended
Start? 8001MB                                                            
End? 63.9GB                                                               
(parted) print                                                            
Model: SD SC64G (sd/mmc)
Disk /dev/mmcblk0: 63.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type      File system  Flags
 1      4194kB  273MB   268MB   primary   fat32        lba
 2      277MB   8000MB  7723MB  primary   ext4
 3      8001MB  63.9GB  55.9GB  extended               lba

(parted) q                                                                
Information: You may need to update /etc/fstab.

pi@raspberrypi:~ $ lsblk                                                  
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0     179:0    0 59.5G  0 disk 
├─mmcblk0p1 179:1    0  256M  0 part /boot
├─mmcblk0p2 179:2    0  7.2G  0 part /
└─mmcblk0p3 179:3    0    1K  0 part 
I just guessed for Start + End for the new partition (probably not the right thing to do). It looked like the partition was made... but when I rebooted my Pi, I couldn't SSH into it anymore... making me think I messed something up.

Any tips would be greatly appreciated! Thanks.

epoch1970
Posts: 3579
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Tue Aug 27, 2019 7:47 pm

An extended partition is not a real partition. It is a container for partitions required when the Msdos disklabel contains more than 3 (primary) partitions.
You can use primary as partition type, you have one available.
The root filesystem for Raspbian should not stay at 277MB, you’ll run out of space very quickly, possibly before Kubernetes is installed in full.

When creating partitions, to enter boundaries (start, end) you can follow the default value proposed by the tool (should be printed in the message). If you need to specify your own values I’d advise to use sectors as units.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

curtis628
Posts: 7
Joined: Tue Aug 27, 2019 6:06 pm

Re: Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Tue Aug 27, 2019 8:57 pm

Thanks for the reply, @epoch1970 . I'll be sure not to use "extended" for the new partition.

By default, it appears either Etcher or Buster defaults to creating a small "part /boot" FAT32 partition (~256MB) and then creates another "Linux" ext4 root partition that takes up the rest of the space on the SD card.

NOTE: As mentioned before, forgive my obvious lack of understanding when it comes to storage devices and persistence. :)

It seems that my first step should be to resize (shrink) the "Linux ext4" root partition so I can then create a new partition that I can format with XFS and use with Gluster.

I tried to do this on a different Pi (using a 16GB card) as seen below. I tried to shrink the existing 15+GB root "Linux" partition down to 4GB so I could use the remaining ~11GB on my future partition for Gluster.

Code: Select all

pi@k8s-worker-1:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0     179:0    0 14.4G  0 disk 
├─mmcblk0p1 179:1    0  256M  0 part /boot
└─mmcblk0p2 179:2    0 14.2G  0 part /
pi@k8s-worker-1:~ $ sudo parted --list
Model: SD SA16G (sd/mmc)
Disk /dev/mmcblk0: 15.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  273MB   268MB   primary  fat32        lba
 2      277MB   15.5GB  15.2GB  primary  ext4


pi@k8s-worker-1:~ $ sudo parted
GNU Parted 3.2
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: SD SA16G (sd/mmc)
Disk /dev/mmcblk0: 15.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  273MB   268MB   primary  fat32        lba
 2      277MB   15.5GB  15.2GB  primary  ext4

(parted) resizepart                                                       
Partition number? 2                                                       
End?  [15.5GB]? 4GB                                                       
Warning: Shrinking a partition can cause data loss, are you sure you want to continue?
Yes/No? Yes                                                               
(parted) print                                                            
Model: SD SA16G (sd/mmc)
Disk /dev/mmcblk0: 15.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  273MB   268MB   primary  fat32        lba
 2      277MB   4000MB  3723MB  primary  ext4

(parted) quit                                                             
Information: You may need to update /etc/fstab.

pi@k8s-worker-1:~ $ sudo reboot    
However, after it reboots, I'm no longer able to SSH into that Pi. I'm not sure if shrinking an active, primary partition is causing this to break? I'm also not sure what changes I might need to make to /etc/fstab.

RonR
Posts: 480
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Tue Aug 27, 2019 10:21 pm

curtis628 wrote:
Tue Aug 27, 2019 8:57 pm
However, after it reboots, I'm no longer able to SSH into that Pi. I'm not sure if shrinking an active, primary partition is causing this to break? I'm also not sure what changes I might need to make to /etc/fstab.

You must use resize2fs to shrink the ext4 filesystem before you shrink the partition. Otherwise, structures and data may be residing in the portion of the partition that you're chopping off.

See image-shrink in Image File Utilities for an example.

curtis628
Posts: 7
Joined: Tue Aug 27, 2019 6:06 pm

Re: Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Wed Aug 28, 2019 3:33 am

You must use resize2fs to shrink the ext4 filesystem before you shrink the partition
I see. Makes sense. I tried to do this with

Code: Select all

pi@raspberrypi:~ $ sudo resize2fs /dev/mmcblk0p2 4G
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
resize2fs: On-line shrinking not supported
... but it looks like on-line shrinking of ext4 root partition isn't supported. I poked around online to find a work-around for shrinking an online partition (like this) ... but couldn't get it to work.

I bought a USB micro SD card reader so I can try and run resize2fs on a non-active partition to see if that gives me better luck.

RonR
Posts: 480
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Wed Aug 28, 2019 4:15 am

curtis628 wrote:
Wed Aug 28, 2019 3:33 am
I bought a USB micro SD card reader so I can try and run resize2fs on a non-active partition to see if that gives me better luck.

The Linux partition in a Raspbian Buster Lite image is partitioned to 1.8 GB. Use image-shrink to resize it down to a minimum of 1.3 GB or up to a maximum of whatever-you-like GB. Then use image-mount to mount the FAT32 partition of the image file and edit cmdline.txt to remove "init=/usr/lib/raspi-config/init_resize.sh" from the end of cmdline.txt. Use Etcher to write that image to an SD card and you will have a root partition of the desired size and you can create your custom partition in the free space that follows it.

curtis628
Posts: 7
Joined: Tue Aug 27, 2019 6:06 pm

Re: Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Fri Sep 13, 2019 6:36 pm

Hey RonR

I followed your (excellent) instructions and completed the following:
  • used image-shrink to create a ~5GB Linux partition with Raspbian Lite
  • removed init_resize from cmdline.txt in the W95 image
  • flashed image using Etcher
I booted up the Pi with the SD card and confirmed that the other ~58GB is unallocated.

Code: Select all

pi@raspberrypi:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0     179:0    0 59.5G  0 disk 
├─mmcblk0p1 179:1    0  256M  0 part /boot
└─mmcblk0p2 179:2    0  5.2G  0 part /
pi@raspberrypi:~ $ sudo parted /dev/mmcblk0
GNU Parted 3.2
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print free                                                       
Model: SD SC64G (sd/mmc)
Disk /dev/mmcblk0: 63.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
        32.3kB  4194kB  4162kB           Free Space
 1      4194kB  273MB   268MB   primary  fat32        lba
        273MB   277MB   4194kB           Free Space
 2      277MB   5853MB  5576MB  primary  ext4
        5853MB  63.9GB  58.0GB           Free Space

(parted) q 
But I'm having problems creating a third XFS partition on this SD card that I want to use for glusterfs. I've tried various times, but afterwards upon reboot, the Pi boots to 4 raspberries and won't move forward (which seems to be indicative that I hosed the Micro SD card somehow).

I've tried creating this third partition using the live Pi's SD card (/dev/mmcblk0) ... and I've tried using a different Pi + a Micro SD Card USB reader and creating the new partition on a non-live Micro SD card - but both led to the Pi stopping after rebooting.

Here's an example of me trying to add the third partition on the Pi's running SD card.

Code: Select all

pi@raspberrypi:~ $ sudo parted /dev/mmcblk0
GNU Parted 3.2
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mkpart                                                           
Partition type?  primary/extended? primary                                
File system type?  [ext2]? xfs
Start? 5857                                                               
End? 100%                                                                 
(parted) print                                                            
Model: SD SC64G (sd/mmc)
Disk /dev/mmcblk0: 63.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  273MB   268MB   primary  fat32        lba
 2      277MB   5853MB  5576MB  primary  ext4
 3      5857MB  63.9GB  58.0GB  primary  xfs          lba

(parted) q                                                                
Information: You may need to update /etc/fstab.

pi@raspberrypi:~ $ lsblk                                                  
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0     179:0    0 59.5G  0 disk 
├─mmcblk0p1 179:1    0  256M  0 part /boot
├─mmcblk0p2 179:2    0  5.2G  0 part /
└─mmcblk0p3 179:3    0   54G  0 part 
But after doing that and rebooting the Pi, the Pi never boots: it just has 4 raspberries with a blinking cursor.

Any tips on what I'm doing wrong?

curtis628
Posts: 7
Joined: Tue Aug 27, 2019 6:06 pm

Re: Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Fri Sep 13, 2019 6:56 pm

FWIW - I get the same outcome when trying to create a 3rd ext4 primary partition as well. After adding the 3rd partition, the Pi refuses to boot. :roll:

RonR
Posts: 480
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Fri Sep 13, 2019 8:22 pm

curtis628 wrote:
Fri Sep 13, 2019 6:36 pm
Information: You may need to update /etc/fstab.
.
.
.
But after doing that and rebooting the Pi, the Pi never boots: it just has 4 raspberries with a blinking cursor.

Any tips on what I'm doing wrong?

PARTED does not preserve the partition table uuid (PARTUUID) and changes it to a new value. Consequently, you must update /boot/cmdline.txt and /etc/fstab with the new PARTUUID value before rebooting.

curtis628
Posts: 7
Joined: Tue Aug 27, 2019 6:06 pm

Re: Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Sat Sep 14, 2019 12:00 am

RonR wrote:
Fri Sep 13, 2019 8:22 pm
PARTED does not preserve the partition table uuid (PARTUUID) and changes it to a new value. Consequently, you must update /boot/cmdline.txt and /etc/fstab with the new PARTUUID value before rebooting.
That was my problem! Thank you SO much for your patience and help in getting this to work. I learned a lot about partitions in the process. :)

In case this helps anyone else in the future, here's a breakdown of how I got this to work.

After flashing my 5GB Raspbian Buster Lite image onto a SD card, I put the card into my Micro SD Card USB reader, and inserted that into an existing Pi's USB port running Buster (with UI). The new SD card was available on /dev/sdb for me. Here's the initial details I saw. Notably, the PARTUUID prefix for this new SD card was 17869b7d and you can see that the SD card has a bunch of unallocated free space on it for me to create the XFS partition:

Code: Select all

pi@k8s-master:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb           8:16   1 59.5G  0 disk 
├─sdb1        8:17   1  256M  0 part /media/pi/boot
└─sdb2        8:18   1  5.2G  0 part /media/pi/rootfs
mmcblk0     179:0    0 14.4G  0 disk 
├─mmcblk0p1 179:1    0  256M  0 part /boot
└─mmcblk0p2 179:2    0 14.2G  0 part /
pi@k8s-master:~ $ sudo blkid
/dev/mmcblk0p1: LABEL_FATBOOT="boot" LABEL="boot" UUID="0C61-73F5" TYPE="vfat" PARTUUID="81da9ecb-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="43f2d0bb-83be-464f-94d0-9a751f376c64" TYPE="ext4" PARTUUID="81da9ecb-02"
/dev/sdb1: LABEL_FATBOOT="boot" LABEL="boot" UUID="F661-303B" TYPE="vfat" PARTUUID="17869b7d-01"
/dev/sdb2: LABEL="rootfs" UUID="8d008fde-f12a-47f7-8519-197ea707d3d4" TYPE="ext4" PARTUUID="17869b7d-02"
/dev/mmcblk0: PTUUID="81da9ecb" PTTYPE="dos"
pi@k8s-master:~ $ sudo parted /dev/sdb print free
Model: Generic- USB3.0 CRW -SD (scsi)
Disk /dev/sdb: 63.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
        32.3kB  4194kB  4162kB           Free Space
 1      4194kB  273MB   268MB   primary  fat32        lba
        273MB   277MB   4194kB           Free Space
 2      277MB   5853MB  5576MB  primary  ext4
        5853MB  63.9GB  58.0GB           Free Space
I then created a new (primary) XFS partition using gparted (and applied the changes).

But now, as RonR helpfully noted, the PARTUUID prefix changed from 17869b7d to d06c378a.

Code: Select all

pi@k8s-master:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb           8:16   1 59.5G  0 disk 
├─sdb1        8:17   1  256M  0 part /media/pi/boot
├─sdb2        8:18   1  5.2G  0 part /media/pi/rootfs
└─sdb3        8:19   1   54G  0 part 
mmcblk0     179:0    0 14.4G  0 disk 
├─mmcblk0p1 179:1    0  256M  0 part /boot
└─mmcblk0p2 179:2    0 14.2G  0 part /
pi@k8s-master:~ $ sudo blkid
/dev/mmcblk0p1: LABEL_FATBOOT="boot" LABEL="boot" UUID="0C61-73F5" TYPE="vfat" PARTUUID="81da9ecb-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="43f2d0bb-83be-464f-94d0-9a751f376c64" TYPE="ext4" PARTUUID="81da9ecb-02"
/dev/sdb1: LABEL_FATBOOT="boot" LABEL="boot" UUID="F661-303B" TYPE="vfat" PARTUUID="d06c378a-01"
/dev/sdb2: LABEL="rootfs" UUID="8d008fde-f12a-47f7-8519-197ea707d3d4" TYPE="ext4" PARTUUID="d06c378a-02"
/dev/mmcblk0: PTUUID="81da9ecb" PTTYPE="dos"
/dev/sdb3: LABEL="gluster" UUID="1c9024f4-8af1-4770-bc0f-4dcc93bc25e7" TYPE="xfs" PARTUUID="d06c378a-03"
pi@k8s-master:~ $ sudo parted /dev/sdb print
Model: Generic- USB3.0 CRW -SD (scsi)
Disk /dev/sdb: 63.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  273MB   268MB   primary  fat32        lba
 2      277MB   5853MB  5576MB  primary  ext4
 3      5857MB  63.9GB  58.0GB  primary  xfs
Before removing the SD card from the USB card reader, I edited the /media/pi/boot/cmdline.txt and /media/pi/rootfs/etc/fstab files, replacing the old PARTUUID (17869b7d for me) with the new PARTUUID (d06c378a for me) generated by gparted.

I also added an entry in the /etc/fstab to mount this new partition to the /data/glusterfs/gv0/brick1 directory (which I also created on the Linux partition at this point). See below.

Code: Select all

pi@k8s-master:~ $ cat /media/pi/rootfs/etc/fstab 
proc            /proc           proc    defaults          0       0
PARTUUID=d06c378a-01  /boot           vfat    defaults          0       2
PARTUUID=d06c378a-02  /               ext4    defaults,noatime  0       1
PARTUUID=d06c378a-03  /data/glusterfs/gv0/brick1               xfs    defaults  0       0
pi@k8s-master:~ $ cat /media/pi/boot/cmdline.txt 
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=d06c378a-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet
I unmounted and removed the SD card from the USB card reader and now put it into my Pi. It booted up like a charm! Here's what it looks like when running on the Pi directly.

Code: Select all

pi@raspberrypi:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0     179:0    0 59.5G  0 disk 
├─mmcblk0p1 179:1    0  256M  0 part /boot
├─mmcblk0p2 179:2    0  5.2G  0 part /
└─mmcblk0p3 179:3    0   54G  0 part /data/glusterfs/gv0/brick1
pi@raspberrypi:~ $ sudo blkid
/dev/mmcblk0p1: LABEL_FATBOOT="boot" LABEL="boot" UUID="F661-303B" TYPE="vfat" PARTUUID="d06c378a-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="8d008fde-f12a-47f7-8519-197ea707d3d4" TYPE="ext4" PARTUUID="d06c378a-02"
/dev/mmcblk0p3: LABEL="gluster" UUID="1c9024f4-8af1-4770-bc0f-4dcc93bc25e7" TYPE="xfs" PARTUUID="d06c378a-03"
/dev/mmcblk0: PTUUID="d06c378a" PTTYPE="dos"
Major props and thanks to RonR.

curtis628
Posts: 7
Joined: Tue Aug 27, 2019 6:06 pm

Re: Micro SD Card Partitions and Raspbian (for k8s + gluster Pi cluster)

Sat Sep 14, 2019 12:03 am

Side Note: I tried doing similar steps outlined above using parted running "live" on the Pi with the freshly flashed SD card, but the Pi refused to boot after
  • adding the new XFS partition using parted
  • updating the PARTUUID prefix on cmdline.txt and /etc/fstab files
  • reboot
The Pi refused to boot up after restarting after those steps for some reason. Not sure if I did something wrong, or if messing with "live" data devices like that is just not feasible.

Return to “General discussion”