Page 1 of 1

Flashing RaspberryPi CM3 in a production context

Posted: Tue May 07, 2019 7:37 pm
by yannickb
Hi,

We've been doing a RPi CM3 Computer Board projet since awhile and we are done with the hardware part, but the software part makes me frankly depressed as of late.

I'm trying to make a small software to flash RPi CM3, so that our production team can flash about a hundred of them each day.

What I don't understand is how can I also checksum the eMMC flash for quality control, right after flashing?

If I do (granted the CM3 was rpiboot'ed properly and is connected to my computer as a USB MSD, designed by /dev/sdb under linux):

Code: Select all

dd if=./sdcard.img of=/dev/sdb conv=fsync
Then I tried doing a small checksumming routine (that uses direct IO in order to bypass Linux internal buffers):

Code: Select all

#define _GNU_SOURCE
#include <fcntl.h>
#include <malloc.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>

#include <zlib.h>

int main(int argc, char **argv) {
    int fd;
    ssize_t numRead;
    size_t length, alignment;
    off_t offset;
    char *buf;

    const char *img_file = "./sdcard.img";
    const char *dev_file = "/dev/sdb";
    struct stat img_stat;
    stat(img_file, &img_stat);

    length = img_stat.st_size;
    offset = 0;
    alignment = 4096;

    fd = open(dev_file, O_RDONLY | O_DIRECT);
    if (fd == -1) {
        printf("error: open");
        return 1;
    }

    /* memalign() allocates a block of memory aligned on an address that
        is a multiple of its first argument. By specifying this argument as
        2 * 'alignment' and then adding 'alignment' to the returned pointer,
        we ensure that 'buf' is aligned on a non-power-of-two multiple of
        'alignment'. We do this to ensure that if, for example, we ask
        for a 256-byte aligned buffer, we don't accidentally get
        a buffer that is also aligned on a 512-byte boundary. */

    buf = memalign(alignment * 2, length + alignment);
    if (!buf) {
        printf("error: memalign");
        return 1;
    }

    buf += alignment;

    if (lseek(fd, offset, SEEK_SET) == -1) {
        printf("error: lseek");
        return 1;
    }

    numRead = read(fd, buf, length);
    if (numRead == -1) {
        printf("error: read");
        return 1;
    }
    
    printf("Read %ld bytes\n", (long)numRead);
Basically all it does is doing a checksumming of the RAW bytes found on /dev/sdb up to the exact count of bytes sdcard.img contains.

Here are my questions.
- When resetting USB power and doing a new rpiboot, the checksum are different, despite the partitions being correct. I suppose this is some kind of wear leveling/TRIM/block redordering done by the eMMC? Would there be a way to circumvent this? Basically be able to checksum the flash even after a power off? TL;DR How can we checksum N bytes on the eMMC consistently?

- When using dd, the checksum is often wrong (after writing with fsync) even though power wasn't turned off. On the other hand, when using something like usb-creator-gtk (on Ubuntu), the checksum is systematically correct. Checking their source code, it seems the latter software relies on udisks. I wonder if anybody has an idea on why it doesn't work correctly with dd, but does with Canonical's usb-creator-gtk.

Any help would be greatly appreciated.

Re: Flashing RaspberryPi CM3 in a production context

Posted: Tue May 07, 2019 10:27 pm
by incognitum
yannickb wrote:
Tue May 07, 2019 7:37 pm
Then I tried doing a small checksumming routine (that uses direct IO in order to bypass Linux internal buffers):
May want to double check if you are also seeing the same problems using standard tools.
E.g. just feed the first 100 MB to sha1sum:

Code: Select all

dd if=/dev/sdb bs=1M count=100 iflag=direct status=none |sha1sum

Re: Flashing RaspberryPi CM3 in a production context

Posted: Wed May 08, 2019 5:47 am
by gsh
I think it's likely that Linux is opening the partitions while you are checksumming. It will then write to the partitions... How about you read the partitions back using dd afterwards and see where the differences are?

Gordon

Re: Flashing RaspberryPi CM3 in a production context

Posted: Wed May 08, 2019 7:58 am
by thof
How about using a 3rd party software? We are using Etcher (https://www.balena.io/etcher/) that software has a check built in and has support for compute modules. It even supports writing one image to multiple devices simultaneously.

Re: Flashing RaspberryPi CM3 in a production context

Posted: Wed May 08, 2019 11:02 pm
by yannickb
gsh wrote:
Wed May 08, 2019 5:47 am
I think it's likely that Linux is opening the partitions while you are checksumming. It will then write to the partitions... How about you read the partitions back using dd afterwards and see where the differences are?

Gordon
Hi Gordon, it indeed was that. Damn! I was starting to think it was a problem with my flashing PCB.

More specifically it was udisks2 service automounting the partitions. Disabling completely the service fixed once and for all my issues. I wasn't too aware mounting (non read-only) actually changed the checksums of a partition! I always thought mounting was "harmless" (in a I/O sense) until you actually wrote to the partition. Is it journaling stuff or something (considering the image contains ext4 partitions)?

On another note since I have your attention. I wonder if it's possible to flash the CM3 with USB3 speed capabilities? Would be useful for us given the shear volume of CM3s that goes outside the factory.
thof wrote:
Wed May 08, 2019 7:58 am
How about using a 3rd party software? We are using Etcher (https://www.balena.io/etcher/) that software has a check built in and has support for compute modules. It even supports writing one image to multiple devices simultaneously.
No it's not an option sadly. Altough I didn't know the software (seems well done indeed, and I might actually start copying some stuff from their UI), we can't simply because the PCB also does voltage examinations on our mainboard, some diagnosis of GPIOs, power tests, even digital HDMI screen testing (using a FPGA). Those are quality control tests that are integrated and done in the flashing software, while flashing (to make speed OK).

Re: Flashing RaspberryPi CM3 in a production context

Posted: Fri May 10, 2019 10:48 am
by incognitum
yannickb wrote:
Wed May 08, 2019 11:02 pm
On another note since I have your attention. I wonder if it's possible to flash the CM3 with USB3 speed capabilities? Would be useful for us given the shear volume of CM3s that goes outside the factory.
If speed is a concern, you can also consider running a script on the CM to let it do its own flashing and verification.
Allows streaming a compressed image, needing less bandwidth.

https://github.com/raspberrypi/scriptexecutor