Page 29 of 33

Re: CAN controller

Posted: Sat Apr 25, 2015 8:33 am
by msperl
Feldspar17 wrote:Martin -
I see valid CAN data on the Pi via candump - but only for a couple of seconds or so, before it stops receiving completely.

1. Do you think this is related to the 2 buffer mcp2515 limitation you've mentioned before?
2. If so, is it possible to update to the 3.18 version of the kernel on a Model B and fix this problem, or would I have to switch to the Pi 2?
ad 1: 2 buffers will only produce missed packets, but would continue (unless - see below)
ad 2: I guess you use a 3.12 kernel plus the mcp251x driver module. With that setup it can happen that the mcp251x driver is stuck when too many packets arrive in a short time. The 3.18 Kernel does not show this behavior, so I would recommend moving to the latest foundation kernel and use the device-tree-overlays as describes several pages earlier...

Re: CAN controller

Posted: Wed Apr 29, 2015 7:56 pm
by Feldspar17
msperl wrote:
Feldspar17 wrote:Martin -
I see valid CAN data on the Pi via candump - but only for a couple of seconds or so, before it stops receiving completely.

1. Do you think this is related to the 2 buffer mcp2515 limitation you've mentioned before?
2. If so, is it possible to update to the 3.18 version of the kernel on a Model B and fix this problem, or would I have to switch to the Pi 2?
ad 1: 2 buffers will only produce missed packets, but would continue (unless - see below)
ad 2: I guess you use a 3.12 kernel plus the mcp251x driver module. With that setup it can happen that the mcp251x driver is stuck when too many packets arrive in a short time. The 3.18 Kernel does not show this behavior, so I would recommend moving to the latest foundation kernel and use the device-tree-overlays as describes several pages earlier...
Thank you for the reply. Is there a detailed write up anywhere of enabling a SPKang PICAN board using the 3.18 kernel, aside from your post from earlier in the thread? I followed all of the instructions in that post after updating to the latest 3.18 on a Pi B+ and upgrading all of the packages. I'm not sure how to enable the can interface now - could you be kind enough to explain it to me? If you have time, that is. I've never used device-trees, so I'm a bit at a loss.

Re: CAN controller

Posted: Thu Apr 30, 2015 5:24 am
by msperl
The producers of pican have created this page that contains all the steps:
http://skpang.co.uk/blog/archives/1165

The only thing left unmentioned there is how to install the can-tools...
You find it here: http://elinux.org/Can-utils

Re: CAN controller

Posted: Wed May 06, 2015 12:10 am
by paulson440
Hi all, using a RPI2 here, kernel 3.18.12-v7+. I have a MCP2515 wired up and working with my pi here.

I'm developing an application to transfer a large amount of data into the MCP2515 from the CAN bus at 500k, quickly (MCP2515 is receiving data). The application is written in c# and uses platform invoke to call the socket APIs to access SocketCAN. Using the spi and mcp251x drivers from "in the box".

I'm having a really hard time getting a reliable receive stream, packets get dropped like crazy. However, outgoing works flawlessly, I can transfer data at full CAN bus speed without delay or problem. But incoming data - the when, or the how badly the packets get dropped seems random.

Maybe I'm missing something fundamental but shouldn't the RPI2 be capable of pulling the data out of the MCP2515 fast enough, especially with a kernel driver?

From what I've read, it sounds like the "non real time aspect" of Linux prevents interrupt servicing quick enough with the small MCP2515 buffers. Is it possible to build better buffering and optimization into the driver? What about multiple buffer pulls per interrupt?

Also, would using mcp2515a.ko provide any improvements?

Re: CAN controller

Posted: Wed May 06, 2015 6:31 am
by msperl
paulson440 wrote: Maybe I'm missing something fundamental but shouldn't the RPI2 be capable of pulling the data out of the MCP2515 fast enough, especially with a kernel driver?
The problem is that sometimes (especially at times of lots of interrupts because of disk-io and such) the OS sometimes decides to run other processes instead, and this means the process handling the request is NOT running for a 1/100s of a second and if that happens then all those packets that come in during those 1/100s of a second are lost.

On top there are also latency issues that are not predetermined and may vary.

this may also be true in the case of you running the transfer entirely from user-space - a reschedule may always occur resulting in those 1/100s gaps! (actually I have seen the spi thread getting stuck for 86 of such 1/100s cycles - mostly because of heavy io to SD-card)
paulson440 wrote: From what I've read, it sounds like the "non real time aspect" of Linux prevents interrupt servicing quick enough with the small MCP2515 buffers. Is it possible to build better buffering and optimization into the driver? What about multiple buffer pulls per interrupt?
well - that would require a different HW device instead of the mcp2515 to get used, as this limitation of 2 Receive-buffers sits inside the mcp2515 hardware. If there were more buffers say 8, then it would be less of an issue - nothing software can do about that...
paulson440 wrote: Also, would using mcp2515a.ko provide any improvements?
There may be some cases where improvements are a possible, but right now I still focus on getting the spi driver working fast.
The main thing is that it is buggy (Ids are calculated badly) and is not using the latest features.

What gives you extra reliability is switching to a 4.0 kernel, as this has some "features" that improve the response-time

As soon as I finish the spi-stuff I can have another look at the mcp251x driver where to improve (besides assigning 2 more GPIOs to handle the reads slightly faster)

Re: CAN controller

Posted: Wed May 06, 2015 1:38 pm
by paulson440
Thank you for your reply Martin.
The problem is that sometimes (especially at times of lots of interrupts because of disk-io and such) the OS sometimes decides to run other processes instead, and this means the process handling the request is NOT running for a 1/100s of a second and if that happens then all those packets that come in during those 1/100s of a second are lost.
I was thinking about this last night - maybe I can start removing unneeded components from the system, like USB, network, video, disable paging, etc. to reduce interrupt demands.

Could the real time kernel be used to solve some of these issues? Or something more blunt? I mean, this is a quad core CPU, and for what I'm doing, I don't even care if I have to dedicate an entire core to servicing the MCP2515 chip, if it gets me reliable data receives (e.g. spin a thread in the MCP251x ISR forever, if I can even get away with doing something like that).
this may also be true in the case of you running the transfer entirely from user-space - a reschedule may always occur resulting in those 1/100s gaps! (actually I have seen the spi thread getting stuck for 86 of such 1/100s cycles - mostly because of heavy io to SD-card)
Forgive my ignorance, Linux architecture is fairly new to me, but isn't SocketCAN running in the kernel, or at the very least, at a low enough level that it should be able to handle the buffering regardless of how high level the application is?
There may be some cases where improvements are a possible, but right now I still focus on getting the spi driver working fast.
The main thing is that it is buggy (Ids are calculated badly) and is not using the latest features.

What gives you extra reliability is switching to a 4.0 kernel, as this has some "features" that improve the response-time

As soon as I finish the spi-stuff I can have another look at the mcp251x driver where to improve (besides assigning 2 more GPIOs to handle the reads slightly faster)
Thanks Martin, really appreciate your contributions.

Re: CAN controller

Posted: Wed May 06, 2015 6:13 pm
by msperl
paulson440 wrote:Thank you for your reply Martin.
I was thinking about this last night - maybe I can start removing unneeded components from the system, like USB, network, video, disable paging, etc. to reduce interrupt demands.

Could the real time kernel be used to solve some of these issues? Or something more blunt? I mean, this is a quad core CPU, and for what I'm doing, I don't even care if I have to dedicate an entire core to servicing the MCP2515 chip, if it gets me reliable data receives (e.g. spin a thread in the MCP251x ISR forever, if I can even get away with doing something like that).

Forgive my ignorance, Linux architecture is fairly new to me, but isn't SocketCAN running in the kernel, or at the very least, at a low enough level that it should be able to handle the buffering regardless of how high level the application is?
The point is that the kernel "outsources" some tasks into "kernel-threads" - if you are running ps you find those with [].

This is essentially splitting the "quick response required" of a driver (interrupt handler) from the heavy lifting that may be needed to complete the job.

This is part of the philosophy of real-time - putting those tasks into "user-space" where when something important occurs it can get interrupted (as you can not interrupt an interrupt handler (except for a fiq))

And if there is lots of CPU load or lots of USB traffic or traffic to the SD-card, then there are more processes that are running to do "their" thing and these can get rescheduled without affecting performance - more CPU-cores helps here evening out the load.

Re: CAN controller

Posted: Tue May 12, 2015 3:03 pm
by paulson440
Looks like I've found a work-around. I need to isolate what exactly solved it (if not all combinations here), but currently I'm running the RPI2 RT kernel (https://github.com/emlid/linux-rt-rpi), and the DMA enabled SPI driver ( https://github.com/notro/spi-bcm2708). I used the "real time priority for message pump" option in spi-bcm2708. Now my read buffer overflow problems are gone.

Re: CAN controller

Posted: Tue May 12, 2015 4:16 pm
by clouded
msperl wrote:500KHz and lots of CAN-messages getting sent back to back over the bus are still happening to trigger delays with the current framework - but even then you can not make 100% sure you receive every message - it is linux (not hard real-time) and the 2 receive buffers on the mcp251x are too few to make certain NOT to loose even a single message because something else is happening on the system.

Those latest improvements that made it into the foundation 4.0 kernel (and might also make it into 3.18 eventually) improve the situation, but you can not work around the limitation of only 2 packet buffers on the mcp251x.

If you need something more reliable, then you either need to go with a different can-controller that has more buffers (only sja1000 comes to my mind, but I have not tested it) or you need to add some microcontroller with some memory in the middle to act as a buffer (but then you need to write your own driver for the kernel).
Do you think this chip can handle the situation reliably?
Holt HI-3110 SPI CAN controller with integrated transceiver:
http://www.digikey.com/product-detail/e ... ND/3585723
Datasheet states that the buffer is 8 messages, which gives us ~376 microseconds.
3.18 is quite good on latency even without RT_PREEMPT.

Update: I also see a family of CC770 controllers, which can have receiver buffer up to 16 messages. Linux even has drivers for those. Wonder why no one tried them with RPi.

Re: CAN controller

Posted: Tue May 12, 2015 4:54 pm
by msperl
Well - things would get more tolerant with 8 FIFO buffers (giving you 0.376ms in the worsted case of 1MBit and 0-byte can messages, 1.2ms in the best case), but you still can not avoid buffer overflows when the OS is under heavy load.

Also you would need to write a new driver for that chip...
Cost is another thing - the chip is quite expensive!

I am actually thinking about a solution where a mcu would sit in front of the mcp251x (also possibly a MCU with a can controller interface) and cache the data (with say up to 4kB buffer which would be 256 messages - that gives you a buffer of 12ms in the worsted-case of 1MBit an 0-byte can messages and 40.4ms for the best case with 8byte can-messages).
It would also need a driver, but it could be streamlined for efficient communication right from the start...

Re: CAN controller

Posted: Tue May 12, 2015 6:03 pm
by clouded
msperl wrote: Cost is another thing - the chip is quite expensive!
But it has an integrated transceiver, I'd say that compensates its higher price.
msperl wrote: I am actually thinking about a solution where a mcu would sit in front of the mcp251x (also possibly a MCU with a can controller interface) and cache the data (with say up to 4kB buffer which would be 256 messages - that gives you a buffer of 12ms in the worsted-case of 1MBit an 0-byte can messages and 40.4ms for the best case with 8byte can-messages).
It would also need a driver, but it could be streamlined for efficient communication right from the start...
Why use mcp251x if there's an MCU? Some microcontrollers have both can controller and transceiver built-in, like LPC11C00.

I would still prefer to use a standalone chip if there was a proper one, just for simplicity (without the need to flash the mcu etc).
CC770 looks like a good candidate with its 16-message buffer, but seems like its linux driver lacks support for SPI.

Re: CAN controller

Posted: Tue May 12, 2015 6:17 pm
by msperl
If you look at other devices, then you can also look at SJA1000 with 64 byte FIFO, for which there exists a driver in the kernel - but it is a 5V device...

As said: a MCU can run with mcp2515 or with an internal CAN-controller (whatever technology).

The price point is 2.5Eu for the mcp25625 (= mcp2515 + mcp2561) - so it is the cheapest you can get...

Re: CAN controller

Posted: Tue May 12, 2015 7:04 pm
by clouded
msperl wrote:If you look at other devices, then you can also look at SJA1000 with 64 byte FIFO, for which there exists a driver in the kernel - but it is a 5V device...
For me the biggest problem with it is the parallel bus. SPI is just much more convenient.
msperl wrote: As said: a MCU can run with mcp2515 or with an internal CAN-controller (whatever technology).

The price point is 2.5Eu for the mcp25625 (= mcp2515 + mcp2561) - so it is the cheapest you can get...
Btw, I've seen LPCs with integrated CAN controller and transceiver as low as $1.6 for 1 piece.

Re: CAN controller

Posted: Tue May 12, 2015 8:51 pm
by skpang
The LPC11C22 is £2.30 so it is comparable to MCP2515+MCP2561. The LPC11C22 can do all the CAN buffering then talk to the Pi via SPI.

Would anyone like to write a driver for the Pi and firmware for the LPC11C22 if I develop a board for the Pi?

skpang
PiCAN Designer

Re: CAN controller

Posted: Wed May 13, 2015 6:30 am
by msperl
I would recommend first testing with this: http://www.nxp.com/demoboard/OM13012.html
to see if it works and only then I would look into the HW portion.

One thing is that I have no Idea about the LCP11CXX and how to program those (I am more used to Atmel chips)
and would require quite a learning-curve.

A streaming linux-driver should be feasible - it is mostly about "protocol" to keep unnecessary latencies out of the picture...

I actually thought of starting with an atmel+mcp version first and then someone could implement other HW + corresponding firmware...

Re: CAN controller

Posted: Wed May 13, 2015 1:35 pm
by skpang
Just ordered the LPCXpresso demo board. Let me see if I can get the basic CAN part working and go on from there.

It doesn't have to be the LPC11C24 but it is a good price MCU.

Re: CAN controller

Posted: Wed May 13, 2015 1:49 pm
by msperl
Well - maybe a STM32F103 is also a good candidate (slightly more expensive)

It comes with 20Kb SRAM, so we could even keep the firmware (besides a boot-loader) in ram and upload it via the kernel firmware APIs from /lib/firmware/...
This makes updating the firmware cheap...

Got one of those boards on my desk now (ordered it long ago, but never did anything with it) so I may investigate this approach as well, but I am still busy with further improvements to spi in general... (some of which would also benefit the "streaming" approach).

Re: CAN controller

Posted: Wed May 13, 2015 9:13 pm
by paulson440
paulson440 wrote:Looks like I've found a work-around. I need to isolate what exactly solved it (if not all combinations here), but currently I'm running the RPI2 RT kernel (https://github.com/emlid/linux-rt-rpi), and the DMA enabled SPI driver ( https://github.com/notro/spi-bcm2708). I used the "real time priority for message pump" option in spi-bcm2708. Now my read buffer overflow problems are gone.
After more testing, I've confirmed the RT kernel plus EITHER setting the realtime=1 kernel module option when loading spi-bcm2708, OR raising the priority of the kernel worker thread for the spi driver (for mcp251x) to SCHED_FIFO and highest priority (essentially what realtime=1 does) is required to fix the buffer overflows.

Re: CAN controller

Posted: Thu May 14, 2015 10:22 am
by msperl
actually I recommend using the spi-bcm2835 driver now with the latest kernel provided by the foundation

All you have to do is switch by adding:

Code: Select all

dtoverlay=spi-bcm2835-overlay
to /boot/config.txt and then load spi-bcm2835 instead of spi-bcm2708.

a future 4.0 (or later) kernel should give even better results!

Re: CAN controller

Posted: Thu May 14, 2015 9:31 pm
by skpang
Got the basic CAN sending and receiving working on the LPC11C24. Using the LPCXpresso IDE seems to be straight forward and the CAN examples works fine. Going to do some high speed buffer test next.


Image

Re: CAN controller

Posted: Fri May 15, 2015 7:22 am
by msperl
@skpang: Well - maybe we should move this to a different thread?

But I got some "basic" protocol ideas that would make things "easy" and a very efficient streaming protocol:

The principles are:
* we only run one spi message to handle the transfers in and out
* almost everything is a CAN message when talking to the bus
* we can read more data than what is in the queue without negative impact

Here an Idea of the message structure...

Code: Select all

struct can_device_frame {
	u8 len;
#define CAN_DEVICE_LEN_VALID_DATA 	BIT(7)
#define CAN_DEVICE_LEN_MASK		(CAN_DEVICE_LEN_VALID_DATA - 1)
#define CAN_DEVICE_LEN(len)		(len | CAN_DEVICE_LEN_VALID_DATA)

	/* can_id as defined in <linux/can.h> in network byte order*/
	canid_t canid_net;

	/* the data */
	u8 data[8];
};

struct can_device_write {
	u8 cmd;
#define CAN_DEVICE_CMD_READ		0
#define CAN_DEVICE_CMD_TRANSMIT		0x40
#define CAN_DEVICE_CMD_RESET		0xC0

	struct can_device_frame frame;
};

struct can_device_read {
	u8 status;
#define CAN_DEVICE_STATUS_TX_QUEUE_FREE	BIT(7)
#define CAN_DEVICE_STATUS_RESERVED1	BIT(6)
#define CAN_DEVICE_STATUS_RESERVED2	BIT(5)
#define CAN_DEVICE_STATUS_RX_QUEUED_MASK (BITS(5) - 1)
#define CAN_DEVICE_STATUS_RX_QUEUED(queuesize)	\
	(queuesize & CAN_DEVICE_STATUS_RX_QUEUED_MASK)

	struct can_device_frame frame;
};

Note that you can chain those "can_device_writes" back to back without CS asserted/deasserted.
but each time the command has to get sent every 14 bytes - it could even switch (but I would not implement that).
So we could "in the blind" transfer out 225 (=16*14+1) zero bytes and we would get up to 16 messages from the buffer in one go.
For each we have to check the received len for CAN_DEVICE_LEN_VALID_DATA to see if it really contains data

Note that in principle we can also handle those CAN FD frames - we would just need to chain 5 of those messages to get to 64 bytes - the first frame contains all the information about the ids and length as well as 8 bytes and the subsequent 4 frames contain the payload.

Error/status handling should get handled by the device and it should "submit" can_messages into the queue for retrieval.
The driver will need to filter out those that are not of interrest to the outside (like BER counters).

Configuration I would handle by sending "specially crafted" can messages to configure CAN speeds and operation modes.
(specially crafted means addresses with CAN_RTR_FLAG, CAN_ERR_FLAG and CAN_EFF_FLAG, which is not expected by the kernel) - this also applies to internal status messages.

Same for firmware updates getting sent in chunks of 8 bytes.

The interrupt line should be asserted while there are messages in the queue...

A message from the queue is "consumed" 14 bytes have been shifted out and this would trigger a interrupt deasserted when the queue is empty.

After a "reset" some identification data should be in the can-buffer and interrupt should be asserted.
This should be generic enough that the driver can get used for other devices as well - say newer versions that may need different firmware.
This would probably be something along the lines of:
* n messages with 8 byte of the device name
* 1 messages with 8 byte of the device firmware info (string)
* 1 message with capabilities

If we reach buffer full we need to send a "special" error message indicating buffer full (possibly also a message for "almost" full, which gets altered to message full if a message arrives since then).

There is NO definition of how big the queues may _actually_ be inside the device - we just indicate that we can have up to 31 messages pending ready for read...

There are probably a few things left out, but it should serve as a basis.

Re: CAN controller

Posted: Fri May 15, 2015 12:16 pm
by clouded
@skpang @msperl

I'd like to suggest another approach - slcan-compatible firmware for LPC.
So basically it would be: set high UART speed on Raspberry (it can handle 4mbps and even more), connect LPC over UART to Raspberry, set up slcan on /dev/ttyAMA0.

Also, if someone would prefer SPI - it is quite easy to turn SPI into serial-like interface on both ends.

Re: CAN controller

Posted: Fri May 15, 2015 12:52 pm
by msperl
Seen slcan - IMO the problem is that:

a) you loose the serial port on the pi (but you could run that over USB)
b) you are limited to low transfer speeds of the UART on the RPI (4MBit max if I remember - and that was not always stable)
c) you are limited by the text protocol of slcan that essentially wastes half of your bandwidth for representing everything in text form 14 bytes binary versus more than 26 bytes (=4*2+1+8*2+1) - so if you run 4MBaud you get effectively 2MBaud of effective bandwidth (and then you still have 10 bit versus 8 not factored in)
d) the UART on the PI is totally interrupt driven in linux adding back lots of interrupts due to the asynchronous transfers.

Re: CAN controller

Posted: Fri May 15, 2015 1:09 pm
by msperl
if you look at the exact format for slcan encapsulation:
<type> <id> <dlc> <data>*

so you get: 1(type)+4*2(id)+1(dlc)+2*8(data)+1("\r")
so 27 bytes.

Re: CAN controller

Posted: Fri May 15, 2015 3:45 pm
by clouded
msperl wrote:Seen slcan - IMO the problem is that:

a) you loose the serial port on the pi (but you could run that over USB)
b) you are limited to low transfer speeds of the UART on the RPI (4MBit max if I remember - and that was not always stable)
c) you are limited by the text protocol of slcan that essentially wastes half of your bandwidth for representing everything in text form 14 bytes binary versus more than 26 bytes (=4*2+1+8*2+1) - so if you run 4MBaud you get effectively 2MBaud of effective bandwidth (and then you still have 10 bit versus 8 not factored in)
d) the UART on the PI is totally interrupt driven in linux adding back lots of interrupts due to the asynchronous transfers.
I know about the overhead, just suggesting it as an almost ready to use solution.
SLCAN can can be used over SPI with minor modifications, utility that would convert SPI port into /dev/virtualserial would take only 50 lines of code. That way UART port can be saved.