Playing with rpmsg on iMX7d

Lately it has been rare that I get to work on some kernel related problems. One of such task was to get “rpmsg” sample modules running on custom iMX7d board. There are many tutorials available, from NXP as well as from other vendors building iMX7d based SoM. So I was a bit confident to get it running pretty soon. But it ended up a bit more complicated than I thought.

First of all, “rpmsg” is a framework which allows main CPU to communicate with another CPU in a system. For example, iMX7d SoC has 2 cores of Cortex-A7 and one Cortex-M4. If you want to establish a communication channel between Cortex-A and Cortex-M, “rpmsg” framework is handy. The framework is based on “virtio” subsystem which I am not much familiar with at the moment. However, I browsed quite some code while debugging the problem that I was running into.

So, as I said, my task was to run a simple rpmsg ping-pong application (or a driver module). The ping-pong application has two parts: 1) Running on Cortex-M and 2) Running on Cortex-A. The software part on Cortex-A is a Linux kernel module. The software part on Cortex-M can either be a bare-metal software code compiled for Cortex-M or a full fledged FreeRTOS code (the entire toolchain for Cortex-M4 with FreeRTOS and demo examples are available from NXP). I used the FreeRTOS based demo example code.

The flow would be something like this:

  1. Initialize the Cortex-M4 before booting linux kernel (i.e. can be done from u-boot)
  2. Boot up Linux kernel
  3. Load ping-pong module from Linux user-space.

As soon as the module is loaded from user-space, the rpmsg communication channel will be initialized and sample data transfer will be started between Cortex-A and Cortex-M.

These looked pretty easy since most of the stuff is readily available (i.e. Cortex-M application, rpmsg framework and ping pong driver in linux kernel etc). I enabled the rpmsg and ping-pong module in linux kernel, and loaded the modules from user-space and boom, nothing on console or on dmesg. Then I figured that I missed enabling the “rpmsg” device node in DTS. I enabled it and rebooted the kernel, wow, kernel did not even boot. The usual debugging started by enabling the debugs (early printk) and for early init calls.  The culprit was rpmsg platform driver for imx (arch/arm/mach-imx/imx_rpmsg.c). The platform driver maps a region of address range which is used as a shared memory between Cortex-M and Cortex-A. So next logical step was to check that how does that not fail for iMX7D sabresd board (and boards from other vendors). Then I found that there is a dedicated device-tree for Cortex-M4. I could notice that in that device tree file, the usable memory (RAM) was reduced, and some portion of that left out memory was used as shared memory between Cortex-A and Cortex-M. I tried to make similar change on the device tree file for our platform and the crash was gone. But now I could see the warning (kernel trace caused by WARN_ON) pointing to a failure in ioremap, which was obvious since I was using wrong memory range in imx rpmsg platform driver. So I made following change:

--- a/arch/arm/mach-imx/imx_rpmsg.c
+++ b/arch/arm/mach-imx/imx_rpmsg.c
@@ -290,8 +290,8 @@ static int imx_rpmsg_probe(struct platform_device *pdev)
                        ret |= of_device_is_compatible(np, "fsl,imx6sx-rpmsg");
                        if (ret) {
                                /* hardcodes here now. */
-                               rpdev->vring[0] = 0xBFFF0000;
-                               rpdev->vring[1] = 0xBFFF8000;
+                               rpdev->vring[0] = 0x9FFF0000;
+                               rpdev->vring[1] = 0x9FFF8000;
                        }
                } else {
                        break;

With this change, the warning was gone, and rpmsg driver was successfully registered as I could see in the dmesg. However, the loading ping-pong module still did not initialize the communication between two cores. I wondered in disbelief and checked the sysfs entries to see if rpmsg device is present or not. I found that rpmsg was registered as a bus driver as well as a device under virtio bus. So I could see a device listing under virtio bus, but no device under rpmsg bus. By looking into code somewhat deeper, I could see that a device under rpmsg bus is registered when an rpmsg endpoint is created and an initial communication is established successfully (by NS Announcement message exchange). This happens in rpmsg_create_channel() routine which is invoked within a callback registered while creating an endpoint. So, the problem was now about the missing communication message between M4 and A7. This took me a while to figure out, but I was a bit relieved that I am not missing any patch in kernel which might have caused missing device registration.

Wandering here and there in code for a while, I thought of looking at the code of FreeRTOS example and thought may be something needs to be changed there, related to memory mapping. And bingo, I had to adjust the memory mapping which I did change in kernel side but not on FreeRTOS side. The memory addresses are hard-coded and should be changed in middleware/multicore/open-amp/porting/imx7d_m4/platform_info.c file (VRING0_BASE and VRING1_BASE). With this change, I rebuilt the example binary, loaded it via uboot, rebooted linux kernel and wow, I could see an initial message exchange, but the entire buffer was filled with 0x00 to my surprise. I had no clue why this was the case, and then I wasted almost my entire day in putting debug prints and adding more debug code in rpmsg, virtio, virtio-ring drivers, in FreeRTOS middleware and platform driver but no luck. With extreme frustration, I decided to make one last change in device-tree file. Our platform has 512 MB of RAM (starting from address range 0x80000000). Since I am using 0x9FFF0000 as a starting range for shared buffer between the cores, I had initially limited the usable memory like this:

+/ {
+       memory {
+               linux,usable-memory = <0x80000000 0x10000000>;
+       };
+};

Logically I thought this should work since, I am mocking Linux MM that system has 256 MB of RAM than 512 MB. And I could safely ioremap address starting from 0x9FFF0000. But to my surprise this never worked. I had to change the mapping to following to get the example running (which was a bit satisfying and a bit discouraging since I wasted entire day):

+/ {
+       memory {
+               linux,usable-memory = <0x80000000 0x1FF00000>;
+       };
+};

This is still a puzzle to me, but I plan to post a question on NXP community to get this answered. If I get the answer, I will update the post.

Overall a good exercise of code browsing and learning something new about rpmsg and virtio framework.

Happy debugging.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s