Understanding USRP2 flow control

Posted by OE1RSA (Guest) on 2010-07-18 18:52

Dear All!

I am starting to learn my new USRP2 device. I bought a reasonable fast quad core PC and a recommended ethernet board.

I wanted to find out if I can get the interpolation 4 working in my environment. I just connect a sine oszillator to the usrp2 sink.

I am observing the signal on a connected scope.

Everything works fine for a while, but then suddenly the output becomes heavily distorted.

I can see from the resource watch of the system monitor that my PC putting out data at a sustained rate of more than 800 Mbits/sec.

I started to try to understand the code in gnuradio/usrp2/host/lib and expected some flow control when the usrp2 input buffers are nearly full . Unfortunately I was not able to understand how this is implemented.

Is there any flow control at all? If not, what is supposed to happen, if the PC is sending (slightly) too fast?

Appreciating any information on this subject,

Roland

--

_________________________________________

_ _ | Roland Schwarz OE1RSA

|_)(_ | sip:speedsnail@ekiga.net

| \__) | mailto:oe1rsa@oevsv.at

________| http://www.blackspace.at

Re: Understanding USRP2 flow control

Posted by Eric Blossom (Guest) on 2010-07-19 22:08

On Sun, Jul 18, 2010 at 06:50:47PM +0200, OE1RSA wrote:

Flow control is handled at the link level using ethernet PAUSE frames.

Be sure to enable real-time scheduling by calling

before starting or running the flow graph.

If you haven't already, ensure that /etc/security/limits.conf contains this line:

and that you are a member of group "usrp".

Eric

Re: Understanding USRP2 flow control

Posted by OE1RSA (Guest) on 2010-07-20 10:27

Thank you for your answer.

Am 2010-07-19 22:06, schrieb Eric Blossom:

I was not aware of this ethernet "feature". Having learned something new.

Made everyting as told, but still no success. Another observation I have made:

I am using two kernels from the ubuntu 10.04 LTS distro:

It is interesting that the generic kernel does better than the rt kernel. I.e. the rt is practically unusable at interpolation 4. The generic does quite well for a while, then suddenly drops off.

Btw.: I am using the e1000e driver for intel cards.

Possibly it may be an issue with the driver?

Do you have an idea how I could find out whether my problems are because of buffer overrun or underrun?

Regards Roland

--

Re: Understanding USRP2 flow control

Posted by OE1RSA (Guest) on 2010-07-23 10:37

Hi Eric,

it took me some time to assemble the RS 232 level shifter.

First to explicitly answer your question about realtime scheduling. Yes I did put in the explicit enabling you have told me, and yes, I get no error, i.e. realtime scheduling should be set succesfully.

2010-07-20 20:18, Blossom:

I tried to tune my BIOS settings, but this did not change the behaviour substantly.

Yes, this is how it looks on my machine.

I also experimentally set it to

to see, if it would change something, but as you guessed, it did not improve the situation.

> On transmit, it'll be a buffer underrun. The host isn't keeping up.

I can now confirm that I am seeing bursts of 'UUU...' on the debug port of the USRP2.

Some more observations:

1) The 6.31 rt kernel from the ubuntu 10.04 LTS is worse than the default stock 6.32 kernel.

2) Altough the USRP2 is signaling underuns ( I guess this is what U stands for) at the same time I can see in wireshark that I am receiving PAUSE frames. This looks like there might be some buffer size problems. How large are the transmit buffers of the USRP2?

3) I wrote a small test app that blasts out UDP frames on the same interface as fast as it can, and I can see a sustained rate of approx 900Mbits/ses. So I guess it is not a limitation of my hardware.

Do you have any other ideas what else I could try?

Turning to receive side: I invoked usrp2_fft.py -e eth2 -d 4 and I can see not see 'S' on the terminal. Am I correct in the assumption this means I have no missed packets on receive?

Thank you for your help!

Roland

--

Re: Understanding USRP2 flow control

Posted by Eric Blossom (Guest) on 2010-07-23 21:21

On Fri, Jul 23, 2010 at 10:35:12AM +0200, OE1RSA wrote:

Roland, all of the stuff below looks good.

Sorry, this may have been in an earlier message, but what ethernet chip/card are you using? (lspci will show the details.) We've had lots of problems with RealTek.

What kind of CPU are you running on?

In my exerience, it takes at least a Core 2 Duo running at 3+ GHz to transmit at full speed. The challenge is that there's not much buffer in the  USRP2, and thus the average Tx rate over a very small window needs to be rock solid. If the cpu gets side-tracked doing something else,  you'll see underruns.

Also, on some machines with multiple CPU sockets, such as my dual quad core Xeon, on some versions of the OS, it really matters as to whether or not all of the computation is running on a single physical chip. If you've got a machine like that, try using numactl to force all thethreads to live on a single physical chip (not core). If you look at the output of cat /proc/cpuinfo, "physical id" indicates which cores are in which sockets. (How the processors are assigned numbers seems to have a random component (at least under some versions of the OS))

Yes.

Results:

References: [ Understanding: 778 ] [ USRP2: 3754 ] [ flow: 1387 ] [ control: 1901 ]

Total 36 documents matching your query.

1. Re: [Discuss-gnuradio] Understanding USRP2 flow control (score: 586)

Date: Mon, 19 Jul 2010 13:06:23 -0700

Flow control is handled at the link level using ethernet PAUSE frames. Be sure to enable real-time scheduling by calling if gr.enable_realtime_scheduling() != gr.RT_OK: print "Note: failed to enable

/archive/html/discuss-gnuradio/2010-07/msg00342.html (6,017 bytes)

2. Re: [Discuss-gnuradio] Understanding USRP2 flow control (score: 576)

Date: Tue, 20 Jul 2010 11:18:43 -0700

Just to be clear, do you get the "Note: failed to enable realtime scheduling" or not. I'm assuming that you're not seeing this. That driver should be fine. If your machine has any kind of power contr

/archive/html/discuss-gnuradio/2010-07/msg00368.html (7,257 bytes)

3. [Discuss-gnuradio] Understanding USRP2 flow control (score: 573)

Date: Sun, 18 Jul 2010 18:50:47 +0200

Dear All! I am starting to learn my new USRP2 device. I bought a reasonable fast quad core PC and a recommended ethernet board. I wanted to find out if I can get the interpolation 4 working in my env

/archive/html/discuss-gnuradio/2010-07/msg00294.html (4,802 bytes)

4. Re: [Discuss-gnuradio] Understanding USRP2 flow control (score: 569)

Date: Fri, 23 Jul 2010 10:35:12 +0200

Hi Eric, it took me some time to assemble the RS 232 level shifter. First to explicitly answer your question about realtime scheduling. Yes I did put in the explicit enabling you have told me, and ye

/archive/html/discuss-gnuradio/2010-07/msg00427.html (7,337 bytes)

5. Re: [Discuss-gnuradio] Understanding USRP2 flow control (score: 556)

Date: Fri, 23 Jul 2010 12:18:17 -0700

Roland, all of the stuff below looks good. What kind of CPU are you running on? In my exerience, it takes at least a Core 2 Duo running at 3+ GHz to transmit at full speed. The challenge is that ther

/archive/html/discuss-gnuradio/2010-07/msg00444.html (7,914 bytes)

6. Re: [Discuss-gnuradio] Understanding USRP2 flow control (score: 556)

Date: Tue, 20 Jul 2010 10:24:51 +0200

Thank you for your answer. Am 2010-07-19 22:06, schrieb Eric Blossom: I was not aware of this ethernet "feature". Having learned something new. Made everyting as told, but still no success. Another o

/archive/html/discuss-gnuradio/2010-07/msg00357.html (6,094 bytes)

7. Re: [Discuss-gnuradio] Understanding flow control (score: 534)

Date: Thu, 14 Jan 2010 22:11:38 -0500

Following up on my previous email, thinking about this some more: I'm guessing that we are sending the USRP2 more data than it can handle, it is sending pause packets back, which when RX is ON, the e

/archive/html/discuss-gnuradio/2010-01/msg00177.html (9,796 bytes)

8. Re: [Discuss-gnuradio] Understanding flow control (score: 517)

Date: Thu, 14 Jan 2010 19:55:16 -0500

That's very interesting... just this morning we were playing with turning rx OFF with ethtool and convincing ourselves that it seemed to stabilize our system. If we turn off RX pause (ethtool -A eth0

/archive/html/discuss-gnuradio/2010-01/msg00176.html (8,068 bytes)

9. Re: [Discuss-gnuradio] Understanding flow control (score: 512)

Date: Sat, 16 Jan 2010 12:03:28 -0500

Hi Matt, I'll try hooking up my ttl line on Monday (I have done that but not with our current configuration, so far only when I was playing around with building custom firmware). Here's the official

/archive/html/discuss-gnuradio/2010-01/msg00221.html (11,614 bytes)

10. Re: [Discuss-gnuradio] Understanding flow control (score: 508)

Date: Thu, 14 Jan 2010 16:18:06 -0800

The USRP2 only applies backpressure to data transmitted from the host. It does this by transmitting PAUSE frames, which timeout after a certain period of time (see the the Gig E spec for details). In

/archive/html/discuss-gnuradio/2010-01/msg00174.html (6,402 bytes)

11. Re: [Discuss-gnuradio] Understanding flow control (score: 506)

Date: Fri, 15 Jan 2010 13:54:24 -0500

We've tried it with and without a switch - definitely better without the switch. Thinking about our setup the behavior actually makes sense to me, although I'm waiting to discuss it with my signal pr

/archive/html/discuss-gnuradio/2010-01/msg00192.html (8,253 bytes)

12. Re: [Discuss-gnuradio] Understanding flow control (score: 494)

Date: Fri, 15 Jan 2010 21:03:14 -0600 (CST)

"Clock the host" at multi-MHz rate? One way would be to connect the A/D converter directly to the PC and omit external radio hardware. Then you would not need FPGA de-modulation, GbE, etc. That woul

/archive/html/discuss-gnuradio/2010-01/msg00211.html (11,906 bytes)

13. Re: [Discuss-gnuradio] Understanding flow control (score: 494)

Date: Fri, 15 Jan 2010 18:53:09 -0500

Thanks Matt, Eric and Jonathan (hope I didn't forget anyone. :-) ). We greatly appreciate the information and need to think about stuff on our end. I've been deliberately vague about our application

/archive/html/discuss-gnuradio/2010-01/msg00206.html (10,777 bytes)

14. Re: [Discuss-gnuradio] Understanding flow control (score: 491)

Date: Fri, 15 Jan 2010 15:24:24 -0800

On 01/15/2010 03:08 PM, Tom Gross wrote: yes of course we have two separate gig-e cards. if the usrp2 is sending us "pause" commands then it seems evident the usrp2 is having trouble keeping up, not

/archive/html/discuss-gnuradio/2010-01/msg00204.html (9,948 bytes)

15. Re: [Discuss-gnuradio] Understanding flow control (score: 487)

Date: Fri, 15 Jan 2010 18:21:34 -0800

On 01/15/2010 03:53 PM, Tom Gross wrote: Thanks Matt, Eric and Jonathan (hope I didn't forget anyone. :-) ). We greatly appreciate the information and need to think about stuff on our end. I've been

/archive/html/discuss-gnuradio/2010-01/msg00210.html (9,175 bytes)

16. Re: [Discuss-gnuradio] Understanding flow control (score: 476)

Date: Fri, 15 Jan 2010 10:12:49 -0800

Tom, are there any switches between the host and the USRP2? If so, try removing them. PAUSE frames and switches don't interact well. I'm not sure I'm following the physical interconnection of the pie

/archive/html/discuss-gnuradio/2010-01/msg00190.html (6,289 bytes)

17. [Discuss-gnuradio] Understanding flow control (score: 476)

Date: Thu, 14 Jan 2010 14:13:01 -0500

I'm trying to understand the flow control between the USRP2 and host machine. I assume that it needs to be worked out where the USRP2 will always have a constant stream of uninterrupted radio data w

/archive/html/discuss-gnuradio/2010-01/msg00163.html (4,678 bytes)

18. Re: [Discuss-gnuradio] Understanding flow control (score: 474)

Date: Sat, 16 Jan 2010 11:09:04 -0800

On 01/16/2010 09:03 AM, Tom Gross wrote: Hi Matt, I'll try hooking up my ttl line on Monday (I have done that but not with our current configuration, so far only when I was playing around with buildi

/archive/html/discuss-gnuradio/2010-01/msg00228.html (10,244 bytes)

19. Re: [Discuss-gnuradio] Understanding flow control (score: 469)

Date: Fri, 15 Jan 2010 14:47:22 -0500

Thanks Eric, that's exactly what I thought. I think in our application it's probably better to drop data, or at least that's why things are more stable for us when RX is OFF. Or maybe we should just

/archive/html/discuss-gnuradio/2010-01/msg00197.html (7,839 bytes)

20. Re: [Discuss-gnuradio] Understanding flow control (score: 469)

Date: Fri, 15 Jan 2010 11:07:10 -0800

Actually, PAUSE handling is all handled in the FPGA. When the FIFO is getting full, a PAUSE frame is sent on the wire telling the host to stop sending for a while. In ethtool lingo, "Rx ON" means tha

/archive/html/discuss-gnuradio/2010-01/msg00193.html (7,235 bytes)

21. Re: [Discuss-gnuradio] Understanding flow control (score: 464)

Date: Mon, 18 Jan 2010 13:05:51 -0500

We are using an LFRx and an LFTx. We would absolutely love to have a version of the USRP2 firmware and host software that can read both receiver channels, as we can on the USRP1. I've been trying for

/archive/html/discuss-gnuradio/2010-01/msg00271.html (8,561 bytes)

22. Re: [Discuss-gnuradio] Understanding flow control (score: 464)

Date: Fri, 15 Jan 2010 15:18:22 -0800

The host software, when creating a data stream to be sent to the USRP2 for TX, will create the data as fast as the processor allows, and TX on the GbE at full wire rate. The USRP2, however, is "consu

/archive/html/discuss-gnuradio/2010-01/msg00203.html (7,734 bytes)

23. Re: [Discuss-gnuradio] Understanding flow control (score: 455)

Date: Fri, 15 Jan 2010 18:08:34 -0500

yes of course we have two separate gig-e cards. if the usrp2 is sending us "pause" commands then it seems evident the usrp2 is having trouble keeping up, not the computer.

/archive/html/discuss-gnuradio/2010-01/msg00201.html (7,764 bytes)

24. Re: [Discuss-gnuradio] Understanding flow control (score: 452)

Date: Fri, 15 Jan 2010 15:26:17 -0800

What is the maximum data rate that the USRP2 transmitter can accept from the host without firing pause signals back to the host? See my other message. The USRP2 will ALWAYS put out pause frames. In f

/archive/html/discuss-gnuradio/2010-01/msg00205.html (6,879 bytes)

25. Re: [Discuss-gnuradio] Understanding flow control (score: 452)

Date: Fri, 15 Jan 2010 18:14:00 -0500

What is the maximum data rate that the USRP2 transmitter can accept from the host without firing pause signals back to the host? -Tom

/archive/html/discuss-gnuradio/2010-01/msg00202.html (7,230 bytes)

26. Re: [Discuss-gnuradio] Understanding flow control (score: 449)

Date: Fri, 15 Jan 2010 15:04:15 -0800

Incidentally my System Engineer/Project Lead points out that if the USRP2 is actually telling the host to stop sending (which certainly appears to be the case) then we are only able to get overall th

/archive/html/discuss-gnuradio/2010-01/msg00200.html (7,604 bytes)

27. Re: [Discuss-gnuradio] Understanding flow control (score: 449)

Date: Fri, 15 Jan 2010 17:52:49 -0500

Incidentally my System Engineer/Project Lead points out that if the USRP2 is actually telling the host to stop sending (which certainly appears to be the case) then we are only able to get overall th

/archive/html/discuss-gnuradio/2010-01/msg00199.html (6,035 bytes)

28. Re: [Discuss-gnuradio] Understanding Data Flow in the USRP2 Firmware (score: 427)

Date: Fri, 23 Jul 2010 12:48:59 -0700

txrx.c is the top-level C file. memory_map.h describes the various peripherals that the firmware sees. Most of the magic occurs in the buffer pool which can simultaneously accept input from 4 sources

/archive/html/discuss-gnuradio/2010-07/msg00448.html (5,000 bytes)

29. [Discuss-gnuradio] Understanding Data Flow in the USRP2 Firmware (score: 427)

Date: Fri, 23 Jul 2010 12:31:19 -0700

Hi, I am trying to understand the data flow in the FPGA. I am trying to understand the Architecture and aemb firmware program control flow. Which one is the top level firmware "C" file. How it gets d

/archive/html/discuss-gnuradio/2010-07/msg00446.html (4,500 bytes)

30. Re: [Discuss-gnuradio] UHD Problem...usrp2 no control response (score: 312)

Date: Fri, 12 Nov 2010 16:14:58 -0800

The no control response might mean the pause frames are messing up the control packets. Does this happen in a receive only application? One way to alleviate this would be to run the flow control bran

/archive/html/discuss-gnuradio/2010-11/msg00246.html (6,863 bytes)

31. Re: Fwd: Re: [Discuss-gnuradio] USRP2, is that possible to skip the Ethernet and pass data through general purpose (physically accessible) inputs to the FPGA? (score: 255)

Date: Fri, 29 Oct 2010 11:44:12 -0700

Please look here for schematics. http://www.ettus.com/download Nick

/archive/html/discuss-gnuradio/2010-10/msg00531.html (177,815 bytes)

32. Fwd: Re: [Discuss-gnuradio] USRP2, is that possible to skip the Ethernet and pass data through general purpose (physically accessible) inputs to the FPGA? (score: 255)

Date: Fri, 29 Oct 2010 10:36:19 -0600

Can anybody help me with this? I need a full schematics of RFX2400, the one available on gnuradio's site is not complete! -- Original Message -- Subject: Re: [Discuss-gnuradio] USRP2, is that possibl

/archive/html/discuss-gnuradio/2010-10/msg00523.html (136,827 bytes)

33. [Discuss-gnuradio] Help finding usrp2 verilog files. (score: 244)

Date: Wed, 21 Jul 2010 23:08:51 +0530

Hi everyone, �� Need a bit help in finding the usrp2 verilog files file for example:� ddc and duc files.I have been searching many sites for the usrp2 verilog file but in vain.Could any help me out i

/archive/html/discuss-gnuradio/2010-07/msg00389.html (70,494 bytes)

34. RE: [Discuss-gnuradio] UHD problem (score: 39)

Date: Fri, 16 Jul 2010 10:50:56 +0000

I changed the code in "usrp2_iface_impl::ctrl_send_and_recv" slighly, see below. It seems to solve the problem for my current application New and removed lines are marked with "PZ". BR/ Per usrp2_ctr

/archive/html/discuss-gnuradio/2010-07/msg00256.html (8,686 bytes)

35. RE: [Discuss-gnuradio] UHD problem (score: 32)

Date: Thu, 15 Jul 2010 20:03:43 +0000

Two questions: How large is the fpga buffer ? Does this imply that it would be fairly straightforward for a FPGA-skilled person (i.e. not myself) to implement decimation=1 ? I mean, not continous str

/archive/html/discuss-gnuradio/2010-07/msg00229.html (6,673 bytes)

36. Re: [Discuss-gnuradio] UHD problem (score: 32)

Date: Thu, 15 Jul 2010 12:40:55 -0700

You have done nothing wrong. This is a result of the transmit SM in the FPGA using ethernet pause frames to flow control the transmit. The pause frame is blocking the transaction from the issue strea

/archive/html/discuss-gnuradio/2010-07/msg00228.html (5,735 bytes)

Current List: 1 - 36