Padding in sFLow packets about sflowtool HOT 46 CLOSED

I think you could relax this check:

static void lengthCheck(SFSample *sample, char *description, uint8_t *start, int len) {
  uint32_t actualLen = (uint8_t *)sample->datap - start;
  uint32_t adjustedLen = ((len + 3) >> 2) << 2;
  if(actualLen != adjustedLen) {
    fprintf(ERROUT, "%s length error (expected %d, found %d)\n", description, len, actualLen);
    SFABORT(sample, SF_ABORT_LENGTH_ERROR);
  }
}

And check case only when difference between adjustedLen and actualLen become negative.

from sflowtool.

from sflowtool.

@AdamVocks could you share pcap dump?

from sflowtool.

capture.pcap.txt

from sflowtool.

@AdamVocks thank you! I see padding for first packet in this dump (sflow seq 908969841) for last flow with seq 2421024819.

Fortunately this issue affects only flows at the end of packet. That's why I guess this behaviour could be related with some sort of padding.

from sflowtool.

@sflow Any recommendations? Thanks!

from sflowtool.

from sflowtool.

Thanks, I'll forward this on to Brocade.

from sflowtool.

@AdamVocks @sflow is this a sflowtool issue? I am also seeing the same issue in 6610.
Is this issue fixed?

from sflowtool.

If you want to send me a binary pcap trace of the sFlow from your 6610 then I'd be happy to check it for you. For example, if the switch is sending with ip source 10.1.2.3:

sudo /usr/sbin/tcpdump -i any -s 0 -w 6610.pcap udp port 6343 and ip src 10.1.2.3

from sflowtool.

@sflow - i dont have access to the lab. Its the same issue mentioned above by @pavel-odintsov @AdamVocks . Same error " flow_sample length error (expected 284, found 280) caught exception: 3
0:1002 dstSubnetMask 0" is seen in ICX 6610
Thats why wanted to check with you what was the resolution in above issue.
Is it a sflowtool issue?

from sflowtool.

Problem seen only in sflow version 5

from sflowtool.

I believe it's an issue with the XDR encoding at source (i.e. a Brocade issue).

from sflowtool.

I do not this that is sflowtool issue. Sflowtool is doing right things here. Brocade need to change their approach for padding.

from sflowtool.

ok thanks ! did we raise an issue with brocade?
did they fix it

from sflowtool.

I reported this issue to Brocade. Will try to ask them to provide reference number for tracking issue.

from sflowtool.

@pavel-odintsov - Can u tell us how did u reproduce the issue? What kind of traffic you are seeing this ?
We tried to reproduce in lab and we are not seeing this issue

from sflowtool.

@sflow @pavel-odintsov -one more question we see this error only when we use sflowtool. We dont see the error when we open the pcap in wire shark. What could be the reason?
Also were you able to see the isuue in wireshark directly without sflowtool?

from sflowtool.

from sflowtool.

Thank you ! @pavel-odintsov - Can u tell us how did u reproduce the issue? What kind of traffic you are seeing this ?
We tried to reproduce in lab and we are not seeing this issue

from sflowtool.

Here you could find example of broken packet

from sflowtool.

Also I could quote my report to Brocade "After some brief research I discovered that Brocade add some padding to the end of sflow sample at packet end."

You could ask @AdamVocks for pcap examples if you need them. Also check pavel-odintsov/fastnetmon#626 for more details.

from sflowtool.

@pavel-odintsov Brocade did not introduce 4 bytes. Sflow version 5 has a new field "stripped". This value can be 4 or 8 depending on whether VLAN is tagged or untagged.
Does the sflowtool support stripped byte functionality?

http://www.sflow.org/sflow_version_5.txt
unsigned int stripped; /* The number of octets removed from
the packet before extracting the
header<> octets. Trailing encapsulation
data corresponding to any leading
encapsulations that were stripped must
also be stripped. Trailing encapsulation
data for the outermost protocol layer
included in the sampled header must be
stripped.

                                  In the case of a non-encapsulated 802.3
                                  packet stripped >= 4 since VLAN tag
                                  information might have been stripped off
                                  in addition to the FCS.

                                  Outer encapsulations that are ambiguous,
                                  or not one of the standard header_protocol
                                  must be stripped. */

from sflowtool.

from sflowtool.

Thanks @pavel-odintsov !
I dont have the packet capture file. Customer didnt share the data. What i saw in the issue meeting was the i am getting below error after the Extended Router data
flow_sample length error (expected 284, found 280)
caught exception: 3
We tried locally and we are not seeing issue.
So wanted to know how to recreate the issue ?

@AdamVocks - could you please share us the steps to recreate the issue? What traffic where you using when you observed this?

from sflowtool.

We were working fine then we started getting these errors. We're not aware of any changes that happened. We have about 3gbps of traffic flowing through and about 1gbps of that traffic is IPTV multicast.

from sflowtool.

@AdamVocks - Thanks ! are you seeing this problem frequently ?
Also what is the brocade ticket no?

from sflowtool.

from sflowtool.

@AdamVocks - do you have the brocade ticket no ?

from sflowtool.

I'm having the same issue on multiple ICX 7750's, which run the same software as the 6610's I can provide a pcap if you guys still need it, and I'd be happy to open a case with Brocade if needed.

from sflowtool.

from sflowtool.

Yes happening now.
Fastnetmon log =
2017-03-21 09:06:35,210 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,211 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,211 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,370 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,370 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,450 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,450 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,517 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,518 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,571 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,571 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,582 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,582 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,582 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,582 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,698 [ERROR] sflow: flow_sample length error: expected 212 found 208
2017-03-21 09:06:35,698 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,710 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,710 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,777 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,777 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,785 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,785 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,952 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,952 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:35,965 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:35,966 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:36,100 [ERROR] sflow: flow_sample length error: expected 224 found 220
2017-03-21 09:06:36,100 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:36,111 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:36,111 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:36,170 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:36,170 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:36,227 [ERROR] sflow: flow_sample length error: expected 284 found 280
2017-03-21 09:06:36,227 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:36,286 [ERROR] sflow: flow_sample length error: expected 224 found 220
2017-03-21 09:06:36,287 [ERROR] sflow: we failed in SFLFLOW_SAMPLE handler
2017-03-21 09:06:36,357 [ERROR] sflow: flow_sample length error: expected 224 found 220

Brocade 7750 config is:

sflow agent-ip
sflow sample 512
sflow source ve 3
sflow destination
sflow enable

for each interface:
int ethernet 1/1/1
sflow forwarding

Currently running Brocade version 8.0.30mb.

I can send the pcap directly if you need it.

from sflowtool.

from sflowtool.

Its just generic internet traffic. Mostly http stuff.

from sflowtool.

It would be really nice if you could open support issue to Brocade. It could speed up fix process significantly! :)

from sflowtool.

I'll open a brocade case today. However, I have one thing to add. I made a change to the Brocade ICX that seems to have resolved the miscalculation of rates inside Fastnetmon. Brocade sets a sflow max packet size of 128 bytes by default. With the default setting, fastnetmon would calculate the live throughput rates incorrectly and they would appear to be much lower than the actual rate. Changing this setting to 1200 bytes seems to have resolved that issue. Unfortunately, the log message still reports the padding issue.

Not sure if this is an sflow issue, or perhaps this belongs back in the fastnetmon github.

from sflowtool.

from sflowtool.

Hello!

There is strong evidence that:

• sflowtools from sFlow authors does not like sFlow prepared by Brocade
• FastNetMon does not like this sFlow prepared by Brocade also

Also, I could confirm that FNM works really well with more than 12 different vendors and we have never experienced any issues related to this behavior.

We deployed a small fix in FastNetMon Advanced and now we could work with any Brocade switch. But for Community version it's still complicated because our parser used inside it's not so flexible.

So at this moment I still think that it's some corner case and brocade know how to fix it :) But I could be wrong because I'm human.

from sflowtool.

I have a case open with Brocade and they are requesting a specific example in the pcap I sent. How or Where do I identify the padding to be present in the pcap?

from sflowtool.

The issue seems to be with the length-field of a packet sample containing an extended_gateway structure. The first message in this thread (from Jan 31) shows the problem quite well. You have an sFlow flow sample that claims to have length = 0x0000011c = 284, but then the elements within that flow-sample (packet-header + extended_switch + extended_router + extended_gateway) only add up to 280 bytes. Here it is with tag+length fields highlighted in bold:

0x0430:  **0000 0001 0000 011c** 904d ea3e 0000 0081  .........M.>....
0x0440:  0000 0400 76bb cee0 0000 0000 0000 0081  ....v...........
0x0450:  0000 0005 0000 0004 0000 0001 0000 0090  ................
0x0460:  0000 0001 0000 042e 0000 0008 0000 0080  ................
      ====== skip packet header bytes =======
0x04f0:  **0000 03e9 0000 0010** 0000 0069 0000 0000  ...........i....
0x0500:  0000 001b 0000 0000 **0000 03ea 0000 0010**  ................
0x0510:  0000 0001 0a00 03fe 0000 0000 0000 0000  ................
0x0520:  **0000 03eb 0000 0028** 0000 0001 0a00 03fe  .......(........
0x0530:  0000 7da0 0000 00ae 0000 00ae 0000 0001  ..}.............
0x0540:  0000 0002 0000 0000 0000 0000 0000 0000  ................
0x0550:  0000 0000                                ....

So pulling out the bold fields we have:

flow_sample with tag=0x00000001 and length=0x0000011c (284) consisting of:
(1) packet header
(2) extended_switch with tag=0x000003e9 (1001) and length=0x00000010 (16)
(3) extended_router with tag=0x000003ea (1002) and length=0x00000010 (16)
(4) extended_gateway with tag=0x000003eb (1003) and length=0x00000028 (40)

But if you count off those 40 bytes of extended_gateway you find that they only account for the bytes up to the end of the row marked "0x0540:". The last 4 bytes (all zeros) are unexpected.

So when sflowtool finishes parsing (1),(2),(3) and (4) it cross-checks to see if it has consumed the whole 284 bytes of the flow sample. But it has only consumed 280, so it complains here:
https://github.com/sflow/sflowtool/blob/master/src/sflowtool.c#L2930

For this to look right the flow sample should only have claimed to be 280 bytes in the first place, and those last 4 bytes should not be there at all. They were not needed for quad-alignment padding.

Earlier in the same datagram there is another flow-sample with an extended_gateway structure of length 0x00000030 and it seems to line up OK:

    0x00f0:  0000 0000 0000 0000 **0000 03eb 0000 0030**  ...............0
    0x0100:  0000 0001 cedf 77b6 0000 7da0 0000 0000  ......w...}.....
    0x0110:  0000 0000 0000 0001 0000 0002 0000 0002  ................
    0x0120:  0000 02ca 0000 1829 0000 0000 0000 0064  .......).......d
    0x0130:  0000 0001 0000 00d0 904d ea3b 0000 0081  .........M.;....

Hope this helps.

from sflowtool.

Perfect, I see it now in my pcaps. I'll update Brocade, and see where things go.

from sflowtool.

Guys who created tickets to Brocade. Could you share ticket numbers with me here or privately to [email protected]? I'll try to escalate them.

from sflowtool.

Case number is 1731309

Last update from Brocade was:
"We still under analysis with this case, senior engineers and product management are involved. Any update I will let you know. "

from sflowtool.

@ssimplesean thank you so much!

from sflowtool.

Had a great call with Brocade TAC a few weeks back, and was able to show them exactly what the problem was. I was informed today that the issue was assigned as DEFECT000641986 and would be fixed in the upcoming 8030p and 8061 releases. No ETA at this time.

from sflowtool.

@ssimplesean thank you so much! I updated progress of this issue in FastNetMon's bug tracker: pavel-odintsov/fastnetmon#626

from sflowtool.