Quantcast

Broken Pipe Exception - failed reporting to console

classic Classic list List threaded Threaded
6 messages Options
Lei
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Broken Pipe Exception - failed reporting to console

Lei
Hello,

I am trying to run grinder agent in Amazon EC2 instance and run console in my local network. All instances including console are within one VPN network. I am pretty positive about the network between EC2 instance and local machine runs grinder console. 

While the agent got started, it connected to console instantly. Agent showed up in console correctly. 

However once the agent start testing, it failed reporting to console with below Broken Pipe exception. 

I am aware similar Exception had been reported by another user a few months ago. Given the network is pretty stable, and this problem happened consistently from my observation. This problem might not because of unstable internet connection. 

Any advice for debugging this issue will be appreciated. 

Thanks,
Lei   

014-08-23 05:02:16,908 INFO  ip-IP-0: starting threads                                                                                                                                [51/1640]
2014-08-23 05:02:17,325 INFO  ip-IP-1: Report to console failed
net.grinder.communication.CommunicationException: Exception whilst sending message
        at net.grinder.communication.AbstractSender.send(AbstractSender.java:57) ~[grinder-core-3.11.jar:na]
        at net.grinder.communication.QueuedSenderDecorator.flush(QueuedSenderDecorator.java:60) ~[grinder-core-3.11.jar:na]
        at net.grinder.engine.process.GrinderProcess.sendStatusMessage(GrinderProcess.java:638) ~[grinder-core-3.11.jar:na]
        at net.grinder.engine.process.GrinderProcess.access$1100(GrinderProcess.java:110) ~[grinder-core-3.11.jar:na]
        at net.grinder.engine.process.GrinderProcess$ReportToConsoleTimerTask.run(GrinderProcess.java:615) ~[grinder-core-3.11.jar:na]
        at java.util.TimerThread.mainLoop(Timer.java:555) [na:1.7.0_67]
        at java.util.TimerThread.run(Timer.java:505) [na:1.7.0_67]
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_67]
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[na:1.7.0_67]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:159) ~[na:1.7.0_67]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.7.0_67]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.7.0_67]
        at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1822) ~[na:1.7.0_67]
        at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:718) ~[na:1.7.0_67]
        at net.grinder.communication.AbstractSender.writeMessageToStream(AbstractSender.java:90) ~[grinder-core-3.11.jar:na]
        at net.grinder.communication.StreamSender.writeMessage(StreamSender.java:70) ~[grinder-core-3.11.jar:na]
        at net.grinder.communication.AbstractSender.send(AbstractSender.java:53) ~[grinder-core-3.11.jar:na]
        ... 6 common frames omitted

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Broken Pipe Exception - failed reporting to console

Joan Picanyol i Puig
* Lei <[hidden email]> [20140823 16:20]:

> I am trying to run grinder agent in Amazon EC2 instance and run console in
> my local network. All instances including console are within one VPN
> network. I am pretty positive about the network between EC2 instance and
> local machine runs grinder console.
>
> While the agent got started, it connected to console instantly. Agent
> showed up in console correctly.
>
> However once the agent start testing, it failed reporting to console with
> below Broken Pipe exception.
>
> I am aware similar Exception had been reported by another user a few months
> ago. Given the network is pretty stable, and this problem happened
> consistently from my observation. This problem might not because of
> unstable internet connection.
>
> Any advice for debugging this issue will be appreciated.

My bet is that you are trying to create processes and/or threads
too fast and overloading your agent. I suggest trying to diagnose wether
it is a maximum agent load issue or a rate issue (i.e.: wether you can
create the intended load over a longer timeframe).

qvb
--
pica

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Broken Pipe Exception - failed reporting to console

Gary Mulder-3
On 25 August 2014 08:34, Joan Picanyol i Puig <[hidden email]> wrote:
* Lei <[hidden email]> [20140823 16:20]: 
> However once the agent start testing, it failed reporting to console with
> below Broken Pipe exception.

My bet is that you are trying to create processes and/or threads
too fast and overloading your agent. I suggest trying to diagnose wether
it is a maximum agent load issue or a rate issue (i.e.: wether you can
create the intended load over a longer timeframe).

To confirm whether Joan's theory is correct run some commands on the agent to measure load. On Linux run:

$ vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 5  1      0 26217732 149992 2266752    0    0     0     3    8   10  0  0 100  0
 7  0      0 26217116 149992 2266780    0    0     0     1 76425 91969 13 14 73  0

The r column shows how many threads want to run on average. lsof -i is another useful command to confirm you're not running out of TCP/IP sockets (maximum 32K allowed).

Of course if you're running Windows it might be harder to see if you're hitting resource limits... 

Regards.
Gary

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Lei
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Broken Pipe Exception - failed reporting to console

Lei
On Mon, Aug 25, 2014 at 7:18 AM, Gary Mulder <[hidden email]> wrote:
On 25 August 2014 08:34, Joan Picanyol i Puig <[hidden email]> wrote:
* Lei <[hidden email]> [20140823 16:20]: 
> However once the agent start testing, it failed reporting to console with
> below Broken Pipe exception.

My bet is that you are trying to create processes and/or threads
too fast and overloading your agent. I suggest trying to diagnose wether
it is a maximum agent load issue or a rate issue (i.e.: wether you can
create the intended load over a longer timeframe).

To confirm whether Joan's theory is correct run some commands on the agent to measure load. On Linux run:

$ vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 5  1      0 26217732 149992 2266752    0    0     0     3    8   10  0  0 100  0
 7  0      0 26217116 149992 2266780    0    0     0     1 76425 91969 13 14 73  0

The r column shows how many threads want to run on average. lsof -i is another useful command to confirm you're not running out of TCP/IP sockets (maximum 32K allowed).

Of course if you're running Windows it might be harder to see if you're hitting resource limits... 

Regards.
Gary


Thank you pica and Gary!

I didn't see the heavy load on the console. Looks like once the agent gets some load (more than 100 TCP connections), the connection between agent and console becomes unstable. 

I have tested different configuration from grinder properties. Here are some observations: 

Reduce the number of below two properties to some small number, the problem goes a way. But it defeat the purpose of load testing.

 grinder.processes
 grinder.threads 

Reducing grinder.processIncrement and grinder.initialProcesses and increasing grinder.processIncrementInterval can mitigate or delay this problem sometimes. 

Since I didn't find any better way to resolve this issue, I might just go with agent without console, and use "grinder to graphite" (https://bitbucket.org/travis_bear/grinder_to_graphite) for collecting testing result. 

Thanks,
Lei


------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Broken Pipe Exception - failed reporting to console

Marc Holden
Hi Lei,

You may want to check the number of file descriptors you can have open.  IIRC, if you run out of descriptors or locks you will get broken pipe exceptions.  A message should also be logged in /var/log/syslog (which is always a good place to start troubleshooting these types of errors....) and with any luck it will provide some additional information around the exception.  Make sure you check the syslogs on both the console and agent machines.

See http://glassonionblog.wordpress.com/2013/01/27/increase-ulimit-and-file-descriptors-limit/ for details on setting ulimits and max number of locks.

Hope this helps.

-Marc






On Tue, Aug 26, 2014 at 3:30 PM, Lei <[hidden email]> wrote:
On Mon, Aug 25, 2014 at 7:18 AM, Gary Mulder <[hidden email]> wrote:
On 25 August 2014 08:34, Joan Picanyol i Puig <[hidden email]> wrote:
* Lei <[hidden email]> [20140823 16:20]: 
> However once the agent start testing, it failed reporting to console with
> below Broken Pipe exception.

My bet is that you are trying to create processes and/or threads
too fast and overloading your agent. I suggest trying to diagnose wether
it is a maximum agent load issue or a rate issue (i.e.: wether you can
create the intended load over a longer timeframe).

To confirm whether Joan's theory is correct run some commands on the agent to measure load. On Linux run:

$ vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 5  1      0 26217732 149992 2266752    0    0     0     3    8   10  0  0 100  0
 7  0      0 26217116 149992 2266780    0    0     0     1 76425 91969 13 14 73  0

The r column shows how many threads want to run on average. lsof -i is another useful command to confirm you're not running out of TCP/IP sockets (maximum 32K allowed).

Of course if you're running Windows it might be harder to see if you're hitting resource limits... 

Regards.
Gary


Thank you pica and Gary!

I didn't see the heavy load on the console. Looks like once the agent gets some load (more than 100 TCP connections), the connection between agent and console becomes unstable. 

I have tested different configuration from grinder properties. Here are some observations: 

Reduce the number of below two properties to some small number, the problem goes a way. But it defeat the purpose of load testing.

 grinder.processes
 grinder.threads 

Reducing grinder.processIncrement and grinder.initialProcesses and increasing grinder.processIncrementInterval can mitigate or delay this problem sometimes. 

Since I didn't find any better way to resolve this issue, I might just go with agent without console, and use "grinder to graphite" (https://bitbucket.org/travis_bear/grinder_to_graphite) for collecting testing result. 

Thanks,
Lei


------------------------------------------------------------------------------
Slashdot TV.
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use



------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Broken Pipe Exception - failed reporting to console

Gary Mulder-3
In reply to this post by Lei
On 26 August 2014 20:30, Lei <[hidden email]> wrote:

Reducing grinder.processIncrement and grinder.initialProcesses and increasing grinder.processIncrementInterval can mitigate or delay this problem sometimes. 

Since I didn't find any better way to resolve this issue, I might just go with agent without console, and use "grinder to graphite" (https://bitbucket.org/travis_bear/grinder_to_graphite) for collecting testing result. 

I'd hunt down this issue, as it may indicate you have deeper network issues as it should work fine.

Check out http://www.netperf.org/netperf, a great little network benchmarking tool that often identifies network issues.

Regards,
Gary

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Loading...