Quantcast

atExit worker state - inconsistency?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

atExit worker state - inconsistency?

Darren Ball
Hi All,

I've come across a pretty unusual situation in my polling logic for workers (use to tell if they are 'done').  I have test logic that runs many scenarios, and I typically poll for workers like this:

workerstate=$(curl --noproxy localhost -s -X GET http://localhost:6373/agents/status | /usr/local/bin/jq -r '.[].workers[].state' | wc -l)

I have an atExit function registered, that cleans up data that was created/generated during the run.  
The odd behaviour can be seen below, captured with a simply loop outside the run for observation (1 sec interval) to check the state of workers while a test was running.  It seams that once test runner exits, the state of status of t
the agents's workers are 'cleared' (which was my indication that things were really done).  Grinder also issues a termination at 10 seconds as well, regrettably. 
As can been seen, the agent workers look like their state went from running to nothing and then back to finished.  Time between this is 30 seconds.   Threads are still up during this period, as the cleanup is still being processed. 
Is there any efficient way to determine successfully that the workers have passed the 'finished' state?  I can't really rely on polling for a state of finished, as there is a chance that it will be missed (it is cleared).

Thanks.


[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"finished","workers":[{"maximum-threads":0,"running-threads":0,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"finished"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"finished","workers":[{"maximum-threads":0,"running-threads":0,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"finished"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"started","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]


------------------------------------------------------------------------------

_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: atExit worker state - inconsistency?

Joel Lucuik

This is likely the scenario where the agent is just starting. To avoid this, consider polling the running status every 60 seconds. With that you can skip those 30 seconds scenario where grinder is trying to start worker processes.

 

Maybe increase the sleeptime between polling status.



On Thu, Sep 29, 2016 at 1:23 PM, Darren Ball <[hidden email]> wrote:
Hi All,

I've come across a pretty unusual situation in my polling logic for workers (use to tell if they are 'done').  I have test logic that runs many scenarios, and I typically poll for workers like this:

workerstate=$(curl --noproxy localhost -s -X GET http://localhost:6373/agents/status | /usr/local/bin/jq -r '.[].workers[].state' | wc -l)

I have an atExit function registered, that cleans up data that was created/generated during the run.  
The odd behaviour can be seen below, captured with a simply loop outside the run for observation (1 sec interval) to check the state of workers while a test was running.  It seams that once test runner exits, the state of status of t
the agents's workers are 'cleared' (which was my indication that things were really done).  Grinder also issues a termination at 10 seconds as well, regrettably. 
As can been seen, the agent workers look like their state went from running to nothing and then back to finished.  Time between this is 30 seconds.   Threads are still up during this period, as the cleanup is still being processed. 
Is there any efficient way to determine successfully that the workers have passed the 'finished' state?  I can't really rely on polling for a state of finished, as there is a chance that it will be missed (it is cleared).

Thanks.


[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"finished","workers":[{"maximum-threads":0,"running-threads":0,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"finished"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"finished","workers":[{"maximum-threads":0,"running-threads":0,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"finished"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"started","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]


------------------------------------------------------------------------------

_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use



------------------------------------------------------------------------------

_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: atExit worker state - inconsistency?

Darren Ball
Hi Joel,

Thanks for your feedback.  
This is actually during the stopping process or the tear down.  

My polling is every 5 seconds, but displayed in the original was a test that was showing the inconsistent state of worker tracking during atexit.

Update on this as well:

If I use __del__, the state of the workers remains 'running', but if atexit is used, and registered outside of ThreadRunner, this state seems to be 'lost'.  During shutdown this is a problem.   By adding the atexit outside of TestRunner, the total time of the test is only that of TestRunner.  
Adding __del__, extends the execution time of TestRunner, calculations are off then.  My cleanup can take several (> 10) seconds...





On Fri, Sep 30, 2016 at 9:53 AM, Joel Lucuik <[hidden email]> wrote:

This is likely the scenario where the agent is just starting. To avoid this, consider polling the running status every 60 seconds. With that you can skip those 30 seconds scenario where grinder is trying to start worker processes.

 

Maybe increase the sleeptime between polling status.



On Thu, Sep 29, 2016 at 1:23 PM, Darren Ball <[hidden email]> wrote:
Hi All,

I've come across a pretty unusual situation in my polling logic for workers (use to tell if they are 'done').  I have test logic that runs many scenarios, and I typically poll for workers like this:

workerstate=$(curl --noproxy localhost -s -X GET http://localhost:6373/agents/status | /usr/local/bin/jq -r '.[].workers[].state' | wc -l)

I have an atExit function registered, that cleans up data that was created/generated during the run.  
The odd behaviour can be seen below, captured with a simply loop outside the run for observation (1 sec interval) to check the state of workers while a test was running.  It seams that once test runner exits, the state of status of t
the agents's workers are 'cleared' (which was my indication that things were really done).  Grinder also issues a termination at 10 seconds as well, regrettably. 
As can been seen, the agent workers look like their state went from running to nothing and then back to finished.  Time between this is 30 seconds.   Threads are still up during this period, as the cleanup is still being processed. 
Is there any efficient way to determine successfully that the workers have passed the 'finished' state?  I can't really rely on polling for a state of finished, as there is a chance that it will be missed (it is cleared).

Thanks.


[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"finished","workers":[{"maximum-threads":0,"running-threads":0,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"finished"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"finished","workers":[{"maximum-threads":0,"running-threads":0,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"finished"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"started","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]


------------------------------------------------------------------------------

_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use



------------------------------------------------------------------------------

_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use



------------------------------------------------------------------------------

_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: atExit worker state - inconsistency?

Darren Ball
Anyone else have any insight/ideas on this?  Development?

On Fri, Sep 30, 2016 at 10:43 AM, Darren Ball <[hidden email]> wrote:
Hi Joel,

Thanks for your feedback.  
This is actually during the stopping process or the tear down.  

My polling is every 5 seconds, but displayed in the original was a test that was showing the inconsistent state of worker tracking during atexit.

Update on this as well:

If I use __del__, the state of the workers remains 'running', but if atexit is used, and registered outside of ThreadRunner, this state seems to be 'lost'.  During shutdown this is a problem.   By adding the atexit outside of TestRunner, the total time of the test is only that of TestRunner.  
Adding __del__, extends the execution time of TestRunner, calculations are off then.  My cleanup can take several (> 10) seconds...





On Fri, Sep 30, 2016 at 9:53 AM, Joel Lucuik <[hidden email]> wrote:

This is likely the scenario where the agent is just starting. To avoid this, consider polling the running status every 60 seconds. With that you can skip those 30 seconds scenario where grinder is trying to start worker processes.

 

Maybe increase the sleeptime between polling status.



On Thu, Sep 29, 2016 at 1:23 PM, Darren Ball <[hidden email]> wrote:
Hi All,

I've come across a pretty unusual situation in my polling logic for workers (use to tell if they are 'done').  I have test logic that runs many scenarios, and I typically poll for workers like this:

workerstate=$(curl --noproxy localhost -s -X GET http://localhost:6373/agents/status | /usr/local/bin/jq -r '.[].workers[].state' | wc -l)

I have an atExit function registered, that cleans up data that was created/generated during the run.  
The odd behaviour can be seen below, captured with a simply loop outside the run for observation (1 sec interval) to check the state of workers while a test was running.  It seams that once test runner exits, the state of status of t
the agents's workers are 'cleared' (which was my indication that things were really done).  Grinder also issues a termination at 10 seconds as well, regrettably. 
As can been seen, the agent workers look like their state went from running to nothing and then back to finished.  Time between this is 30 seconds.   Threads are still up during this period, as the cleanup is still being processed. 
Is there any efficient way to determine successfully that the workers have passed the 'finished' state?  I can't really rely on polling for a state of finished, as there is a chance that it will be missed (it is cleared).

Thanks.


[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[{"maximum-threads":20,"running-threads":20,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"running"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"finished","workers":[{"maximum-threads":0,"running-threads":0,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"finished"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"finished","workers":[{"maximum-threads":0,"running-threads":0,"id":"i-0a98f94c8d82cb1fd-24:371439501|1475085007713|1565583566:25","name":"i-0a98f94c8d82cb1fd-24","number":24,"state":"finished"}]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"started","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]
[{"id":"ip-10-82-147-72:371439501|1475085007713|1565583566:0","name":"i-0a98f94c8d82cb1fd","number":0,"state":"running","workers":[]}]


------------------------------------------------------------------------------

_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use



------------------------------------------------------------------------------

_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Loading...