Test instrumentation using a self.method

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Test instrumentation using a self.method

Darren Ball
Is it possible in the testrunner init to do something like:

tests["abc"].record(self.abc)?

Why is it that a specific java object method can not be instrumented?  Seem like only the whole object, meaning every call within?

------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Test instrumentation using a self.method

Philip Aston-2
On 16/09/15 14:29, Darren Ball wrote:
> Is it possible in the testrunner init to do something like:
>
> tests["abc"].record(self.abc)?
>
> Why is it that a specific java object method can not be instrumented?
> Seem like only the whole object, meaning every call within?
>

You can't instrument a specific method by passing it to record as above
is that under the covers because Java does not provide a references to
bound methods, and so the Jython expression "self.abc" is actually a
Jython wrapper. The wrappers change with each version of Jython so we
(no longer) rely on Jython internals to figure out what exactly they
refer to.

Instead you can use an overloaded version of record that accepts a
filter that specifies just the methods you care about. See
http://grinder.sourceforge.net/g3/instrumentation.html#Selective+instrumentation


- Phil



------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Test instrumentation using a self.method

Darren Ball
Hi Philip,

I just tried this, with no luck.
Here is what I have attempted, where cscAppPerf is the java object being used to run tests, and CscAppPerfFilter.matches returning my valid test methods.

class CscAppPerfFilter(Test.InstrumentationFilter):
    def matches(self, method):
        return method.name in ["perfGetTableRecord", "perfRetrieveReport","perfRetrieveTable","perfGetTableRecord","perfUpdateTableRecord","perfDeleteTableRecord","perfAddBulkTableRecord"]


# Instrumentation for tests
tests["createReport"].record(cscAppPerf, CscAppPerfFilter())
tests["createTable"].record(cscAppPerf, CscAppPerfFilter())
tests["createRecord"].record(cscAppPerf, CscAppPerfFilter())
tests["readReport"].record(cscAppPerf, CscAppPerfFilter())
tests["readTable"].record(cscAppPerf, CscAppPerfFilter())
tests["readRecord"].record(cscAppPerf, CscAppPerfFilter())
tests["updateReport"].record(cscAppPerf, CscAppPerfFilter())
tests["updateTable"].record(cscAppPerf, CscAppPerfFilter())
tests["updateRecord"].record(cscAppPerf, CscAppPerfFilter())
tests["deleteReport"].record(cscAppPerf, CscAppPerfFilter())
tests["deleteTable"].record(cscAppPerf, CscAppPerfFilter())
tests["deleteRecord"].record(cscAppPerf, CscAppPerfFilter())
tests["bulkAddRecord"].record(cscAppPerf, CscAppPerfFilter())

My stats summary is now somewhat unusual:


2015-09-16 19:06:34,305 INFO  i-20ff63e6-0 : Final statistics for this process:
2015-09-16 19:06:34,313 INFO  i-20ff63e6-0 :

             Tests        Errors       Mean Test    Test Time    TPS          
                                       Time (ms)    Standard                  
                                                    Deviation                
                                                    (ms)                      


(Test 1      126          0            744.15       1612.83      1.32)         "createReport"
(Test 2      126          0            744.14       1612.80      1.32)         "createTable"
(Test 3      126          0            744.11       1612.81      1.32)         "createRecord"
(Test 4      126          0            744.10       1612.81      1.32)         "readReport"
(Test 5      126          0            744.08       1612.82      1.32)         "readTable"
(Test 6      126          0            744.06       1612.82      1.32)         "readRecord"
(Test 7      126          0            744.02       1612.76      1.32)         "updateReport"
(Test 8      126          0            744.01       1612.76      1.32)         "updateTable"
(Test 9      126          0            744.00       1612.76      1.32)         "updateRecord"
(Test 10     126          0            743.99       1612.76      1.32)         "deleteReport"
(Test 11     126          0            743.99       1612.76      1.32)         "deleteTable"
(Test 12     126          0            743.96       1612.76      1.32)         "deleteRecord"
Test 13      126          0            743.95       1612.76      1.32          "bulkAddRecord"

Totals       126          0            743.95       1612.76      1.32        
             (1512)       (0)                                                

Note:
1.  I am using weighted algorithm for running my tests, and 126 total count for each test is wrong.  All statistics are wrong now.
2.  Test 13 does not have parentheses around it?  Why is that?

Here is what the results look like when I use the instrumentation of javaobject.method like the following:
tests["createReport"].record(cscAppPerf.perfGetTableRecord)
tests["createTable"].record(cscAppPerf.perfGetTableRecord)
tests["createRecord"].record(cscAppPerf.perfGetTableRecord)
tests["readReport"].record(cscAppPerf.perfRetrieveReport)
tests["readTable"].record(cscAppPerf.perfRetrieveTable)
tests["readRecord"].record(cscAppPerf.perfGetTableRecord)
tests["updateReport"].record(cscAppPerf.perfGetTableRecord)
tests["updateTable"].record(cscAppPerf.perfGetTableRecord)
tests["updateRecord"].record(cscAppPerf.perfUpdateTableRecord)
tests["deleteReport"].record(cscAppPerf.perfGetTableRecord)
tests["deleteTable"].record(cscAppPerf.perfGetTableRecord)
tests["deleteRecord"].record(cscAppPerf.perfDeleteTableRecord)
tests["bulkAddRecord"].record(cscAppPerf.perfAddBulkTableRecord)

Statistics:
2015-09-16 19:20:09,472 INFO  i-20ff63e6-1 : Final statistics for this process:
2015-09-16 19:20:09,480 INFO  i-20ff63e6-1 :

             Tests        Errors       Mean Test    Test Time    TPS          
                                       Time (ms)    Standard                  
                                                    Deviation                
                                                    (ms)                      

(Test 1      3            0            255.33       16.74        0.15)         "createReport"
(Test 2      3            0            255.33       16.74        0.15)         "createTable"
(Test 3      3            0            255.33       16.74        0.15)         "createRecord"
Test 4       4            0            47.75        12.75        0.20          "readReport"
Test 5       4            0            4734.25      1530.59      0.20          "readTable"
(Test 6      3            0            255.33       16.74        0.15)         "readRecord"
(Test 7      3            0            255.33       16.74        0.15)         "updateReport"
(Test 8      3            0            255.33       16.74        0.15)         "updateTable"
Test 9       2            0            53.50        3.50         0.10          "updateRecord"
(Test 10     3            0            255.33       16.74        0.15)         "deleteReport"
Test 11      3            0            255.33       16.74        0.15          "deleteTable"
Test 12      2            0            39.50        0.50         0.10          "deleteRecord"
Test 13      1            0            74.00        0.00         0.05          "bulkAddRecord"

Totals       16           0            1259.62      2148.51      0.78        
             (21)         (0)                

This is more realistic and is actually happening - but seemingly not possible?                            

On Wed, Sep 16, 2015 at 1:56 PM, Philip Aston <[hidden email]> wrote:
On 16/09/15 14:29, Darren Ball wrote:
> Is it possible in the testrunner init to do something like:
>
> tests["abc"].record(self.abc)?
>
> Why is it that a specific java object method can not be instrumented?
> Seem like only the whole object, meaning every call within?
>

You can't instrument a specific method by passing it to record as above
is that under the covers because Java does not provide a references to
bound methods, and so the Jython expression "self.abc" is actually a
Jython wrapper. The wrappers change with each version of Jython so we
(no longer) rely on Jython internals to figure out what exactly they
refer to.

Instead you can use an overloaded version of record that accepts a
filter that specifies just the methods you care about. See
http://grinder.sourceforge.net/g3/instrumentation.html#Selective+instrumentation


- Phil



------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use


------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Test instrumentation using a self.method

Darren Ball
FYI,

I have solved this in a different way, as the instrumentation seems to not work as expect (or I am simply using it incorrectly).

I have moved testing related methods outside of TestRunner and wrapped them based on this example:

http://open.bekk.no/performance-testing-using-grinder

It would be great to have a large example base, is there any way people can add to the sourceforge script gallery?

I ended up with instrumentation something like the following:

self.wrapped_createReportTest = tests["createReport"].wrap(createReportTest)
self.wrapped_createTableTest = tests["createTable"].wrap(createTableTest)
self.wrapped_createRecordTest = tests["createRecord"].wrap(createRecordTest)
self.wrapped_readReportTest = tests["readReport"].wrap(readReportTest)
self.wrapped_readTableTest = tests["readTable"].wrap(readTableTest)
self.wrapped_readRecordTest = tests["readRecord"].wrap(readRecordTest)
self.wrapped_updateReportTest = tests["updateReport"].wrap(updateReportTest)
self.wrapped_updateTableTest = tests["updateTable"].wrap(updateTableTest)
self.wrapped_updateRecordTest = tests["updateRecord"].wrap(updateRecordTest)
self.wrapped_deleteReportTest = tests["deleteReport"].wrap(deleteReportTest)
self.wrapped_deleteTableTest = tests["deleteTable"].wrap(deleteTableTest)
self.wrapped_deleteRecordTest = tests["deleteRecord"].wrap(deleteRecordTest)
self.wrapped_addBulkRecordTest = tests["bulkAddRecord"].wrap(addBulkRecordTest)


On Wed, Sep 16, 2015 at 3:22 PM, Darren Ball <[hidden email]> wrote:
Hi Philip,

I just tried this, with no luck.
Here is what I have attempted, where cscAppPerf is the java object being used to run tests, and CscAppPerfFilter.matches returning my valid test methods.

class CscAppPerfFilter(Test.InstrumentationFilter):
    def matches(self, method):
        return method.name in ["perfGetTableRecord", "perfRetrieveReport","perfRetrieveTable","perfGetTableRecord","perfUpdateTableRecord","perfDeleteTableRecord","perfAddBulkTableRecord"]


# Instrumentation for tests
tests["createReport"].record(cscAppPerf, CscAppPerfFilter())
tests["createTable"].record(cscAppPerf, CscAppPerfFilter())
tests["createRecord"].record(cscAppPerf, CscAppPerfFilter())
tests["readReport"].record(cscAppPerf, CscAppPerfFilter())
tests["readTable"].record(cscAppPerf, CscAppPerfFilter())
tests["readRecord"].record(cscAppPerf, CscAppPerfFilter())
tests["updateReport"].record(cscAppPerf, CscAppPerfFilter())
tests["updateTable"].record(cscAppPerf, CscAppPerfFilter())
tests["updateRecord"].record(cscAppPerf, CscAppPerfFilter())
tests["deleteReport"].record(cscAppPerf, CscAppPerfFilter())
tests["deleteTable"].record(cscAppPerf, CscAppPerfFilter())
tests["deleteRecord"].record(cscAppPerf, CscAppPerfFilter())
tests["bulkAddRecord"].record(cscAppPerf, CscAppPerfFilter())

My stats summary is now somewhat unusual:


2015-09-16 19:06:34,305 INFO  i-20ff63e6-0 : Final statistics for this process:
2015-09-16 19:06:34,313 INFO  i-20ff63e6-0 :

             Tests        Errors       Mean Test    Test Time    TPS          
                                       Time (ms)    Standard                  
                                                    Deviation                
                                                    (ms)                      


(Test 1      126          0            744.15       1612.83      1.32)         "createReport"
(Test 2      126          0            744.14       1612.80      1.32)         "createTable"
(Test 3      126          0            744.11       1612.81      1.32)         "createRecord"
(Test 4      126          0            744.10       1612.81      1.32)         "readReport"
(Test 5      126          0            744.08       1612.82      1.32)         "readTable"
(Test 6      126          0            744.06       1612.82      1.32)         "readRecord"
(Test 7      126          0            744.02       1612.76      1.32)         "updateReport"
(Test 8      126          0            744.01       1612.76      1.32)         "updateTable"
(Test 9      126          0            744.00       1612.76      1.32)         "updateRecord"
(Test 10     126          0            743.99       1612.76      1.32)         "deleteReport"
(Test 11     126          0            743.99       1612.76      1.32)         "deleteTable"
(Test 12     126          0            743.96       1612.76      1.32)         "deleteRecord"
Test 13      126          0            743.95       1612.76      1.32          "bulkAddRecord"

Totals       126          0            743.95       1612.76      1.32        
             (1512)       (0)                                                

Note:
1.  I am using weighted algorithm for running my tests, and 126 total count for each test is wrong.  All statistics are wrong now.
2.  Test 13 does not have parentheses around it?  Why is that?

Here is what the results look like when I use the instrumentation of javaobject.method like the following:
tests["createReport"].record(cscAppPerf.perfGetTableRecord)
tests["createTable"].record(cscAppPerf.perfGetTableRecord)
tests["createRecord"].record(cscAppPerf.perfGetTableRecord)
tests["readReport"].record(cscAppPerf.perfRetrieveReport)
tests["readTable"].record(cscAppPerf.perfRetrieveTable)
tests["readRecord"].record(cscAppPerf.perfGetTableRecord)
tests["updateReport"].record(cscAppPerf.perfGetTableRecord)
tests["updateTable"].record(cscAppPerf.perfGetTableRecord)
tests["updateRecord"].record(cscAppPerf.perfUpdateTableRecord)
tests["deleteReport"].record(cscAppPerf.perfGetTableRecord)
tests["deleteTable"].record(cscAppPerf.perfGetTableRecord)
tests["deleteRecord"].record(cscAppPerf.perfDeleteTableRecord)
tests["bulkAddRecord"].record(cscAppPerf.perfAddBulkTableRecord)

Statistics:
2015-09-16 19:20:09,472 INFO  i-20ff63e6-1 : Final statistics for this process:
2015-09-16 19:20:09,480 INFO  i-20ff63e6-1 :

             Tests        Errors       Mean Test    Test Time    TPS          
                                       Time (ms)    Standard                  
                                                    Deviation                
                                                    (ms)                      

(Test 1      3            0            255.33       16.74        0.15)         "createReport"
(Test 2      3            0            255.33       16.74        0.15)         "createTable"
(Test 3      3            0            255.33       16.74        0.15)         "createRecord"
Test 4       4            0            47.75        12.75        0.20          "readReport"
Test 5       4            0            4734.25      1530.59      0.20          "readTable"
(Test 6      3            0            255.33       16.74        0.15)         "readRecord"
(Test 7      3            0            255.33       16.74        0.15)         "updateReport"
(Test 8      3            0            255.33       16.74        0.15)         "updateTable"
Test 9       2            0            53.50        3.50         0.10          "updateRecord"
(Test 10     3            0            255.33       16.74        0.15)         "deleteReport"
Test 11      3            0            255.33       16.74        0.15          "deleteTable"
Test 12      2            0            39.50        0.50         0.10          "deleteRecord"
Test 13      1            0            74.00        0.00         0.05          "bulkAddRecord"

Totals       16           0            1259.62      2148.51      0.78        
             (21)         (0)                

This is more realistic and is actually happening - but seemingly not possible?                            

On Wed, Sep 16, 2015 at 1:56 PM, Philip Aston <[hidden email]> wrote:
On 16/09/15 14:29, Darren Ball wrote:
> Is it possible in the testrunner init to do something like:
>
> tests["abc"].record(self.abc)?
>
> Why is it that a specific java object method can not be instrumented?
> Seem like only the whole object, meaning every call within?
>

You can't instrument a specific method by passing it to record as above
is that under the covers because Java does not provide a references to
bound methods, and so the Jython expression "self.abc" is actually a
Jython wrapper. The wrappers change with each version of Jython so we
(no longer) rely on Jython internals to figure out what exactly they
refer to.

Instead you can use an overloaded version of record that accepts a
filter that specifies just the methods you care about. See
http://grinder.sourceforge.net/g3/instrumentation.html#Selective+instrumentation


- Phil



------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use



------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
grinder-use mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/grinder-use
Loading...