HOWTO - HTTP EndPoint Collector with Cookie and user/pass

The same - cookie_file

Like this ?

cookie_file: /tmp/vma_cookie.txt 

Yes, this is my httpcheck.conf I used for testing:

jobs:
  - name: beast
    url: http://192.168.1.200:19999/api/v1/info
    cookie_file: /opt/cookie.txt

Seems ok but I am not sure if this is accurate :frowning:

Within Netdata I get 3 to 4 ms
Screenshot_20230321_181212

While within Catalina logs I get no bigger than 2 ms

[ DEBUG ] build[manager] build.go:164 received config group ('/etc/netdata/go.d/httpcheck.conf'): 3 jobs (added: 3, removed: 0)                                                                                   
[ DEBUG ] build[manager] build.go:313 building httpcheck[http://10.1.1.1/vma/mail/msgs] job, config: map[__provider__:file reader __source__:/etc/netdata/go.d/httpcheck.conf autodetection_retry:0 cookie_file
:/tmp/vma_cookie.txt headers:map[User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 body:j_username=user&j_password=pass] module:http
check name:http://10.1.1.1/vma/mail/msgs priority:70000 timeout:40 update_every:5 url:http://10.1.1.1/vma/mail/msgs]                                                                                        
[ DEBUG ] httpcheck[http://10.1.1.1/vma/mail/msgs] httpcheck.go:94 using URL http://10.1.1.1/vma/mail/msgs                                                                                                  
[ DEBUG ] httpcheck[http://10.1.1.1/vma/mail/msgs] httpcheck.go:95 using HTTP timeout 40s                                                                                                                      
[ DEBUG ] httpcheck[http://10.1.1.1/vma/mail/msgs] httpcheck.go:96 using accepted HTTP statuses [200]                                                                                                          
[ DEBUG ] httpcheck[http://10.1.1.1/vma/mail/msgs] collect.go:137 reading cookie file '/tmp/vma_cookie.txt'                                                                                                    
[ INFO  ] httpcheck[http://10.1.1.1/vma/mail/msgs] collect.go:144 &{{} {0 0} map[10.1.1.1:map[10.1.1.1;/vma;JSESSIONID:{JSESSIONID 93C47F180F77F91406B5B68D68D4F7D7 10.1.1.1 /vma  false false false true {0 315537897599 <nil>} {13906715261237331070 50457683 0x40aa8c0} {13906715261237331070 50457683 0x40aa8c0} 0}]] 1}
[ DEBUG ] httpcheck[http://10.1.1.1/vma/mail/msgs] collect.go:88 endpoint 'http://10.1.1.1/vma/mail/msgs' returned 200 (200 OK) HTTP status code
[ DEBUG ] run[manager] run.go:43 tick 0
CHART 'netdata.execution_time_of_httpcheck_http://10.1.1.1/vma/mail/msgs' '' 'Execution time' 'ms' 'go.d' 'netdata.go_plugin_execution_time' 'line' '145000' '5' '' 'go.d' 'httpcheck'                         
CLABEL '_collect_job' 'http://10.1.1.1/vma/mail/msgs' '1'                                                                                                                                                      
CLABEL_COMMIT                                                                                                                                                                                                     
DIMENSION 'time' '' 'absolute' '1' '1' ''                                                                                                                                                                         
                                                                                                                                                                                                                  
CHART 'httpcheck_http://10.1.1.1/vma/mail/msgs.response_time' '' 'HTTP Response Time' 'ms' 'response' 'httpcheck.response_time' 'line' '70000' '5' '' 'go.d' 'httpcheck'                                       
CLABEL 'url' 'http://10.1.1.1/vma/mail/msgs' '1'                                                                                                                                                               
CLABEL '_collect_job' 'http://10.1.1.1/vma/mail/msgs' '1'                                                                                                                                                      
CLABEL_COMMIT                                                                                                                                                                                                     
DIMENSION 'time' '' 'absolute' '1' '1' ''
                                                                                                         
BEGIN 'httpcheck_http://10.1.1.1/vma/mail/msgs.response_time'
SET 'time' = 3
END
                                                                                                         
CHART 'httpcheck_http://10.1.1.1/vma/mail/msgs.response_length' '' 'HTTP Response Body Length' 'characters' 'response' 'httpcheck.response_length' 'line' '70001' '5' '' 'go.d' 'httpcheck'
CLABEL 'url' 'http://10.1.1.1/vma/mail/msgs' '1'                                                      
CLABEL '_collect_job' 'http://10.1.1.1/vma/mail/msgs' '1'            
CLABEL_COMMIT                                                                                            
DIMENSION 'length' '' 'absolute' '1' '1' ''

BEGIN 'httpcheck_http://10.1.1.1/vma/mail/msgs.response_length'
SET 'length' = 35046
END

CHART 'httpcheck_http://10.1.1.1/vma/mail/msgs.request_status' '' 'HTTP Check Status' 'boolean' 'status' 'httpcheck.status' 'line' '70002' '5' '' 'go.d' 'httpcheck'
CLABEL 'url' 'http://10.1.1.1/vma/mail/msgs' '1'
CLABEL '_collect_job' 'http://10.1.1.1/vma/mail/msgs' '1'
CLABEL_COMMIT
DIMENSION 'success' '' 'absolute' '1' '1' ''
DIMENSION 'no_connection' '' 'absolute' '1' '1' ''
DIMENSION 'timeout' '' 'absolute' '1' '1' ''
DIMENSION 'bad_content' '' 'absolute' '1' '1' ''
DIMENSION 'bad_status' '' 'absolute' '1' '1' ''

I think that is accurate.

  • For Netdata it is the time to send a request and receive a response.
  • For Catalina I guess it is the time to receive/process a request and return a response.

So for Netdata it is +

  • in flight time Netdata->Catalina
  • in flight time Catalina->Netdata
  • read response time

Nice that it works. Thanks for testing.


Do I understand your first post right?

  1. get cookies from http://10.1.1.1/myapp/index
  2. using cookies from 1. do request to http://10.1.1.1/myapp/j_security_check and get new cookies.
  3. using cookies from 2. do request to http://10.1.1.1/myapp/myappfolder (the actual endpoint you want to monitor).

Hi ilyam,

Ok I will test again with wget and measure the times from the same netdata server.

Yes, well in my case it’s enough to start from step 2. So first get the cookie from 2 and use it with user/pass in submitted form. Then use this cookie to request the rest of pages. Is a simple cookie mechanism that most “legacy” and currently web app are using. Is there a plan to implement somehow this as a feature within netdata? I believe that would be very useful to other ppl too.

Glad that I helped to test phase though :slight_smile:

BR

I still believe that measure is not accurate. Below are more samples

/usr/local/jboss-6.0.0.Final/server/default/log $ tail -f -n 1000 access.log  | grep pgro
0.972 10.1.1.5 - 8844  [22/Mar/2023:12:26:10 +0000] http-0.0.0.0-80-8 "GET /vma/index HTTP/1.1" 200 13492 "http://10.1.1.74/vma/index" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36-pgro"
0.378 10.1.1.5 - 8844  [22/Mar/2023:12:26:17 +0000] http-0.0.0.0-80-8 "GET /vma/index HTTP/1.1" 200 13492 "http://10.1.1.74/vma/index" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36-pgro"
1.026 10.1.1.5 - 8844  [22/Mar/2023:12:26:21 +0000] http-0.0.0.0-80-8 "GET /vma/index HTTP/1.1" 200 13492 "http://10.1.1.74/vma/index" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36-pgro"

Below is a curl from the same server running netdata

time curl....
.
    xhrAlcohol.send();    
  </script>
</td>
   </tr>
  </table>
  </center>
</body>
</html>

real    0m1.044s
user    0m0.010s
sys     0m0.007s

And this is NetData Graph

We have a ms time so with curl I get 1 sec and 44 ms while in NetData I am getting <> 1 ms

Maybe I am missing something here?

Edit : I added an other http check public like www.google.gr and it’s accurate, but that’s not the case when this running internally :frowning:

I said accurate because:

  1. we get the current time (start_time).
  2. do HTTP requests.
  3. get the current time again (end_time), and calculate duration: end_time - start_time.

This is code (it is ± 1 millisecond because of precision).

It is accurate because there is nothing in between. It is exactly how long it took to do the HTTP request.

Hi Again,

On the Google Chrome I am getting the same Response speed as Catalina, how this is possible? I am using the same cable the same switch

3.014 10.1.1.1.1 - 8844  [22/Mar/2023:14:40:52 +0000] http-0.0.0.0-80-8 "GET /vma/mail/msgs?highfid=49&pageno=1080700 HTTP/1.1" 200 25894 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"

Screenshot_20230322_134305

Below is similar from netdata server executed as wget, please notice the ms/time

root@10.1.1.5:/etc/netdata$ wget --proxy=off  --load-cookies /tmp/vma_cookie.txt -O /dev/null http://10.1.1.74/vma/mail/msgs                                                                                  
--2023-03-22 14:07:45--  http://10.1.1.74/vma/mail/msgs                                                                                                                                                         
Connecting to 10.1.1.74:80... connected.                                                                                                                                                                        
HTTP request sent, awaiting response... 200 OK                                                                                                                                                                    
Length: unspecified [text/html]                                                                                                                                                                                   
Saving to: ‘/dev/null’                                                                                                                   
                                                                                                                                                                                                                  
/dev/null                      [  <=>                                 ]  34.22K  35.5KB/s    in 1.0s

2023-03-22 14:07:46 (35.5 KB/s) - ‘/dev/null’ saved [35046]

root@10.1.1.5:/etc/netdata$--2023-03-22 14:07:40--  http://10.1.1.74/vma/mail/msgs
Connecting to 10.1.1.74:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/dev/null’

/dev/null                      [  <=>                                 ]  34.22K  52.4KB/s    in 0.7s

2023-03-22 14:07:40 (52.4 KB/s) - ‘/dev/null’ saved [35046]

root@10.1.1.5:/etc/netdata$ wget --proxy=off  --load-cookies /tmp/vma_cookie.txt -O /dev/null http://10.1.1.74/vma/mail/msgs
--2023-03-22 14:07:41--  http://10.1.1.74/vma/mail/msgs
Connecting to 10.1.1.74:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/dev/null’

/dev/null                      [   <=>                                ]  34.22K  32.2KB/s    in 1.1s

2023-03-22 14:07:42 (32.2 KB/s) - ‘/dev/null’ saved [35046]

root@10.1.1.5:/etc/netdata$ wget --proxy=off  --load-cookies /tmp/vma_cookie.txt -O /dev/null http://10.1.1.74/vma/mail/msgs
--2023-03-22 14:07:43--  http://10.1.1.74/vma/mail/msgs
Connecting to 10.1.1.74:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/dev/null’

/dev/null                      [   <=>                                ]  34.22K  33.1KB/s    in 1.0s

2023-03-22 14:07:44 (33.1 KB/s) - ‘/dev/null’ saved [35046]

Ok so for example 5.0 means 5 seconds and 0 milliseconds? or it’s 5 milliseconds?

Screenshot_20230322_163902

it means 5 milliseconds (see units - ms)

I apologize for reviving a dead thread, but we did not help you sufficiently here @pgro

Did you ever figure out where the discrepancy in the response times came from? The
‘httpcheck_http://10.1.1.1/vma/mail/’ response time that is received in milliseconds seems to align very well with the slightly lower times that you saw in Catalina, but I never saw a wget for that step.

Further down I saw a completely different request though, to “http://10.1.1.74/vma/index” and no logs from netdata for that request. The chart image you added has the title cut off, so I don’t know what I was looking at. If I was indeed looking at a request to /index, then I’m fairly certain it was a failed request. i.e. you received a response, but it was access denied or something like that. See below on
how to validate the response.

Since the implementation @ilyam8 did is not for multiple steps, what you could do is the following:

  • Set up three separate httpcheck jobs, one for each step:
    • index: no cookie - this process would result in receiving a cookie that is thrown away
    • j_security_check: cookie.txt - this process would result in receiving a cookie that is thrown away
    • myappfolder: cookie1.txt

You would need to generate cookie.txt and cookie1.txt yourself, say every minute or so.
So the timings you see would not be for requests done in order.

For this to work, your application should be able to allow someone getting data from myappfolder with cookie1, while the same user is authenticating a second time, to get another cookie.txt.

If the timings seem too short, make certain you validate the responses received. HTTPcheck allows you do look for a specific pattern in the response, to verify that it is successful.