Problem enabling exporting to Kinesis.

Help
1

Environment

Ubuntu 18.04

Problem/Question

Installed AWS sdk for Kinesis
Recompiled netdata
Configured exporting.

No data is showing up in Kinesis and I have this error in the error log.

2021-12-09 18:37:53: netdata INFO : EXPORTING : thread created with task id 11386

2021-12-09 18:37:53: netdata INFO : EXPORTING : set name of thread 11386 to EXPORTING

2021-12-09 18:37:53: netdata INFO : EXPORTING : Processing connector instance (test_instance)

2021-12-09 18:37:53: netdata INFO : EXPORTING : Instance (test_instance) on connector (kinesis) is enabled and scheduled for activation

2021-12-09 18:37:53: netdata INFO : EXPORTING : Instance test_instance on kinesis

2021-12-09 18:37:53: netdata ERROR : EXPORTING : AWS Kinesis support isn’t compiled

During the compile I see

checking for AWS_CPP_SDK_KINESIS… yes. so it looks like the sdk is there but it is not using it?

and

checking if kinesis backend should be enabled… no

But I’m not sure if the backend matters with the new config, or even how to enable it if it does.

Any help in debugging this or getting it to work would be greatly appreciated, I’ve search till my eyes hurt and I can’t find the solution to this.

Is there some other compile option I need to use?

2

Please look at a PR that will be merged in a few days. Meanwhile, you can apply the changes manually to your configure.ac.

3

Thanks, I tried the changes.

I still get this error. I can’ figure out why it thinks the libs are not there.

checking for AWS_CPP_SDK_CORE… yes

checking for AWS_CPP_SDK_KINESIS… yes

checking for aws_checksums_crc32 in -laws-checksums… no

checking for aws_default_allocator in -laws-c-common… no

checking for aws_event_stream_library_init in -laws-c-event-stream… no

configure: error: AWS SKD third party dependencies required but not found

4

Maybe you installed the libs in /usr/local/lib. pkg-config can’t look for libs there. Try installing them into /usr/lib - use /usr as a prefix instead of /usr/local.

5

Still getting same error. It looks like the libs are installed in /usr/lib. though.

:/usr/lib$ ls libaws*
libaws-c-auth.a libaws-c-common.a libaws-c-event-stream.a libaws-c-http.a libaws-c-mqtt.a libaws-cpp-sdk-kinesis.a libaws-c-s3.a
libaws-c-cal.a libaws-c-compression.a libaws-checksums.a libaws-c-io.a libaws-cpp-sdk-core.a libaws-crt-cpp.a

6

Please run

nm -D /usr/lib/libaws-checksums.so | grep aws_checksums_crc32
nm -D /usr/lib/libaws-c-common.so | grep aws_default_allocator
nm -D /usr/lib/libaws-c-event-stream.so | grep aws_event_stream_library_init

Do you see the symbols? Please also look into config.log. What errors do you see? Expected output:

configure:12541: checking for aws_checksums_crc32 in -laws-checksums
configure:12564: gcc -o conftest -Og -ggdb -Wall -Wextra -Wformat-signedness -fno-omit-frame-pointer -fstack-protector-all -DNETDATA_INTERNAL_CHECKS=1 -D_FORTiFY_SOURCE=2 -DNETDATA_VERIFY_LOCKS=1 -pthread   conftest.c -laws-checksums    >&5
configure:12564: $? = 0
configure:12574: result: yes
configure:12585: checking for aws_default_allocator in -laws-c-common
configure:12608: gcc -o conftest -Og -ggdb -Wall -Wextra -Wformat-signedness -fno-omit-frame-pointer -fstack-protector-all -DNETDATA_INTERNAL_CHECKS=1 -D_FORTiFY_SOURCE=2 -DNETDATA_VERIFY_LOCKS=1 -pthread   conftest.c -laws-c-common    >&5
configure:12608: $? = 0
configure:12618: result: yes
configure:12629: checking for aws_event_stream_library_init in -laws-c-event-stream
configure:12652: gcc -o conftest -Og -ggdb -Wall -Wextra -Wformat-signedness -fno-omit-frame-pointer -fstack-protector-all -DNETDATA_INTERNAL_CHECKS=1 -D_FORTiFY_SOURCE=2 -DNETDATA_VERIFY_LOCKS=1 -pthread   conftest.c -laws-c-event-stream    >&5
configure:12652: $? = 0
configure:12662: result: yes
7

There are no .so files in /usr/lib. only .a files for the libaws stuff. perhaps this is a problem?

First instance of aws_checksums_crc32 in config.log fails with this…

configure:11913: checking for aws_checksums_crc32 in -laws-checksums

configure:11938: gcc -o conftest -O2 -pthread conftest.c -laws-checksums >&5

/usr/lib/gcc/x86_64-linux-gnu/7/…/…/…/x86_64-linux-gnu/libaws-checksums.a(crc.c.o): In function `aws_checksums_crc32’:

crc.c:(.text+0x2e): undefined reference to `aws_cpu_has_feature’

/usr/lib/gcc/x86_64-linux-gnu/7/…/…/…/x86_64-linux-gnu/libaws-checksums.a(crc.c.o): In function `aws_checksums_crc32c’:

crc.c:(.text+0x8e): undefined reference to `aws_cpu_has_feature’

crc.c:(.text+0xbe): undefined reference to `aws_cpu_has_feature’

/usr/lib/gcc/x86_64-linux-gnu/7/…/…/…/x86_64-linux-gnu/libaws-checksums.a(crc32c_sse42_asm.c.o): In function `loop_small_220’:

crc32c_sse42_asm.c:(.text+0xacb): undefined reference to `aws_cpu_has_feature’

collect2: error: ld returned 1 exit status

configure:11938: $? = 1

configure: failed program was:

8

Could you add to configure.ac aws-c-common in a couple of AC_CHECK_LIB

AC_CHECK_LIB(
    [aws-checksums],
    [aws_checksums_crc32],
    [have_libaws_checksums=yes],
    [have_libaws_checksums=no],
    [aws-c-common]
)

AC_CHECK_LIB(
    [aws-c-common],
    [aws_default_allocator],
    [have_libaws_c_common=yes],
    [have_libaws_c_common=no]
)

AC_CHECK_LIB(
    [aws-c-event-stream],
    [aws_event_stream_library_init],
    [have_libaws_c_event_stream=yes],
    [have_libaws_c_event_stream=no],
    [aws-c-common]
)

Please also let me know what error do you get for aws-c-common itself in config.log.

9

Still having no luck with this.

Clean ubuntu 18.04 installation.

Followed these instructions for sdk installation.

aws_kinesis

Then attempt to install with…

bash <(curl -Ss https://my-netdata.io/kickstart.sh) --enable-backend-kinesis

Still fails with libaws-cpp-sdk-kinesis required but not found. try installing aws C++ SDK

libaws-cpp-sdk-kinesis.a is in /usr/lib

Can anyone confirm these instructions work for them on Ubuntu 18.04?

10

I was able to reproduce the issue. I’ve updated the fix.

11

The fix has been merged. @Mark_Round can you try to install again? Should be fixed now.