뉴스레터

이메일로 Hortonworks의 새 업데이트를 받으세요.

한 달에 한 번 빅 데이터와 관련한 최신 인사이트, 동향, 분석 정보, 지식을 받아 보세요.

AVAILABLE NEWSLETTERS:

Sign up for the Developers Newsletter

한 달에 한 번 빅 데이터와 관련한 최신 인사이트, 동향, 분석 정보, 지식을 받아 보세요.

CTA

시작하기

클라우드

시작할 준비가 되셨습니까?

Sandbox 다운로드

어떤 도움이 필요하십니까?

* 저는 언제든지 구독을 해지할 수 있다는 점을 이해합니다. 또한 저는 Hortonworks이 개인정보 보호정책에 추가된 정보를 확인하였습니다.
닫기닫기 버튼
December 18, 2018
이전 슬라이드다음 슬라이드

Monitoring Kafka Streams Microservices with Hortonworks Streams Messaging Manager (SMM)

In last week’s blog Secure and Governed Microservices with HDF/HDP Kafka Streams Support, we walked through how to build microservices with the new Kafka Streams support in HDF 3.3 and HDP 3.1 that is fully integrated with Ranger, Schema Registry and other platform services. This blog is all about monitoring these microservices with Hortonworks Streams Messaging Manager (SMM).

Monitoring MicroServices with SMM

In a microservices architecture, you will see a proliferation of stand-alone decoupled services. Hence, monitoring and managing these services becomes extremely critical. SMM provides users a powerful tool to monitor and visualize their microservices and understand how data flows across these services.

Working off the trucking fleet use case example from the previous blog, you can view each of the three microservices as a consumer and producer as depicted in the below diagram.

Lets first focus on monitoring the stream between MicroService 1 and MicroService 2 where MicroService 1 is a Kafka producer into the driver-violation-events topic and MicroServce 2 is a consumer from that topic. The below video showcases how to monitor the interactions between these two Kafka Streams microservices.

 

As the above video showcased, SMM cured the Kafka blindness for the streams app between the two microservices shedding light on some important information including the rate at which MicroService 1 was producing data, the lag of MicroService 2 and understanding the details of the internal Kafka changelog topics created by the Kafka streams join operator.

Another common use case for monitoring Kafka streams application using SMM is the following:

  • Detect a Kafka Streams microservice that is lagging considerably behind in processing messages.
  • Scale out the Kafka Streams microservice by spinning up new instances of the app (adding new instances to the consumer group).
  • Validate that adding new instances has decreased the lag of the consumer group.

The below video showcases how SMM address this use case.

 

The addition of Kafka Streams to HDF 3.3 and HDP 3.1 integrated with platform services like Ranger, Schema Registry and other platform services provides app developers a comprehensive platform to build secure and governed microservices. With SMM, devops and platform operations teams have enterprise  tools to debug, monitor, and troubleshoot these microservices built using Kafka Streams.

What’s Next

In the next installment of the Kafka Analytics blog series, we walk through the new Hive and Kafka integration for the SQL access pattern.

답변을 남기십시오

귀하의 이메일 주소는 공개되지 않을 것입니다. 필수 내용은 *로 표시되어 있습니다.