Livoa LogoLivoa
Technology Map – Opensource Adoption @ VIL
DATA Ingestion
Apache Kafka
Flume
Spark Streaming
Data Processing (Data-lake)
IIAS* → imply → Druid 0.19
Hive TEZ
HDFS
Spark
Hbase
Data Querying and exploration
IBM COGNOS
SQL Query
Druid Pivot
Redis (In memory DB)
Data Management & Governance
APACHE ATLAS
APACHE RANGER
Hadoop KMS
Platform Management Services
Apache Ambari
Druid Clarity
Kerberos
CRUISE CONTROL
Workflow and Scheduling
Oozie
Resource Manager
Hadoop YARN
File system and Storage
Network Analytics - Near real-time Analytical Reporting
OSS Sites (Pull Files from Remote Network OSS servers SSH)
Node-1 Node-2 Node-3 ... Node-10 (Custom XML Based Kafka Producers Agents)
Kafka
Apache Spark Structured Streaming
Druid Timeseries database for Realtime Aggregates Hourly / Daily
Imply Pivot for Visualization
Ingestion
Consumption
Analytics In Motion – Druid Data Warehouse
Raw data (clicks, ad impressions) (network telemetry) (application events)
Staging (and Processing) Data lakes (Hadoop HDFS, Amazon S3) Message buses (Kafka, AWS Kinesis, Spark)
Druid Analytics Database
End User Application (Visualization and Reports)
Interactive queries Sub-second OLAP queries Self-service data analytics Point-and-explore integrated UI
Unlimited scale High query concurrency PB+ data volumes High ingestion throughput
Real-time & historical data Kafka/Kinesis integrations Data lake integrations Flexible & adaptive schemas
Best Cost/performance Configurable price performance Predictable performance and costs Fine-tune based on usage

VIL

by sandeep

0
0 uses