Documentation

Find Docs in docs folder

TOODs on Speeding of Datasets

• Fix the script to generate data in sequence => older timestamps to newer timestamps. Impacts the speed of insertion
  • 10 mins for for 10M records
• Add pgpool to the server to server as a cache
• Add a Express Server to read queries using an API
• Add Varnish in front on it to verify the queries
• TimescaleDB to be configured with a data retention policy
• Setup continuous aggregates
• Queries to be modified based on this doc

TOODs on Speeding up Ingestions

• Implement 7Zip as the compression algorithm @htvenkatesh
• POC on Insertion of 30M records from a CSV file (from 7Zip file) @htvenkatesh
• Log Management - Log Rotation; Log Compression; Log Forwarding; @htvenkatesh
• NVSK - Cloudflare
• Throttle based on IP - Nest applications
• Add a cache on the fronted (etags)

TOODs on cQube Implementation

• Can be tracked here.
• Adapter Implemenation with the following use case
  • Dataset1
  • Dataset2
  • Dataset3
  • Dataset4
  • Dataset5
• Implement Dashlet Specification
• Auto Dashboard and Chart Creation using Metabase API and making it public
• A UI to import data and define configs
• Aggregation of Events
• Tranformers for Bulk Events management - Use Polars to do it in native JS/TS.