• Working on building data pipelines which will be used to perform ETL using Pyspark.
• Build a server-less pipeline using Python and AWS components like Lambda, API
gateway, SNS, SQS, EMR cluster, Step functions, Kinesis, Redshift, DynamoDB etc
which will run the ETL, training (ML models) and prediction jobs.
• These pipelines are highly scalable and can handle terrabytes of data.
To deploy the above-mentioned infrastructure we use terraforms & cloudformation

• Build a large-scale end-to-end multi-threaded distributed web crawler from scratch single-handedly. • The product is based on the micro-services architecture where the services talk to each other using REST API calls or Asynchronously using Pub/Sub services. • Build a distributed job scheduler using Apache Kafka. • Apart from that my work also includes automating the whole framework, writing shell scripts (to take backups, to automate some processes, to archive the data etc), writing unit test cases, integration testing of the whole product, product infrastructure set up and making system level improvements.

• Working on a Large Distributed System in the domain of Web crawling. • Worked on a new asynchronous version of our web crawling product. • Writing complex web crawlers based on client’s requirement • Worked with infrastructure team to optimise the performance ofexisting system by making the system highly available and scalable. • My work at Promptcloud includes building distributed system, writing crawlers, writing shell scripts, writing unit test cases, working with git, Jenkins, fixing code issues, system level improvements.