November 2018 Summaries

Pulumi simplifies the integration of serverless functions within cloud infrastructures and applications, enabling users to create AWS Lambdas using straightforward JavaScript code. By allowing the seamless incorporation of numerous Lambdas into larger infrastructure and application code, Pulumi facilitates the development of comprehensive serverless applications. Epsagon complements Pulumi by providing a monitoring solution that offers an application-centric view of entire serverless applications, rather than focusing on individual functions. Users can easily integrate Epsagon's monitoring capabilities into their Pulumi applications by incorporating the @pulumi/epsagon package, which automatically instruments AWS Lambdas created from JavaScript callbacks. A practical example demonstrates how a small Pulumi application, consisting of about 200 lines of code, provisions infrastructure, hooks up Lambda event handlers, and creates a REST API to compare the latency of various Lambda event sources. Epsagon's console provides a visual overview of the application's architecture, detailing the interactions and performance across different components, such as event sources and data stores. Together, Pulumi and Epsagon facilitate the creation and monitoring of scalable, event-driven serverless applications.

Pulumi invites attendees of AWS re:Invent to visit their booth at Startup Central, Aria Quad, to discuss AWS and Pulumi. They are eager to engage in conversations about various topics, including serverless functions, containers and Kubernetes, managed services, and cloud-native infrastructure as code. Pulumi offers insights on managing AWS cloud resources more effectively using general-purpose programming languages and can assist in migrating from CloudFormation to Pulumi. Visitors can schedule a specific meeting time or visit the booth to explore these opportunities.

CircleCI has introduced CircleCI Orbs, reusable components designed to simplify CI/CD workflows by allowing users to incorporate pre-existing jobs or commands. Pulumi, a technology partner of CircleCI, has developed Pulumi Orbs for CircleCI, enabling easier deployment of Pulumi projects within CircleCI environments. These Orbs facilitate more declarative CI/CD processes by eliminating the need for custom scripts, and they are available for immediate use. The Orbs can be parameterized to customize aspects such as the Pulumi client version and other settings, streamlining the integration process. Additionally, Pulumi's integration with CircleCI extends beyond Orbs, with future updates set to include build metadata association and enhanced GitHub integration, offering visibility into cloud infrastructure changes from pull requests. Pulumi continues to enhance its tools to support continuous deployment in CI/CD workflows, with community support available for any questions or issues users may encounter.

Pulumi, inspired by technologies like Terraform and AWS CloudFormation, offers a cloud programming framework that uses infrastructure as code with general-purpose languages such as JavaScript, TypeScript, Python, and Go. This approach allows for more flexible and productive infrastructure provisioning with enhanced abstraction and reuse capabilities. The text details a process of converting existing Terraform configurations into Pulumi TypeScript, demonstrating how general-purpose languages simplify the creation and management of cloud infrastructure, such as a load-balanced web server on AWS EC2, with features like conditional SSH access. Pulumi's integration with familiar programming constructs, such as if statements and map functions, reduces boilerplate code and enhances maintainability and readability. The transformation of a Terraform configuration into a Pulumi TypeScript project involves using the tf2pulumi tool, which has been replaced by Pulumi CLI's conversion feature, allowing developers to leverage programming language ecosystems and idiomatic code structures for improved infrastructure management. Pulumi is open-source and encourages exploration to achieve expressive and reusable infrastructure as code solutions.

Simon Zelazny of Wallaroo Labs explores how Wallaroo's capabilities can improve data processing efficiency by leveraging Pulumi to provision ad-hoc clusters for on-demand data science tasks. Initially, a pandas classifier was optimized using Wallaroo's parallelization on a local machine, reducing the processing time of a million rows to approximately 16 minutes. However, to handle larger datasets, the approach shifted to deploying a cloud-based Wallaroo cluster using Pulumi and Ansible, allowing for automated setup and teardown of infrastructure. This strategy enables scalable, cost-effective processing of extensive datasets, demonstrating a significant speedup when scaling out with multiple machines. The post highlights how Wallaroo abstracts the complexities of distributed computing, allowing developers to focus on business logic while the platform manages infrastructure scaling. The results showed that a cluster of four machines offers an optimal balance between cost and performance for processing data ranging from 1 to 10 million rows, fitting within a one-hour processing window without incurring unnecessary costs.