January 2022 Summaries

"Let's Talk Engineering" is a community initiative at CircleCI, facilitated over the past three years to foster interdisciplinary discussions among employees, including those from non-engineering backgrounds. The sessions, which attract 10 to 30 participants, emphasize a casual, inclusive environment where participants can connect, share experiences, and gain insights into various topics, ranging from technical subjects to leadership and personal growth. Inspired by a personal experience of engaging with tech leaders, the initiative aims to create a space that replicates informal discussions akin to "dinner chat," helping bridge the gap created by remote work. It encourages transparent, empathetic interactions within the community, underscoring the importance of diverse perspectives in engineering.

The text provides a detailed guide on integrating security into the application development process through DevSecOps, with a focus on automating security checks within the CI/CD pipeline using CircleCI. It outlines a tutorial for setting up periodic security scans using a web form that tests for malicious domain entries, leveraging Webshrinker for domain threat assessment. The tutorial includes prerequisites such as Node.js, Git, and a CircleCI account, and walks through cloning a demo application, implementing automated security tests with Puppeteer, and configuring a continuous integration pipeline. The tutorial emphasizes the importance of separating development and testing teams to ensure comprehensive security evaluations and concludes by explaining how to set up scheduled pipelines in CircleCI to automate regular security scans, ensuring that the application's security status is continuously updated and managed even in the absence of new code commits.

The text discusses the roles of monitoring and observability in software development, emphasizing their importance in maintaining the health and performance of applications, particularly within cloud-native and complex, distributed environments. Monitoring involves tracking predefined metrics and key performance indicators to detect deviations from expected behavior, providing real-time insights and alerts for quick responses. In contrast, observability offers a deeper understanding by analyzing a system's internal state through its external outputs, using telemetry data like logs, metrics, and traces to diagnose complex issues. Both practices, when integrated into CI/CD workflows, enhance productivity by enabling early issue detection, faster feedback loops, and more confident releases. The text highlights how CircleCI, by integrating observability and monitoring tools, supports a proactive approach to application health and reliability, aiding teams in building resilient systems that meet performance and user expectations.

Performance testing is crucial for evaluating the stability and responsiveness of systems under varying workloads, and k6 is an open-source framework designed to facilitate this process for developers. This tutorial guides users through setting up k6 to perform load testing on an API hosted on Heroku, demonstrating how to automate these tests within a continuous integration pipeline for consistent feedback. It covers the basics of installing k6, writing and executing tests that simulate multiple users, and interpreting results both locally and via the Grafana Cloud dashboard. The tutorial also explores advanced features such as using k6's Trend metrics to analyze API request times and integrating tests with CircleCI for automated execution. Through practical examples, users learn to identify potential bottlenecks in their systems by observing performance under stress, ultimately enabling them to prepare for future demands on their applications.

The article provides a detailed guide on setting up the CircleCI runner agent on a local machine to facilitate running tests within a CircleCI pipeline, emphasizing its utility for teams transitioning from local hardware to cloud-based solutions. It highlights the benefits of using CircleCI runners, such as leveraging existing hardware and enhancing security by running jobs on local infrastructure while still utilizing CircleCI's cloud features. The guide specifically outlines the steps to install and configure the runner agent on a Mac system, including registering the runner within an organization, setting up the necessary environment variables, and executing tests using connected local devices. The piece underscores the versatility of CircleCI runners in accommodating various platforms and the seamless integration of local resources with the cloud-based CI/CD processes.

CircleCI has expanded access to private orbs for all its customers, including those on the Free plan, enabling organizations to share CI/CD configurations securely within their own projects. Private orbs, unlike public ones, are restricted to the organization that publishes them, offering a way to manage internal tools and frameworks without public exposure. The article provides a guide on creating and deploying private orbs using CircleCI's CLI, emphasizing the importance of the --private flag during the initialization process. It also discusses the benefits of private orbs for tasks like deploying to proprietary infrastructure or handling custom hardware, and explains the development and testing process using CircleCI itself. The piece highlights the similarities between creating public and private orbs, with key differences lying in access permissions and the addition of the --private flag.

Creating an optimal CI/CD workflow with CircleCI involves overcoming challenges in configuring the infrastructure, where Docker layer caching (DLC) emerges as a significant technique to enhance efficiency. Docker, a containerization technology used on CircleCI, allows teams to build, test, and deploy applications in a consistent and collaborative manner by packaging everything into portable units, avoiding environment conflicts. Docker layer caching speeds up the process by saving and reusing Docker image layers so that unchanged layers don't need to be rebuilt in subsequent runs, significantly reducing build times and saving resources. This caching is particularly beneficial when the same image is built repeatedly in CI/CD pipelines, and it involves setting specific configurations in the config.yml file to enable it. While DLC provides noticeable efficiency improvements by reusing cached layers, it only impacts the time taken to build Docker images, not the time to initiate the primary Docker container. Furthermore, cached layers are retained for three days unless reused, and additional optimizations, such as dependency caching, can further enhance the pipeline's performance.

Software applications rely on efficient data exchange between interconnected systems to meet business needs, transitioning from inefficient polling requests to real-time webhooks. This text discusses how webhooks offer real-time data exchange with lower system overhead by notifying systems only when pre-agreed events occur. A tutorial guides users through building a Laravel API to function as a webhook for a CircleCI pipeline, detailing prerequisites like PHP and Laravel knowledge and software setup steps including creating a new Laravel project, setting up a CircleCI configuration, and linking to a GitHub repository. The tutorial also covers creating a database model, setting up environment variables, and developing a helper class for managing CircleCI requests. By implementing webhooks, systems can automate notifications, aggregate events, and analyze data more efficiently, opening possibilities for real-time operations like incident detection and management.

The tutorial provides a comprehensive guide to creating a React-based dashboard for visualizing data from CircleCI webhooks, focusing on translating workflow and job event data into interactive visual representations. It emphasizes the importance of data visualization in identifying trends, outliers, and insights from large datasets. The tutorial covers the prerequisites, including a strong understanding of JavaScript ES6 and basic React concepts, and guides users through setting up a new React application using Ant Design and react-chartjs-2 for the user interface and chart rendering. It details the creation of utility functions for API communication and date formatting, as well as the development of various components to display data in both table and chart formats. The tutorial concludes by demonstrating how to toggle between table and chart views using a switch, allowing users to visualize pipeline events effectively and gain a high-level understanding of their data. It also mentions the potential for exporting data for further analysis and encourages sharing the project for extended learning, with code available on GitHub and documentation for CircleCI webhooks provided.

The tutorial provides a comprehensive guide on using the k6 framework for testing HTTP request endpoints in full-stack application development. It explains how k6, an open-source performance testing tool written in the goja language, is optimized for JavaScript developers and provides features not found in other API testing frameworks. The tutorial walks through the installation of k6 on different operating systems, the creation of test files, and the unique structure of k6 tests, which utilize methods like group() and check() instead of traditional JavaScript testing methods. It also covers the lifecycle of a k6 test, including init, setup, VU, and teardown stages, and demonstrates how to integrate k6 with CI/CD pipelines using CircleCI. Additionally, it explains how to configure k6 tests for different environments using the scenarios feature, allowing developers to run tests in multiple environments with varying configurations. The tutorial culminates in setting up a CI/CD pipeline on CircleCI to ensure automated testing and verification of the tests' success.