January 2025 Summaries

This blog post provides an updated guide on setting up and validating large language models (LLMs) like Deepseek-R1 using a combination of local and remote resources to balance simplicity, cost, and efficiency. It discusses the advantages and challenges of developing LLMs locally versus using hosted or self-hosted remote compute solutions, highlighting the constraints of local development due to hardware limitations and collaboration difficulties. The text recommends a tech stack involving a Linux virtual machine with GPU acceleration, Ollama for managing LLM operations, and ngrok for secure and persistent access to remote services, facilitating a quick and effective setup for testing and collaborating on AI models without broad internet exposure. The guide outlines the process of launching a remote VM, installing necessary tools, and securely connecting them to external services, emphasizing the importance of implementing access control measures to protect the models from unauthorized use. Additionally, it invites users to further explore security options and engage with the community for ongoing support and learning opportunities.

Ngrok has introduced enhanced security features for protecting online endpoints with the addition of IP Intelligence to their Traffic Policy management system. This feature provides real-time information about IP addresses accessing user endpoints, including geolocation data, autonomous system information, and IP categories, which are collections of IP addresses identified from public sources. The system allows users to filter traffic and make informed decisions to manage access, such as blocking Tor traffic, specific countries, or unwanted entities like known bots and scrapers. The IP Intelligence data is dynamically updated, reducing the need for users to maintain lists and allowing them to focus on managing traffic efficiently. Ngrok's IP Intelligence also aids in mitigating threats, as demonstrated by its ability to handle a significant DDoS attack by identifying and redirecting traffic from known proxy IPs. The service is designed to integrate seamlessly into existing Traffic Policy systems, offering users granular control to safeguard their services against various online threats, while ensuring legitimate traffic is not disrupted.

Ngrok's new circuit-breaker Traffic Policy action enhances the reliability of upstream services by rejecting requests when services respond with 500-level error codes and re-evaluating the health of the service before resuming normal traffic. This feature, inspired by electrical circuit breakers, helps prevent cascading failures, save CPU cycles, and improve user experience by blocking problematic requests before they impact the system. Implementing circuit breaking at the API gateway level provides consistent failure handling across services without the need for custom code, and ngrok's Traffic Policy allows flexible configuration of circuit breaker actions. This approach is particularly beneficial for systems with critical external dependencies, high-throughput environments, and microservice architectures. Ngrok's circuit breaker can replace rate limits for protecting service health, and its composable nature enables tailored configurations for cloud and internal endpoints. The circuit breaker action is available with every ngrok account, and users are encouraged to explore ngrok's developer documentation and community support for optimal use and configuration.

Ngrok has introduced a Basic Auth Traffic Policy action that allows users to enforce HTTP Basic Authentication on incoming requests, aligning with RFC 7235 guidelines. This feature simplifies adding Basic Auth to applications by specifying up to ten sets of credentials that can be applied to endpoints via Traffic Policy rules. Basic Auth operates by letting a user agent send a username and password with an HTTP request, which ngrok then verifies, forwarding authorized requests to the specified upstream service. Users can apply these policies using ngrok's dashboard, API, or CLI, facilitating secure application access without complex configurations. The Basic Auth action is particularly beneficial for protecting internal APIs, rapid prototyping, and integrating into legacy applications, although modern production environments might require advanced authentication methods like OAuth or JWT. Additionally, the feature supports testing and debugging by allowing unauthorized requests to proceed conditionally, and users can inspect and analyze authentication events through ngrok's dashboard or logs. Transitioning from command line-based Basic Auth to Traffic Policy rules is straightforward, enhancing ngrok’s traffic management capabilities.