[C#] Setting the User-Agent in HttpClient Request Headers

Overview

When performing web scraping or using specific Web APIs, it is often mandatory to set the User-Agent header to inform the server about the client making the request. In HttpClient, there are two main ways to specify this: by directly adding a string or by using structured classes. Implementing this correctly helps avoid access denials such as 403 Forbidden.

---

Specifications (Input/Output)

• Input: Destination URL.
• Output: The User-Agent information recognized by the server (console display).
• Operation: Sends a GET request with a custom User-Agent header attached.

---

Basic Usage

Use the DefaultRequestHeaders property. The ParseAdd method is the most straightforward and common approach.

// Directly specifying as a string
client.DefaultRequestHeaders.UserAgent.ParseAdd("MyApp/1.0 (compatible; MyBot)");

// Or using Add without validation
client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0...");

---

Full Code Example

The following code uses a test service (httpbin.org) that echoes back the request content to verify the custom User-Agent.

using System;
using System.Net.Http;
using System.Net.Http.Headers; // Required for ProductHeaderValue
using System.Threading.Tasks;

class Program
{
    // Share HttpClient throughout the application
    private static readonly HttpClient _sharedClient = new HttpClient();

    static async Task Main()
    {
        // 1. Setting the User-Agent
        // Method A: Adding as a raw string (useful for mimicking browsers)
        // _sharedClient.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64)...");

        // Method B: Structured specification (Product name and version)
        // Here, we act as a bot named "MyDataCollector"
        var productValue = new ProductHeaderValue("MyDataCollector", "1.0");
        var commentValue = new ProductInfoHeaderValue("(+https://example.com/bot-info)");

        // Clear existing settings before adding
        _sharedClient.DefaultRequestHeaders.UserAgent.Clear();
        _sharedClient.DefaultRequestHeaders.UserAgent.Add(new ProductInfoHeaderValue(productValue));
        _sharedClient.DefaultRequestHeaders.UserAgent.Add(commentValue);

        // Target URL (Endpoint to verify User-Agent)
        string url = "https://httpbin.org/user-agent";

        Console.WriteLine($"Sending request to: {url}");

        try
        {
            // 2. Execute request
            string responseJson = await _sharedClient.GetStringAsync(url);

            Console.WriteLine("--- Response from Server ---");
            Console.WriteLine(responseJson);
        }
        catch (HttpRequestException ex)
        {
            Console.WriteLine($"Communication Error: {ex.Message}");
        }
    }
}

---

Execution Result Example

--- Response from Server ---
{
  "user-agent": "MyDataCollector/1.0 (+https://example.com/bot-info)"
}

---

Customization Points

• Add vs. TryAddWithoutValidation: Add and ParseAdd strictly check if the header value conforms to RFC standards. To set non-standard or “poorly formed” User-Agent strings, use TryAddWithoutValidation.
• Per-Request Changes: If you share an HttpClient but need different User-Agents for each request, create an HttpRequestMessage instead of modifying DefaultRequestHeaders.

var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.UserAgent.ParseAdd("OneTimeBot/1.0");
await client.SendAsync(request);

---

Important Notes

• Well-Behaved Bots: When scraping, it is proper etiquette to include contact information (URL or email) in the User-Agent so server administrators can identify the source.
• Spoofing Risks: Mimicking common browsers (Chrome, Edge) might violate a site’s terms of service. Always check the target site’s rules.
• Shared Instance Impact: DefaultRequestHeaders apply to all requests using that HttpClient instance. Modifying them at runtime can lead to race conditions affecting requests on other threads. Ideally, set these only during initialization.

---

Advanced Application

User-Agent Rotation

To avoid detection during high-frequency access, you can randomly select a User-Agent from a list for each request using HttpRequestMessage.

var agents = new[] 
{ 
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...", 
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)..." 
};
var randomAgent = agents[new Random().Next(agents.Length)];

var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.TryAddWithoutValidation("User-Agent", randomAgent);
var response = await _sharedClient.SendAsync(request);

---

Conclusion

Setting the User-Agent in HttpClient is best handled by adding values to the DefaultRequestHeaders.UserAgent property during initialization to provide appropriate client metadata to the server. Since HttpClient is recommended for singleton use, headers should be configured once at startup, or handled individually per request message if dynamic changes are required.