Can I return JSON from an .asmx Web Service if the ContentType is not JSON?

If you will use WFC RESTfull Service instead of .asmx webservice you can implement all your requirements from your question. But usage of .asmx webservice with JSON as output required that you use at least contentType: 'application/json'. In different places you can find as a reason – security reason (see JSON Hijacking).

Probably ‘x-www-form-urlencoded’ is not your real problem. If you use dataType: "json" the parameters will be also send in the form “test1=value1&test2=value2”!!! The only difference, that all values should be JSON encoded. And jQuery don’t makes JSON encoding of the data. (You can look in the code of jQuery.) The main difference is only that “Accept: application/json” will be explicitly set in the request header.

Look at JQuery ajax call to httpget webmethod (c#) not working which I wrote recently. In this post was asked example of GET request. But it is almost the same. The only difference is, that data after encoding will be appended to the URL (for GET request). For POST request the data will be send in body. By the way if one set “processData: false” (see http://api.jquery.com/jQuery.ajax/) the GET data will be also send inside of body. So read my code example from JQuery ajax call to httpget webmethod (c#) not working. I hope you receive ideas how you can implement what you will.

I think you had problems with encoding complex data for your .asmx webservice call. Here is example with “complex” data:

[WebMethod]
[ScriptMethod (UseHttpGet = true, ResponseFormat = ResponseFormat.Json)]
public OutputData AjaxGetMore (InputData input) {
    return new OutputData () {
        id = input.id,
        message = new List { "it's work!", "OK!" },
        myInt = new int[] { input.myInt[0] + 1, input.myInt[1] + 1, 20, 75 },
        myComplexData = new InternalData () { blaBla = "haha", iii = new int[] { 1, 2, 3, 4, 5 } }
    };
}

where

public class InternalData {
    public string blaBla { get; set; }
    public int[] iii { get; set; }
}
public class OutputData {
    public string id { get; set; }
    public List message { get; set; }
    public int[] myInt { get; set; }
    public InternalData myComplexData { get; set; }
}
public class InputData {
    public string id { get; set; }
    public int[] myInt { get; set; }
    public InternalData data { get; set; }
}

and on the client side

var myData = { id: "li1234", myInt: [100, 200], data : {blaBla: "Hahhh!", iii: [10,20,30]}}
var myDataForjQuery = {input:$.toJSON(myData)};
$.ajax({
    type: "GET",
    url: "/Service1.asmx/AjaxGetMore", // + idAsJson,
    data: myDataForjQuery, // idAsJson, //myData,
    dataType: "json",
    contentType: "application/json; charset=utf-8",
    success: function(msg) {
        // var msg = {__type: "Testportal.outputData", id: "li1234", message: "it's work!", myInt:101}
        alert("message=" + msg.d.message + ", id=" + msg.d.id + ", myInt=" + msg.d.myInt); 
    },
    error: function(res, status) {
        if (status ==="error") {
            // errorMessage can be an object with 3 string properties: ExceptionType, Message and StackTrace
            var errorMessage = $.parseJSON(res.responseText);
            alert(errorMessage.Message);
        }
    }
});

($.toJSON come from the JSON plugin )
You can modify this example to HTTP POST, if you want.

One more small advice. If you use jQuery 1.4.x you can try use ‘none’ as dataType. See documentation: “If none is specified, jQuery will intelligently try to get the results, based on the MIME type of the response (an XML MIME type will yield XML, in 1.4 JSON will yield a JavaScript object, in 1.4 script will execute the script, and anything else will be returned as a string)”

Best regards