How to handle parameters that can be an ARRAY or OBJECT in Retrofit on Android?

The trick is to write your own Gson deserializer for your Locations class. This would check whether the monument element is an object or an array. Like so:

public class LocationsDeserializer implements JsonDeserializer<Locations> {

    @Override
    public Locations deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException {

        JsonElement monumentElement = json.getAsJsonObject().get("monument");
        if (monumentElement.isJsonArray()) {
            return new Locations((Monument[]) context.deserialize(monumentElement.getAsJsonArray(), Monument[].class));
        } else if (monumentElement.isJsonObject()) {
            return new Locations((Monument) context.deserialize(monumentElement.getAsJsonObject(), Monument.class));
        } else {
            throw new JsonParseException("Unsupported type of monument element");
        }
    }
}

For the convenience, add a vararg constructor to your Locations class:

public class Locations {
    public List<Monument> monuments;

    public Locations(Monument ... ms) {
        monuments = Arrays.asList(ms);
    }
}

Your Monument class stays the same. Something like:

public class Monument {
    public int key;
    public String name;
    // public Categories categories;
    // public Address address;
}

Finally, create your own Gson object and pass it to the retrofit RestAdapter.

Gson gson = new GsonBuilder().registerTypeAdapter(Locations.class, new LocationsDeserializer()).create();

RestAdapter restAdapter = new RestAdapter.Builder()
            .setEndpoint(baseUrl)
            .setConverter(new GsonConverter(gson))
            .build();