Thursday 31 May 2018

API Data Returning Unicode Characters in Console

I am facing a rather confusing problem since the last two days. I am working on a document management system, that uses an API that pulls in data from SOLR. The data is in tune of around ~15Mbs, and pulls records of more than 4000+ documents. The API has response in this format -

{
    "documents" :[
                {
                   id: 123,
                   some_field : "abcd",
                   some_other_field : "abcdef"
} ,
                 {
                    id: 124,
                   some_field : "abcd1",
                   some_other_field : "abcdef1"
}
]
}

Everything works fine in browser. If I hit the endpoint in Chrome or Firefox browser, it gives me the correct output and I am able to see the JSON output.

However, if I try hitting the same API endpoint with a Java or JS code - the response code is 200, but the output in console (Terminal or Eclipse) shows unicode characters like \u0089 \u0078 U+0080 - all the output comes in this way, and since there are around 4000+ records being fetched by the API, the console kinda fills with all of these unicode characters.

The only difference that I see between the requests made from browser and the code is that in browser I can see Content-Encoding : gzip, while I cannot find this header from the code that I written . For eg - in JS code, through Chakram framework, I can check

expect(response).to.be.encoded.with.gzip

mentioned here. However, this returns a failure stating expected undefined to match gzip

What am I missing here? Is this something related to encoding/decoding or something entirely different?

Edit 1 : The Response Headers as seen in Network tab of Chrome :

cache-control: max-age=0, private, must-revalidate, max-age=315360000
content-encoding: gzip
content-type: application/json; charset=utf-8
date: Tue, 22 May 2018 06:07:26 GMT
etag: "a07eb7c1eef4ab97699afc8d61fb9c5d"
expires: Fri, 19 May 2028 06:07:26 GMT
p3p: CP="NON CUR OTPi OUR NOR UNI"
server: Apache
Set-Cookie : some_cookie
status: 200 OK
strict-transport-security: 
transfer-encoding: chunked
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-request-id: abceefr4-1234-acds-100b-d2bef2413r47
x-runtime: 3.213943
x-ua-compatible: chrome=1
x-xss-protection: 1; mode=block

The Request Headers as seen in Network tab of Chrome

Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: some_cookie
Host: abcd.bcd.com
IV_USER: demouser123
IV_USER_L: demouser123
MAIL: demouser@f.com
PERSON_ID: 123
Referer: http://abcd.bcd.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
X-CSRF-TOKEN: some_csrf_token

Edit 2 : The tests that I am using

describe('Hits required API',()=>{

    before(()=>{
        return chakram.wait(api_response = chakram.get(url,options));
    });

    it('displayes response',()=>{
        return api_response.then((t_resp)=>{
            console.log(JSON.stringify(t_resp));
            expect(t_resp).to.have.header('Content-Encoding','gzip');
        });
    });



from API Data Returning Unicode Characters in Console

1 comment:

  1. https://www.wizweb.in

    Wizweb Technology is a leading software development company custom website design, software development, SMS Provider, Bulk sms, transactional sms, promotional sms, mobile app development, Hosting Solution, seo(search engine optimization) and Digital marketing etc.

    ReplyDelete