Friday, 19 July 2019

MongoDB search and pagination Aggregation Performance issue

I'm new in node js and MongoDB. I'm working on MongoDB search and pagination it is working good, but I have an issue is with time. it is tacked too much time in execution of county record and search record.

I use the small word for search then it works fast, if I use "long string" or "no record in database" then it's tack too much time 50 to 186.30 seconds. (it too too much time, I'm expected it 1 to 2 seconds)

I have more than 10,00,000 data on my record.

If I'm not including a count of the search word. it is tack 0.20 to 1.5 seconds, but when I'm counting recode with search word it is tack 25.0 to 35.0 seconds.

I have no idea how to decrease this time for count recodes with the search word(query optimization).

I have maximum try to query optimization.

I have also tried with

{
  $count: "passing_scores"
}

but no change on time. I'm stuck on it. I have to decrease the time of count with the search word.

SQL Query for example

  SELECT * FROM `post`
    Left JOIN catagory ON post.catid=catagory.id
    WHERE post_name LIKE '%a%' OR post_data LIKE '%a%' OR tags LIKE '%a%' OR post_url LIKE '%a%'

NODE and MongoDB

PostObj.count({},function(err,totalCount) {
        if(err) {
            response = {"error" : true,"message" : "Error fetching data"}
        }
        PostObj.aggregate([
        { $lookup:
                {
                    from: 'catagories',
                    localField: 'catagory.catagory_id',
                    foreignField: '_id',
                    as: 'catagories_data'
                }
        },
        {

            $match:
                {
                    $or: [
                        {"catagories_data.catagory_name": { $regex: new RegExp(search_data)}},
                        {"postname": { $regex: new RegExp(search_data) }},
                        {"posturl": { $regex: new RegExp(search_data) }},
                        {"postdata": { $regex: new RegExp(search_data) }},
                        {"tags": { $regex: new RegExp(search_data) }}
                    ]
                }
        },            
        { $limit : search_limit },
        { $skip : search_skip },
        { $group : { _id : "$_id", postname: { $push: "$postname" } , posturl: { $push: "$posturl" }  } } 
    ]).exec(function (err, data){  

        //end insert log data        
        if(err) {
            response = {"error" : true,"message" :err};
        } 

        if(search_data != "")
        {
            // count record using search word

            PostObj.aggregate([
                    { $lookup:
                        {
                            from: 'catagories',
                            localField: 'catagory.catagory_id',
                            foreignField: '_id',
                            as: 'catagories_data'
                        }
                },
                {

                    $match:
                        {
                            $or: [
                                {"catagories_data.catagory_name": { $regex: new RegExp(search_data)}},
                                {"postname": { $regex: new RegExp(search_data) }},
                                {"posturl": { $regex: new RegExp(search_data) }},
                                {"postdata": { $regex: new RegExp(search_data) }},
                                {"tags": { $regex: new RegExp(search_data) }}
                            ]
                        }
                },    
                { $group: { _id: null, myCount: { $sum: 1 } } },
                { $project: { _id: 0 } }   
            ]).exec(function (err, Countdata){  
                res.json({
                sEcho : req.body.draw,
                iTotalRecords: Countdata.myCount,
                iTotalDispla,yRecords: Countdata.myCount,
                aaData: data
            });
        }

        res.json({
            sEcho : req.body.draw,
            iTotalRecords: totalPages,
            iTotalDisplayRecords: totalPages,
            aaData: data
        });
    });
});

Also, I have to try this way but it is tack 35.0 to 49.0 seconds more than 1st code.

PostObj.aggregate([
    { $lookup:
               {
                            from: 'catagories',
                            localField: 'catagory.catagory_id',
                            foreignField: '_id',
                            as: 'catagories_data'
                        }
                },
                {

                    $match:
                        {
                            $or: [
                                {"catagories_data.catagory_name": { $regex: new RegExp(search_data)}},
                                {"postname": { $regex: new RegExp(search_data) }},
                                {"posturl": { $regex: new RegExp(search_data) }},
                                {"postdata": { $regex: new RegExp(search_data) }},
                                {"tags": { $regex: new RegExp(search_data) }}
                            ]
                        }
                }, 
    { '$facet'    : {
        metadata: [ { $count: "total" }, { $addFields: { page: NumberInt(3) } } ],
        data: [ { $skip: 20 }, { $limit: 10 } ] // add projection here wish you re-shape the docs
    } }
] )

If I do not use search word it is work good. I have an issue with when searching any word(count of records of that work without skip and limit)

Any solution for it?



from MongoDB search and pagination Aggregation Performance issue

1 comment: