Do any pattern exists for renaming live a MongoDB collection?

Do any pattern exists for renaming a MongoDB collection live in production environment? The collection is accessed both for reading and for writing in a non sharded environment. The requirement is to not have downtime. Some different microservices access the collection in different ways. The result must be obtained in a progressive way, releasing updated services independently from each other.

For relational databases several patterns were discovered and documented in the excellent book Refactoring Databases: Evolutionary Database Design by Scott Ambler and Pramod Sadalage and I would have used the Rename Table pattern to implement if the database was relational, that but it seems that with MongoDB this is not possible.

MongoDB index issue

I have a simple collection that contains two fields: timestamp and value (float). I need to get min and max values in the timestamp range. The collection contains approx 10000000 documents. I’ve add indexes like {"value" : 1, "timestamp" : 1} and {"value" : -1, "timestamp" : 1}. Queries like db.getCollection('Sample').find({'timestamp' : {'$ gte': 1714284644, '$ lte': 1745820644}}).sort({'value':-1}).limit(1) took about 1 second. Count without limit shows 525600 documents. "Explain" show that query covered by one of the idexes. IXSCAN takes most of time. Is there any way to speed this query up?

How to increase number of open connections for mongoDB from clients (C#)

mongoDB v4.0.4 >> replica set of 3 nodes (not sharded)

Previously

su - mongo7937 No directory, logging in with HOME=/ $   ulimit -Hn 1048576 $   ulimit -Sn 1024 $    cat /etc/pam.d/common-session | grep pam_limits.so  cat /etc/sysctl.conf | grep fs.file-max 

changed to

su - mongo7937 No directory, logging in with HOME=/ $   ulimit -Hn 1048576 $   ulimit -Sn 64000 $    cat /etc/pam.d/common-session | grep pam_limits.so session required        pam_limits.so  cat /etc/sysctl.conf | grep fs.file-max fs.file-max = 2097152  sysctl -p 

So earlier my max number of connections were limited to 818 which is 80% of 1024. As per recommendation from mongoDB documentation I’ve increased the limit to 64000 but still when mongo c# client reaches max open connections to 818, my mongoDB abruptly stopping.

Is there working command set for c sharp driver to limit number of connection client end ? because whatever we tried is not working, can someone share if that is working for you.

Else can someone tell me the right steps to limit number of open connections from mongoDB side and at server user level.

How to disable mongodb cache

I am performing some calculations on datasets and adding some new information to each collection row. I have over 8 billion records in the database.

I do not need the cache as I am only going to traverse all rows and update them. MongoDB is caching data in memory. There is a way to make it memory-optimized. Is it possible to disable cache or clear it as soon as I have done processing? I have a 64 GB computer, it’s taking more than 25 GB of memory.

MongoDB Shema to support concurrent update on a document

We were working on a project with a 300 documents with currentValue field in a main collection, in order to track the history of each document of first collection. we created another collection named history with approximately 6.5 millions of documents. For each input of system we have to add around 30 history item and update currentValue field of main collection, so, We tried computational field design pattern for currentValue, which lead us to have writeConfilict in concurrent situations (at concurrency of around 1000 requests).

Then we tried to compute currentValue field with sum (amount field) and groupBy(mainId field) on history collection which takes too long (> 3s).

Main collection docs:

{     "_id" : ObjectId(...),     "stock" : [          {             "currentAmount" : -313430.0,             "lastPrice" : -10.0,             "storage" : ObjectId("..."),             "alarmCapacity" : 12         },          {             "currentAmount" : 30,             "lastPrice" : 0,             "storage" : ObjectId("..."),             "alarmCapacity" : 12         },         .        .        .     ],     "name" : "name",  }   

History collection docs:

{     "_id" : ObjectId("..."),     "mainId" : ObjectId("..."),     "amount" : 5, } 

If you have any other idea to handle this situation(application or db level), I would be thankful.

UPDATE 1

The update query if I use computed pattern would be:

mainCollection.findOneAndUpdate(             {                 $  and: [                     { _id: id },                     { "stock.storage": fromId },                     { "stock.deletedAt": null }                 ],             },             {                 $  inc: {                     "stock.$  .currentAmount": -1 * amount,                 }             },             {                 session             }         ) 

And Aggregation pipeline if I want to calculate currentAmount everytime:

mainCollection.aggregate([             {                 $  match: {                      branch: new ObjectId("...")                 }             },             {                 $  group: {                     _id: "$  ingredient",                     currentAmount: {                         $  sum: "$  amount"                     }                 }             }]) 

MongoDB running sum over 3 days per ID

I have the following json file (small sample) which consists of purchase records of 4 different customers across different dates. I am required to use MongoDB/nosql to identify which customers have made at least a total of 8 new purchases over a consecutive period of 3 days.

In this case, customer ABC made 32 total purchases from 2020-05-01 to 2020-05-03 (3 consecutive days). Customer GHI also had 20 purchases from 2020-07-28 to 2020-07-30 (3 consecutive days). So my output should only contain customers ABC and GHI. What would be the code to get the output? Many thanks!

{"cust_id":"ABC", "date":"2020-05-01", "new_purchase":2},  {"cust_id":"ABC", "date":"2020-05-02", "new_purchase":16},  {"cust_id":"ABC", "date":"2020-05-03", "new_purchase":14},  {"cust_id":"ABC", "date":"2020-05-04", "new_purchase":0},  {"cust_id":"ABC", "date":"2020-05-05", "new_purchase":5},  {"cust_id":"DEF", "date":"2020-05-11", "new_purchase":3},  {"cust_id":"DEF", "date":"2020-05-12", "new_purchase":0},  {"cust_id":"DEF", "date":"2020-05-13", "new_purchase":0},  {"cust_id":"DEF", "date":"2020-05-14", "new_purchase":0},  {"cust_id":"DEF", "date":"2020-05-15", "new_purchase":1},  {"cust_id":"GHI", "date":"2020-07-28", "new_purchase":0},  {"cust_id":"GHI", "date":"2020-07-29", "new_purchase":3},  {"cust_id":"GHI", "date":"2020-07-30", "new_purchase":17},  {"cust_id":"GHI", "date":"2020-07-31", "new_purchase":0},  {"cust_id":"GHI", "date":"2020-08-01", "new_purchase":1},  {"cust_id":"JKL", "date":"2020-06-04", "new_purchase":7},  {"cust_id":"JKL", "date":"2020-06-05", "new_purchase":0},  {"cust_id":"JKL", "date":"2020-06-06", "new_purchase":0},  {"cust_id":"JKL", "date":"2020-06-07", "new_purchase":0},  {"cust_id":"JKL", "date":"2020-06-08", "new_purchase":0},  {"cust_id":"JKL", "date":"2020-06-08", "new_purchase":2} 

MongoDB cross datacenter replication without elections or high data availaility

I want to replicate a MongoDB database to another node, located in another datacenter. This is to help guard against data loss in the case of a hardware failure.

We don’t want/need high availability or elections; just a ‘close to live’ read-only copy of the database, located in another DC.

Everything I read says that you need an odd number of nodes, due to elections, but this isn’t something we need/want and I can’t find anything related to just having one primary, and one secondary (I might be being blind).

Is this something we can achieve with MongoDB, and if so are there any ‘gotchas’ or serious downsides we should consider?

Mongodb TTL Index not expiring documents from collection

I have TTL index in collection fct_in_ussd as following

db.fct_in_ussd.createIndex(     {"xdr_date":1},     { "background": true, "expireAfterSeconds": 259200} )   {     "v" : 2,     "key" : {         "xdr_date" : 1     },     "name" : "xdr_date_1",     "ns" : "appdb.fct_in_ussd",     "background" : true,     "expireAfterSeconds" : 259200 } 

with expiry of 3 days. Sample document in collection is as following

{     "_id" : ObjectId("5f4808c9b32ewa2f8escb16b"),     "edr_seq_num" : "2043019_10405",     "served_imsi" : "",     "ussd_action_code" : "1",     "event_start_time" : ISODate("2020-08-27T19:06:51Z"),     "event_start_time_slot_key" : ISODate("2020-08-27T18:30:00Z"),     "basic_service_key" : "TopSim",     "rate_event_type" : "",     "event_type_key" : "22",     "event_dir_key" : "-99",     "srv_type_key" : "2",     "population_time" : ISODate("2020-08-27T19:26:00Z"),     "xdr_date" : ISODate("2020-08-27T19:06:51Z"),     "event_date" : "20200827" } 

Problem Statement :- Documents are not getting removed from collection. Collection still contains 15 days old documents.

MongoDB server version: 4.2.3

Block compression strategy is zstd

storage.wiredTiger.collectionConfig.blockCompressor: zstd

Column xdr_date is also part of another compound index.

What’s the best way to encrypt and store text in a MongoDB database?

I have a "cloud service", which consists of 2 parts:

  • Web application, written in Next.js;
  • MongoDB database (uses MongoDB Atlas).

I allow users to sign in with GitHub and handle authentication using JWT. User can create & delete text files, which are saved in the database as so:

{     "name": string,     "content": string,     "owner": number    <-- User ID } 

I would like to encrypt the content so that I can’t see it in the database. I was thinking about using the Web Crypto API, but I’m not sure how I’m going to store the encryptions/decryption key securely.

What’s the best way to handle this case and which encryption algorithm should I use?

Best Way to loop through huge mongodb collection every minute?

I need to loop through mongodb collection EVERY MINUTE which will be huge in future, I have to loop through each object and then use some if else conditions and then perform an http post request inside IF condition. I am planning to use https://www.npmjs.com/package/node-cron this library for the purpose that will run a task every minute. In this task, i will get a mongodb collection, loop through all the objects using ForEach loop and then do some actions using if else conditions. Is this approach okay or there is better/efficient way?