【技术学习】Mongo Administration
Latest: Support Plan and Cloud Manager
Access Cloud Manager:
Access Mongo Support
Tuning
https://www.percona.com/blog/2016/08/12/tuning-linux-for-mongodb/
ALSO:
Copy collections across MongoDB databases and servers
NOTE:
In AWS we started using Mongo3.0. This created a problem for authentication from clients not supporting the new authentication mechanism (meteor can‘t connect).
Therefore, if using 3.0 we must downgrade the schema
// Use MongoDB 2.6 auth schema (default auth mechanism: MONGODB-CR)use admin;db.system.version.save({ "_id" : "authSchema", "currentVersion" : 3 }) |
Administration Tasks
Increasing Replica Timeout
Due to vmware (veeam) backups, it can hang our servers for a bit. We tried to increase the heartbeat by 10 seconds to prevent a mongo switch. Here is how you increase the timeout for > 3.2 mongo
cfg = rs.config() cfg.settings.electionTimeoutMillis=20000 #default is 10000 rs.reconfig(cfg)
Creating a replica set for single instance Mongo (AQA and Collector Fix)
Per Shiming‘s request, here is a short instruction how oplog can be enabled on stand-alone Mongo system: http://tuttlem.github.io/2014/06/13/how-to-setup-an-oplog-on-a-single-mongodb-instance.html This is year 2014 so it may not be very fresh but I hope it still works. Here is another reference: http://stackoverflow.com/questions/20487002/oplog-enable-on-standalone-mongod-not-for-replicaset. Accepted answer is outdated but take a look at comments and second answer. I believe it goes in line with the first link above. -Sergey
Adding Authentication to MongoDB Replica Set – Full Example
https://docs.mongodb.org/manual/tutorial/enable-authentication/
1) Login to the PRIMARY replica and create users.
# FULL root privilageuse admindb.createUser({user: "root", pwd: "root", roles: ["root"] } )# Create admin user for sxa databaseuse sxadb.createUser({user: "sxaAdmin", pwd: "sxaAdmin", roles: [ { role: "userAdmin", db: "sxa" }, { role: "readWrite", db: "sxa" } ] } )# readonly user for sxa dbuse sxadb.createUser({user: "readonlysxa", pwd: "readonlysxa", roles: [ { role: "read", db:"sxa" } ] } )# readonly user for anythinguse admindb.createUser({user: "readonlyadmin", pwd: "readonlyadmin", roles: [ { role: "readAnyDatabase", db:"admin" } ] } ) |
2) We add additional security. Use keyFile authentication for Intra-Replica communication (amongst themselves). Create a keyfile.
Reference: Mongo KeyFile Instructions
http://docs.mongodb.org/manual/tutorial/generate-key-file/
openssl rand -base64 741 > /etc/mongodb-keyfilechmod 600 /etc/mongodb-keyfilechown mongodb:mongodb /etc/mongodb-keyfile |
3) Copy this /etc/mongodb-keyfile to every replica instance. Don‘t forget to chmod and chown as above or mongod won‘t start.
4) Edit the /etc/mongod.conf file on every replica. Enable these lines
vi /etc/mongod.conf# Enable user authenticationauth=true#Enable keyFile for intra-replica authenticationkeyFile=/etc/mongodb-keyfile |
PAUSE: We‘re at the point where we want this all to take effect. We can do this without downtime by keeping a PRIMARY replica up at all times.
Reference: Mongo Maintenance Procedure
http://docs.mongodb.org/manual/tutorial/perform-maintence-on-replica-set-members/
5) Now we need to restart the replicas for these changes to take effect. Begin by logging into a SECONDARY replica and restart the mongod.
restart mongod |
6) Continue for all secondaries
7) Last, login to the PRIMARY. Tell it to give up mastership and become a SECONDARY
rs.stepDown() |
8) Now we‘re ok to restart mongod on the last replica without downtime.
restart mongod |
10) Extra Cred: The mongo URL that is needed for other applications such as meteor is in a very particular format when you use authentication.
MONGO_URL="mongodb://[username:]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]"Example:MONGO=URL="mongodb://sxaAdmin::27017,ec2-52-24-148-47.us-west-2.compute.amazonaws.com:27017,ec2-52-26-60-142.us-west-2.compute.amazonaws.com:27017/sxa?replicaSet=mongodb&readPreference=primaryPreferred" |
Get/Drop Users
use admindb.getUsers() use sxadb.getUsers()db.dropUser("sxaAdmin") use admindb.dropAllUsers() #! |
Add Roles
use sxadb.grantRolesToUser("sxaAdmin", [{ role: "readWrite", db: "local" } , { role: "userAdmin", db:"local" } ] ) |
Revoke Roles
use sxa
db.revokeRolesFromUser("sxaAdmin",[{ role: "readWrite", db: "local" } , { role: "userAdmin", db:"local" } ])Running Mongo Stand-Alone for debugging purposes.
We use --auth if we want to enable logging.
screenmongod --port 37017 --dbpath /data --authctrl-A-D |
Drop down Schema to 2.6 due to Authentication errors with 3.0
Support for the new authentication in MongoDB 3.0 has caused problems with meteor, and with tools like RoboMongo. Work around is to drop the schema down.
// Use MongoDB 2.6 auth schema (default auth mechanism: MONGODB-CR)use admin;db.system.version.save({ "_id" : "authSchema", "currentVersion" : 3 }) |
This may not be that simple. I found I needed to start mongod in stand-alone mode for every replica memeber with authentication disabled (remove the --auth), then set the authSchema value.
Resync MongoDB Replica Set
- Check MongoDB Replica Set status - if the secondary MongoDB replica member stays in "RECOVERING" state, it will require resync to get it back to correct state.
- login one of the MongoDB VM instance (199.71.143.62 or 199.71.143.63)
- Run: "mongo 199.71.143.62/sxa"
Run rs.status command in MongoDB shell. The following will be example of result
cmdctr:PRIMARY> rs.status(){"set":"cmdctr","date": ISODate("2015-06-08T01:00:51Z"),"myState":1,"members": [{"_id":0,"name":"199.71.142.249:27017","health":1,"state":7,"stateStr":"ARBITER","uptime":3818114,"lastHeartbeat": ISODate("2015-06-08T01:00:49Z"),"lastHeartbeatRecv": ISODate("2015-06-08T01:00:49Z"),"pingMs":0},{"_id":1,"name":"199.71.143.62:27017","health":1,"state":1,"stateStr":"PRIMARY","uptime":20854145,"optime": Timestamp(1433725251,3),"optimeDate": ISODate("2015-06-08T01:00:51Z"),"electionTime": Timestamp(1430122091,1),"electionDate": ISODate("2015-04-27T08:08:11Z"),"self":true},{"_id":2,"name":"199.71.143.63:27017","health":1,"state":3,"stateStr":"RECOVERING","uptime":1633250,"optime": Timestamp(1433184037,6),"optimeDate": ISODate("2015-06-01T18:40:37Z"),"lastHeartbeat": ISODate("2015-06-08T01:00:50Z"),"lastHeartbeatRecv": ISODate("2015-06-08T01:00:50Z"),"pingMs":3,"lastHeartbeatMessage":"still syncing, not yet to minValid optime 5574e47c:a2"}],"ok":1}
- Steps to resync MongoDB replica set - recommend to perform this procedure during light MongoDB traffic.
- Login the secondary MongoDB VM instance - 199.71.143.63 in the above example.
- Stop MongoDB process
- supervisorctl
- stop mongo
- exit
- back up and delete MongoDB data folder
- cd /var/lib
- mv mongo mongo_bak_<timestamp> (replace <timestamp> to string like "06072015")
- mkdir mongo
- chown mongod:mongod mongo
- Start MongoDB process
- supervisorctl
- start mongo
- exit
- Wait for sync to complete by checking rs.status() or db.printSlaveReplicationInfo()
The result of db.printSlaveReplicationInfo() should show something like the following when sync is complete:
cmdctr:PRIMARY> db.printSlaveReplicationInfo()source:199.71.143.63:27017syncedTo: Thu May28201512:02:25GMT-0700(PDT)0secs (0hrs) behind the primary
Renaming a Mongo Replica
Example: Mongo replica set is using the name "compass-dev-2" instead of "compass-dev-2.xx.com" causing DNS name resolution errors. We can rename it as follows. Note: it is also the current primary server.
mongo
rs.stepDown()
Login to new primary
cfg = rs.config()
cfg.members[0].host="compass-dev-2.xx.com:27017”
rs.reconfig(cfg)
Maintenance
http://docs.mongodb.org/manual/tutorial/perform-maintence-on-replica-set-members/
For command center deployments, you must disable cron jobs on JBOSS, and the CC+ backend during the mongo restore or it can conflict with the restore process.
We should even shut down these components as well during the restore.
Backup/Restore
mongodump --out <backup_file>Example:This dumps the "sxa" database from the mongo server 199.71.142.151. It can be run anywhere, though it is faster to run locally and then tar the results and scp it over to your destination. mongodump -h 199.71.142.151 --out /tmp/goldbackup --db sxa Example:Note: You can restore to an alternate DB if desired.mongorestore --drop -d new_db_name mongodump/old_db_name mongorestore --drop --host 199.71.142.151 -d sxa gold-backup-080216/sxa |
Restoring from production DB for testing and staging systems
Note: We copy the backup NFS to the local disk. It seemed to me I could just restore directly from the NFS location and save time and a step, but in one case restoring from NFS failed on me, and I thought it might have been due to some NFS instability. I did not pursue that problem, I worked around it by copying the data to the local disk before the restore.
Example: Restore prod DB to gold system. #Identify gold system primary. ssh -p 1035 #Ensure there is enough local disk to hold the backup. Copy your backup over to the local disk. # Recommend you use screen in case you lose the ssh connection screen mkdir /root/restores cp -frp /backup/command-center-db/mongo/2017-02-21_020001/199.71.143.62 /root/restores cd /root/restores # WARNING: ENSURE THE HOST IP IS YOUR GOLD SYSTEM AND NOT PRODUCTION. It is possible to overwrite production remotely if you accidentally provide a production mongo IP. mongorestore --drop --host 199.71.142.151 --db sxa dump/sxa
Reference: http://docs.mongodb.org/manual/tutorial/backup-with-mongodump/
Have not watched this yet, but should be good: https://www.mongodb.com/presentations/webinar-backups-disaster-recovery
Fail Over Explanation
Tuning and/or Performance Monitoring
Script provided by David Jansen
General
Mongo URI URL syntax
For other applications, such as meteor.
%vmlnx-djansen<225> /opt/script/devops/mongo/dump_system_profile.py -husage: dump query info from system.profile with optional date query [-h] [--mongo_url MONGO_URL] [--after AFTER] [--before BEFORE] optional arguments: -h, --help show this help message and exit --mongo_url MONGO_URL Mongo Url (default mongodb://localhost:27017/sxa) --after AFTER DateTime of format like "2016-04-12T08:00:00Z" --before BEFORE DateTime of format like "2016-04-12T09:00:00Z" %vmlnx-djansen<226> /opt/script/devops/mongo/dump_system_profile.py --after 2016-04-13T16Z --before 2016-04-13T16:01Zbefore : 2016-04-13T16:01Z ==> 2016-04-13 16:01:00+00:00after : 2016-04-13T16Z ==> 2016-04-13 16:00:00+00:00using query: "{"ts": {"$lte": {"$date": 1460563260000}, "$gte": {"$date": 1460563200000}}}"---------------------------------- |
Installing 3.0 version
Installing mongodb-org-tools for 3.0 instructions are located here:
https://docs.mongodb.org/v3.0/tutorial/install-mongodb-on-ubuntu/
Reference
Installing a Specific Version of mongodb
Enabling Authentication in 2.6
https://docs.mongodb.org/v2.6/tutorial/enable-authentication/
Upgrade Mongo 2.6.5 to 3.2
Overview - Production Migration/Upgrade Plan
1) Launch 2 brand new centos7 servers.
2) Install mongo 2.6.5 on them ahead of time. Before the migration, like 1-2 hours before, add them to the replica set so the data syncs. We could set priority to zero to prevent these from becoming a primary, but this is not tested yet, so I‘m not going to include that step at this point.
3) Once all replicas are synced then you can begin upgrading the standbys.
4) Switch IP for meteor1 and meteor2 to new mongo servers mongo1 and mongo2
5) Assign NEW IPs for meteor1 and meteor2 and update nginx to reflect this.
Migrating 2.6.5 to stand-alone Servers
Create new centos 7.2 servers.
Install Mongo 2.6.5.
https://docs.mongodb.com/v2.6/tutorial/install-mongodb-on-red-hat/
[ ~]# vi /etc/yum.repos.d/mongodb.repo [mongodb] name=MongoDB Repository baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/ gpgcheck=0 enabled=1 [ ~]# sudo yum install -y mongodb-org-2.6.5 mongodb-org-server-2.6.5 mongodb-org-shell-2.6.5 mongodb-org-mongos-2.6.5 mongodb-org-tools-2.6.5
Setup supervisord /etc/supervisor/conf.d/mongo.conf
Had to change this to use sudo otherwise if I try to run as mongod with supervisord the process limits are not set to our specs.
[program:mongo] #command=mongod --smallfiles -f /etc/mongod.conf command=sudo -u mongod bash -c ‘mongod --smallfiles -f /etc/mongod.conf‘ numprocs=1 #directory=/var/lib/mongo stdout_logfile=/var/log/mongodb/ stderr_logfile=/var/log/mongodb/ autostart=true autorestart=true stopasgroup=true user=root ~
Create /etc/mongod.conf
Create path:
mkdir -p /home/mongo/mongodb chmod 777 /home/mongo/mongodb chown -R mongod:mongod /home/mongo chown -R mongod:mongod /home/mongo/mongodb
# mongod.conf #where to log logpath=/var/log/mongodb/mongod.log logappend=true # fork and run in background #fork=true #port=27017 dbpath=/home/mongo/mongodb # location of pidfile pidfilepath=/home/mongo/mongod.pid # Listen to local interface only. Comment out to listen on all interfaces. bind_ip=0.0.0.0 # Disables write-ahead journaling # nojournal=true # Enables periodic logging of CPU utilization and I/O wait #cpu=true # Turn on/off security. Off is currently the default #noauth=true #auth=true # Verbose logging output. #verbose=true # Inspect all client data for validity on receipt (useful for # developing drivers) #objcheck=true # Enable db quota management #quota=true # Set oplogging level where n is # 0=off (default) # 1=W # 2=R # 3=both # 7=W+some reads #diaglog=0 # Ignore query hints #nohints=true # Enable the HTTP interface (Defaults to port 28017). httpinterface=false # Turns off server-side scripting. This will result in greatly limited # functionality #noscripting=true # Turns off table scans. Any query that would do a table scan fails. #notablescan=true # Disable data file preallocation. #noprealloc=true # Specify .ns file size for new databases. # nssize=<size> # Replication Options # in replicated mongo databases, specify the replica set name here replSet=gold # maximum size in megabytes for replication operation log #oplogSize=1024 # path to a key file storing authentication info for connections # between replica set members #keyFile=/path/to/keyfile
Sudoers File
Need to disable need for tty, or supervisor process won‘t start.
vi /etc/sudoers
# #Defaults requiretty
Start mongo
supervisorctl update mongo
Add this new mongo server to the replica set.
Login to the primary server (assuming you know how to determine this)
> rs.add("192.35.85.16:27017")Access Analyze
Note, you can use this format: <replica>host:port,host:port,host:port, and you don‘t need to determine the primary node.
We use mongoexport to get AA organization list. As following mongoexport -h 199.71.143.62 -d sxa -c sxacms_organizations -f shortName,name,orgId,email -o $MONGOEXPORT --csv
Mongo to Redshift / Spark Aggregation
/home/ubuntu/sxadp-config-file
Routing Rules (Optional) - CURRENTLY NOT WORKING, and not sure if this is a good idea because two IPs would be going to the same mongo service.
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j DNAT --to 192.168.1.2:8080 iptables -A FORWARD -p tcp -d 192.168.1.2 --dport 8080 -j ACCEPT
Upgrading Mongo 2.6 to 3.0
This is required on the path to 3.2. You can‘t go from 2.6 to 3.2.
https://docs.mongodb.com/manual/release-notes/3.0-upgrade/
New centos servers must have this mtu setting or aws storm will get hung up.
sudo ip link set dev <interface> mtu 1300
Must be set permanently. Not discussed here, it is located if you search the wiki for mtu.
https://docs.mongodb.com/v3.0/tutorial/install-mongodb-on-red-hat/
vi /etc/yum.repos.d/mongodb-org-3.0.repo
[mongodb-org-3.0] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/ gpgcheck=0 enabled=1
supervisorctl stop mongo sudo yum install -y mongodb-org-3.0.12 mongodb-org-server-3.0.12 mongodb-org-shell-3.0.12 mongodb-org-mongos-3.0.12 mongodb-org-tools-3.0.12
Upgrading Mongo 3.0 to 3.2
New centos servers must have this mtu setting or aws storm will get hung up.
sudo ip link set dev <interface> mtu 1300
Must be set permanently. Not discussed here, it is located if you search the wiki for mtu.
Centos
Create file
vi /etc/yum.repos.d/mongodb-org-3.2.repo [mongodb-org-3.2] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.2/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-3.2.asc
Do this for all the secondaries, then step-down the primary as the last step and upgrade that one last.
supervisorctl stop mongo sudo yum install -y mongodb-org-3.2.8 mongodb-org-server-3.2.8 mongodb-org-shell-3.2.8 mongodb-org-mongos-3.2.8 mongodb-org-tools-3.2.8 supervisorctl start mongo
Ubuntu 14.04
Reference:https://docs.mongodb.com/manual/release-notes/3.2-upgrade/
Reference: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
Begin with a standby member of the replica set.
- Login to a Standby mongo instance
- Shutdown mongo on the server
Add repo key
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list apt-get update
Install specific version of mongo. This happens to be the latest stable as well, but might be useful to make sure we‘re explict in all our upgrades.
sudo apt-get install -y mongodb-org=3.2.8 mongodb-org-server=3.2.8 mongodb-org-shell=3.2.8 mongodb-org-mongos=3.2.8 mongodb-org-tools=3.2.8
- Repeat for the other Standby.
Finally, let‘s do the primary.
Login to the primary
rs.stepDown()
Wait until this system no longer reports itself as Primary. Keep using rs.status() to check. When it finally is a standby then proceed.
Perform the usual upgrade of the binaries like you did before.
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
Install specific version of mongo. This happens to be the latest stable as well, but might be useful to make sure we‘re explict in all our upgrades.
sudo apt-get install -y mongodb-org=3.2.8 mongodb-org-server=3.2.8 mongodb-org-shell=3.2.8 mongodb-org-mongos=3.2.8 mongodb-org-tools=3.2.8
All servers are upgraded to 3.2. Lastly, connect to the current primary server and run this.
cfg = rs.conf(); cfg.protocolVersion=1; rs.reconfig(cfg);
DONE
On ubuntu you can pin the versions down to prevent accidental upgrade. Not sure on centos.
echo "mongodb-org hold" | sudo dpkg --set-selections echo "mongodb-org-server hold" | sudo dpkg --set-selections echo "mongodb-org-shell hold" | sudo dpkg --set-selections echo "mongodb-org-mongos hold" | sudo dpkg --set-selections echo "mongodb-org-tools hold" | sudo dpkg --set-selections
Downgrade
https://docs.mongodb.com/manual/release-notes/3.2-downgrade/#downgrade-replica-set
Note, for downgrade, you need to set a variable in the replica set called protocolVersion first, Then you can do the usual downgrade of each standby.
sudo apt-get install mongodb-org=3.0.1 mongodb-org-server=3.0.1 mongodb-org-shell=3.0.1 mongodb-org-mongos=3.0.1 mongodb-org-tools=3.0.1
Testing Upgrade
1) Restart storm topology
2) Possibly need to restart the collector if FA+/RT doesn‘t come back entirely. Do that after the storm topo has been restarted
Centos
reference: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-red-hat/
Mongo Cloud Manager
Fix for centos7 and automation agent
To the best of my knowledge, Fedora 20 is equivalent to RHEL 7 which is not yet supported by MongoDB. There is work planned for this latter OS, but I cannot guarantee a timetable for that release. In the meantime, I believe it is possible to work around the issue with the following: cd /lib64 sudo ln -s libsasl2.so.3.0.0 libsasl2.so.2 Hopefully this helps, and allows you to start the agent.