Creating An Oracle Solaris DevOps Like Environment, Update ZoneTypes, Roles, Reporting – Version-08

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

Finally had the time to updated the latest version of the Solaris DevOps Manager script. Just published Version 0.8 of the Solaris DevOps Manager, full details plus code are available on my Github repository

This version adds many new features, including creating different zone types, security like user/password/roles to create zones, better zone reporting. make sure to read the release notes for all changes.

Below is a copy of the release notes.

I also included below – an example of how to secure the configuration manager system.

Version 0.8 – Whats new

Added: Added a new db zone type. the db zone type will not rely on zfs snapshots. to snap a new db zone, it will use snapshots for the zone, and rsync to copy the db content, the full process is outlined below.

The DB zone type is initially created like any other zone, cloned from z-source (or whatever name you specify in devops_config.ini).
You create a file called db_version.ini with a db version number (for example 1), or the system will automatically created it with the first version number.

By installing the zone you specify the type as DB with option -t db for the zone type.

The clone process will work as follows.

  1. The system create a new DB zone when you run the script with option -t db.
  2. At install time the system will try to find the latest DB version, and create the new DB zone with the next available version. for example if the current version is 5, it will created the new zone with version 6.
  3. At time of the zone install a new DB file system will get initialized.
  4. At first boot of the new DB zone, the zone will mount the current db version (for example version 5), and also mount the new version (for example version 6).
  5. Automatically copy all data from current version for example version 5 => to new version for example 6.
  6. When completed, umount version 5 and version 6. re-mount version 6 as the normal db mount and start the db.

Example installing a new db zone.

./devops_manager.py -i db105 -u usera -t db -p
Evaluating system resources availability. Please wait...
Getting latest available db version for ifxdb-do_v-.
Successfully got next version as ifxdb-do_v-5.
Cloning VM/Zone z-db-v5-1556825338-db105 and associated file systems
Progress is being logged to zone_vm.log
--------------------------------
Cloning VM/Zone z-db-v5-1556825338-db105 and associated file systems
Progress is being logged to zone_vm.log
--------------------------------
Sync to new db ifxdb-do_v-5 is in progress.. please be patient...
This can take approximately 10-15 minutes to complete.
Note: The sync is running in HA only i.e. DR will complete first with data available once HA is up.

-------========= Standby data center =========-------
        VM/Zone Name: z-db-v5-1556825338-db105
        Hostname: dc1-devops2
        Zone Port: 31018
        DB Port: 31518
        Internal IP Address: 10.25.0.18
        VM Mount source: apps1_z-source
        DB Mount source: ifxdb-do_v-5
        VM Mount destination: /apps1
        DB Mount destination: /ifxsrv
...[snip]

Output of the updated devops_manager help script for the new option is below.

./devops_manager.py -h
...
  -t [{app,db}], --appType [{app,db}]
                        select zone/VM type. app or db(default is app)

By default, all new zones with type app, will get created with the current db version. the current/active db is the one specified in db_version.ini.
The version can be updated with the new -n option.

For example to update the current active db version you can run the below:

/devops_manager.py -n 6 -u usera
Please enter usera's LDAP password :
Note: you are accessing this application as a: admin

Getting latest available db version for ifxdb-do_v-.
Successfully updated DB version from 5 to 6.

Note: You need admin access to do so, otherwise you will get something like the message below(for more info check out the new user role section).

./devops_manager.py -n 5 -u usera
Please enter usera's LDAP password :
Note: you are accessing this application as a: user

Getting latest available db version for ifxdb-do_v-.
Access denied.

In addition. you can specify a DB version at zone creation or when rotating/updating the db with the -r db -v 5 options, otherwise the current db version will be used.

Added: The user creating a zone will now be stored as part of an SMF property. at time of login you will get the below message (if you are not the developer created the zone).

ssh -l USERB hosta
...
Oracle Corporation      SunOS 5.11      11.4    November 2018

----------------============= ++ =============---------------
*************** ATTENTION ATTENTION ATTENTION! **************
This virtual environment / zone was created by user:
USERA
And you are:
USERB

Please make sure this is REALLY the virtual environment you are looking for.
----------------============= ++ =============---------------
Are you sure you want to continue?

Added: Two new required options ware added, -u user and -p password.
In addition. the -U was also added, more information is available below.
Note: for the -p option if you only specify -p without the password the system will prompt you for the password.

Note: The devops JSON database now includes the user created the zone, the database file is stored in ports.db, and might need to be manually updated.

Updated: The zone live status view got updated with the zones db real mount, an example is below.

./devops_manager.py -s -u usera -i jira122
Pulling status...
------------------------------
Finding server containing zone for jira122 in DR.
Pulling status...
------------------------------
Finding server containing zone for jira122 in HA.
Found jira122 on dc1-devops2 in HA.
Found jira122 on dc2-devops2 in DR.
===============================================================
******* NOTE: Informix is only running on dc1-devops2 *******
                         (devops2)                      
===============================================================

-------========= Active data center =========-------
        VM/Zone Name: z-1556827881-jira122
        Hostname: devops2
        Zone Port: 31019
        DB Port: 31519
        Internal IP Address: 10.25.0.19
        VM Mount: /apps1
        DB Mount: /ifxsrv
        VM Mount source: /export/apps1_z-1556827881-jira122
        DB Mount source: /export/ifxdb-do_v-5-z-1556827881-jira122
...

New: Rotating a VM/Zone DB file system will rotate using the db image initially used for creation.

An example rotate db is below.
Note: not much has changed in usage, however this was updated under the hood to accommodate the db changes (more about the at the end).

./devops_manager.py -i jira121 -r db -u usera
Finding server containing zone for jira121 in HA.
Found jira121 on dc1-devops1 in HA.
(/ifxsrv)Rotating HA(ifxdb-do_v-5-z-1556827550-jira121) mount in zone z-1556827550-jira121.. please wait...
(/ifxsrv)Rotation of HA(ifxdb-do_v-5-z-1556827550-jira121) mount in zone z-1556827550-jira121 completed successfully.
Finding server containing zone for jira121 in DR.
Found jira121 on dc2-devops1 in DR.
(/ifxsrv)Re-mounting DR(ifxdb-do_v-5-z-1556827550-jira121) mount in zone z-1556827550-jira121.. please wait...
(/ifxsrv)Re-mount of DR(ifxdb-do_v-5-z-1556827550-jira121) mount in zone z-1556827550-jira121 completed successfully.

Added/updated: The zone status was totally re-done, to include most information required(as requested by our developers), a few examples are below.

Default results for only the current authenticated user, without any options.

./devops_manager.py -l -u usera        
Please enter usera's LDAP password :
Note: you are accessing this application as a: admin

Checking system resources. please wait...

Note: Use -a option to display all active VM/Zones....
----------------============= HA =============---------------

Global Zone:         devops1 / (dc1-devops1)
----------------============= ++ =============---------------
Please wait... while we gather information...
z-1559233309-jiraMenu5           : [------------------------->  ]92%
Active Zones:
------------------------------------------------------------
   ZONE NAME                        TYPE  VER  PORT   CREATED BY
------------------------------------------------------------
   z-db-v6-1557163375-db106         DB    v6   31018  usera     
   z-1557858367-jiraMenu            APP   v6   31015  usera     
   z-1559231863-jiraMenu3           APP   v5   31020  usera     
   z-1559233309-jiraMenu5           APP   v5   31021  usera     
-------------------------------------------------------------
   * Denotes the default(active) DB
----------------============= ++ =============---------------
... [snip]  

Getting results for another user by adding -U userb

./devops_manager.py -l -u usera -U userb
Please enter usera's LDAP password :
Note: you are accessing this application as a: admin

Checking system resources. please wait...

Note: Use -a option to display all active VM/Zones....
----------------============= HA =============---------------

Global Zone:         devops1 / (dc1-devops1)
----------------============= ++ =============---------------
Please wait... while we gather information...
z-1559233309-jiraMenu5           : [------------------------->  ]92%
Active Zones:
------------------------------------------------------------
   ZONE NAME                        TYPE  VER  PORT   CREATED BY
------------------------------------------------------------
   z-1555593119-migration-master    APP   N/A  31016  userb    
   z-1557156336-UNX-159412          APP   v5   31017  userb    
   z-1557175109-UNX-159411          APP   v5   31019  userb    
-------------------------------------------------------------
   * Denotes the default(active) DB
----------------============= ++ =============---------------
... [snip]

Showing results for all zones by adding -a.

./devops_manager.py -l -u user -a
Please enter usera's LDAP password :
Note: you are accessing this application as a: admin

Checking system resources. please wait...

----------------============= HA =============---------------

Global Zone:         devops1 / (dc1-devops1)
----------------============= ++ =============---------------
Please wait... while we gather information...
z-1559233309-jiraMenu5           : [------------------------->  ]92%
Active Zones:
------------------------------------------------------------
   ZONE NAME                        TYPE  VER  PORT   CREATED BY
------------------------------------------------------------
   z-db-source                      DB    N/A  31002  root     
   z-1551898417-jircopy020519       APP   N/A  31012  userc  
   z-1552923560-jira124             APP   N/A  31013  userc  
   z-1554480751-UNX-999             APP   N/A  31011  userc  
   z-1555593119-migration-master    APP   N/A  31016  userb    
   z-fs-source                      FS    v1   31003  root     
   z-1557156336-UNX-159412          APP   v5   31017  userb    
   z-db-v6-1557163375-db106         DB    v6   31018  usera     
   z-1557175109-UNX-159411          APP   v5   31019  userb    
   z-1557858367-jiraMenu            APP   v6   31015  usera     
   z-1559231863-jiraMenu3           APP   v5   31020  usera     
   z-1559233309-jiraMenu5           APP   v5   31021  usera     
-------------------------------------------------------------
   * Denotes the default(active) DB
----------------============= ++ =============---------------
... [snip]

Getting the most information…(adding -l det and -a)
Note: -a for all zones, and -l det for full details.

./devops_manager.py -l det -u usera -a
Please enter usera's LDAP password :
Note: you are accessing this application as a: admin

Checking system resources. please wait...

----------------============= HA =============---------------

Global Zone:         devops1 / (dc1-devops1)
----------------============= ++ =============---------------
Please wait... while we gather information...
z-1559233309-jiraMenu5           : [------------------------->  ]92%
Active Zones:
------------------------------------------------------------
   ZONE NAME                        TYPE  VER  PORT   CREATED BY
------------------------------------------------------------
   z-db-source                      DB    N/A  31002  root     
   z-1551898417-jircopy020519       APP   N/A  31012  userc  
   z-1552923560-jira124             APP   N/A  31013  userc  
   z-1554480751-UNX-999             APP   N/A  31011  userc  
   z-1555593119-migration-master    APP   N/A  31016  userc    
   z-fs-source                      FS    v1   31003  root     
   z-1557156336-UNX-159412          APP   v5   31017  userb    
   z-db-v6-1557163375-db106         DB    v6   31018  usera     
   z-1557175109-UNX-159411          APP   v5   31019  userb    
   z-1557858367-jiraMenu            APP   v6   31015  usera     
   z-1559231863-jiraMenu3           APP   v5   31020  usera     
   z-1559233309-jiraMenu5           APP   v5   31021  usera     
-------------------------------------------------------------
   * Denotes the default(active) DB
----------------============= ++ =============---------------
Zones Details:
------------------------------------------------------------
   ID:                               1        
   Active Zone Count:                12       
   Global Zone Free Memory:          187464 Mb
   15 Minute Load Average:           0.25     
   Current DB version:               5        
------------------------------------------------------------

Global Zone:         devops2 / (dc1-devops2)
----------------============= ++ =============---------------
Please wait... while we gather information...
z-1559232422-jiraMenu4           : [------------------------>   ]89%
Active Zones:
------------------------------------------------------------
   ZONE NAME                        TYPE  VER  PORT   CREATED BY
------------------------------------------------------------
   z-1550602019-jira103             APP   N/A  31011  userc  
   z-1550610548-jira104             APP   N/A  31012  userc  
   z-1554125109-genero-poc          APP   N/A  31014  userc  
   z-fs-source                      FS    v1   31003  root     
   z-db-v5-1556825338-db105         DB    v5*  31018  usera     
   z-1557163823-UNX-159413          APP   v5   31013  userb    
   z-1557239055-family-test         APP   v5   31015  userb    
   z-1558040749-jiraMenu2           APP   v5   31016  usera     
   z-1559232422-jiraMenu4           APP   v5   31017  usera     
-------------------------------------------------------------
   * Denotes the default(active) DB
----------------============= ++ =============---------------
Zones Details:
------------------------------------------------------------
   ID:                               2        
   Active Zone Count:                9        
   Global Zone Free Memory:          174867 Mb
   15 Minute Load Average:           0.27     
   Current DB version:               5        
------------------------------------------------------------
----------------============= DR =============---------------

Enhanced: If you don’t specify an argument for the -i option, the script will prompt you for one.

Added: Added a -v option at zone creation or at zone rotate, the -v will set which db version to install/use.

Added: The system now checks for user and application roles.

There are a few roles defined out of the box feel free to change them around as needed.

Below is explained how it works.

devops_config.ini
...
[APP_ROLES]
# Define each part of the application which role required.
# roles are:
# superAdmin: 1, admin level1: 2, user: 5
delete_vm = 10
add_vm = 10
create_db = 10
rotate_app = 10
update_db_ver = 10
rotate_db = 5

Now, you have to create a file called access.db, the file defines what access the user gets, the higher the number the more access(based on whats defined in the devops_config.ini APP_ROLES section.

cat access.db
# Roles are specfied in devops_manager.ini
# roles are superAdmin:10, admin:9, user:5, guest:1
usera:5:user
admin1:10:superAdmin
admin2:10:superAdmin
joe:5:user

If the user is not in the access.db. access will be denied.

Enhancement: The script will now check and make sure the devops zone/ports db is not in-use while trying to update.

Securing the configuration manager system

Below is an example how to configure/secure a system were the devops_manager application is running on.

The application will only run as user confmgr.

Create a user confmgr in /etc/passwd, etc.., then add the below lines to /etc/user_attr

confmgr::::auths=*;profiles=Primary Administrator,System Administrator,Zone Cold Migration,Zone Migration,Zone Configuration,Zone Security,All;defaultpriv=all;lock_after_retries=no

Note: Non of our developers can login as user confmgr, they login by using their own login to the devops configuration manager system.

When they login they get a menu which will look something like the below.
In the below example we have two groups of users, an admin, and a regular user(developer) (you can setup many types of users – as many as you needed).

The admin menu looks like so.

    *******************************************************************
                        Please Enter Your Choice
    *******************************************************************
    1) Create New Zone       2) List Your Zones       3) List All Zones

    4) Refresh Database      5) Refresh Applicati     6) Delete Zone

    7) Update DB Version     8) Admin Shell

                             Q) Quit
   ____________________________________________________________________
    Enter Choice:

The regular user menu looks like so.

    *******************************************************************
                        Please Enter Your Choice
    *******************************************************************
    1) Create New Zone       2) List Your Zones       3) Refresh Database

    4) Refresh Applicati     5) Delete Zone

                             Q) Quit
   ____________________________________________________________________
    Enter Choice:

In /etc/profile we append the below lines.

if [ “${LOGNAME}” != “root” \
-a “${LOGNAME}” != “confmgr” ] ; then
exec /export/home/confmgr/multi_choice 0
fi

Below is how the multi_choice application looks like looks like, you place that in the confmgr home, typically in /export/home/confmgr.

#!/bin/bash

trap "" 2 3
window=$1

menu_list_dir="/export/home/confmgr"
menu_access=`grep ^$LOGNAME: ${menu_list_dir}/access.db|awk -F\: '{print $2}'`
while [ "${num}" = "" ]; do

clear
echo ""
echo "    *******************************************************************
                        Please Enter Your Choice
    *******************************************************************"

grep "^${window}" "${menu_list_dir}/menu_list_${menu_access}" |awk -F\, '{print $2}' |pr -3 -a -n\) -d -t -w76
echo ""
echo "                             Q) Quit"

        if [ "${repeat}" = 1 ] ; then
echo "    *******************************************************************
    >>>>>>>>>>  SORRY, you did not enter a proper Selction  <<<<<<<<<<<
    *******************************************************************"
        fi
echo  "   ____________________________________________________________________"
/usr/gnu/bin/echo -n  "    Enter Choice: "
read num
count=1
if [ "${num}" = Q ] || [ "${num}" = q ] ;then
   exit 0
fi
flag="Y"
   while [ "${flag}" = "Y" ]
        do
     if [ "${num}" = "" ] ; then
      flag="N"
     fi
     if [ "${num}" != "" ] ; then
getlinenum=`grep "^${window}" ${menu_list_dir}/menu_list_${menu_access} |wc -l |awk '{print $1}'`
if [ "${num}" -gt "${getlinenum}" ] || [ "${num}" -lt 1 ] || [ "${num}" = "" ] ; then
repeat=1; flag="N"; num=""
        else
numresult0=`grep "^${window}" ${menu_list_dir}/menu_list_${menu_access} | head -${num} | tail -1 |awk -F\, '{print $3}'`
flag="N"
     fi
fi
   done
done

eval $numresult0
echo "\nHit enter to continue."
read junk
clear
exec /export/home/confmgr/multi_choice 0

As you can see the menu_access variable will get set to the users access in access.db. i.e. it will call /export/home/confmgr/menu_list_[10|5] (based on how defined in access.db).

Next, you will have to create the menu_list_10 and menu_list_5 (or whatever your user/admin is mapped to).
An example of an admin menu_list is below.

0, Create New Zone, sudo -u confmgr devops_manager.py -u $LOGNAME -p -i
0, List Your Zones, sudo -u confmgr devops_manager.py -u $LOGNAME -p -l
0, List All Zones, sudo -u confmgr devops_manager.py -u $LOGNAME -p -l det -a
0, Refresh Database, sudo -u confmgr devops_manager.py -u $LOGNAME -p -r db -i
0, Refresh Application Code, sudo -u confmgr devops_manager.py -u $LOGNAME -p -r app -i
0, Delete Zone, sudo -u confmgr devops_manager.py -u $LOGNAME -p -d -i
0, Update DB Version, sudo -u confmgr devops_manager.py -n -u $LOGNAME -p
0, Admin Shell, /bin/bash

An example of a regular user menu

0, Create New Zone, sudo -u confmgr devops_manager.py -u $LOGNAME -p -i
0, List Your Zones, sudo -u confmgr devops_manager.py -u $LOGNAME -p -l
0, Refresh Database, sudo -u confmgr devops_manager.py -u $LOGNAME -p -r db -i
0, Refresh Application Code, sudo -u confmgr devops_manager.py -u $LOGNAME -p -r app -i
0, Delete Zone, sudo -u confmgr devops_manager.py -u $LOGNAME -p -d -i

The menu list is what options the user / developer will get when logging in to the system.

Of course the devops_manager.py application has many more options, but this simplifies usage for most users / developers / used cases using the application.

One last configuration is sudo. we need to configure sudo for all developers logging in to this system.

In our case we ware using LDAP, but you can use your local /etc/suders, will work as well.
The below example is what was appended to LDAP.

dn: cn=confmgr,ou=SUDOers,o=domain.com,dc=domain,dc=com
sudoOption: !authenticate
sudoHost: confmgr
sudoHost: dc1-confmgr1
sudoCommand: /export/home/confmgr/devops_manager.py
cn: confmgr
sudoRunAs: confmgr
objectClass: top
objectClass: sudoRole
sudoUser: usera
sudoUser: userb
sudoUser: userc

Note: Make sure the devops_config.ini is owned by confmgr user and only confmgr user can read it(as it contains passwords).

An example of file permissions

-rw-r--r--   1 root     confmgr      191 May 31 09:19 access.db
drwxr-xr-x   2 confmgr  confmgr        5 Feb  4 13:48 bin
drwxr-xr-x   2 confmgr  confmgr       12 Feb  4 13:48 conf
-rw-r--r--   1 confmgr  confmgr        1 May 16 10:05 db_version.ini
-rw-r-----   1 confmgr  confmgr     5593 May 30 17:10 devops_config.ini
-rwx--x---   1 root     confmgr    91887 May 30 17:12 devops_manager.py
lrwxrwxrwx   1 root     root          20 May 31 09:17 menu_list_10 -> menu_list_superAdmin
-rw-r--r--   1 root     root         382 May 31 09:18 menu_list_5
-rw-r--r--   1 root     root         560 May 16 10:17 menu_list_admin
-rw-r--r--   1 root     root         560 May 31 09:17 menu_list_superAdmin
-rw-r--r--   1 root     root         382 May 14 16:10 menu_list_user
-rwxr-xr-x   1 root     root        1664 May 14 17:16 multi_choice
-rw-r--r--   1 confmgr  confmgr     2764 May 30 12:22 ports.db
-rw-r--r--   1 confmgr  confmgr   148565 May 30 17:11 zone_vm.log

For additional details please check out the git repository here.

For a batter explanation you can check out Part 1 and Part 2 on how Creating A DevOps Like Environment In Oracle Solaris.

You might also like - Articles related to Oracle Solaris 11.4/Solaris 12.

Like what you're reading? please provide feedback, any feedback is appreciated.

Leave a Reply

avatar
3000
  Subscribe  
Notify of